Add dns work flows

[Genne23v]

Close #217 and parts of #270

This PR adds workflows as below.

1. Create a record in DB with PENDING status. Return id to update status later.
2. Create a record in Route53 (TTL property added due to error)
3. Poll Route53 in every second to get the updated status (INSYNC or something else)
4. Update the DB either with active or error

To test this, it needs manual steps as below.

• Create a hostedZone using unit test dns.server.test.ts. If you run all test, it will remove hostedZone. Run partial test.
• Get hostedZoneId from localhost:5053/moto-api and set AWS_ROUTE53_HOSTED_ZONE_ID
• Set ROOT_DOMAIN to starchart.com
• Go to http://localhost:8080/dev and hit Request DNS Record button

[SerpentBytes]

@Genne23v E2E tests are failing because WS_ROUTE53_HOSTED_ZONE_ID is missing.

[Genne23v]

@Genne23v E2E tests are failing because WS_ROUTE53_HOSTED_ZONE_ID is missing.

It shouldn't throw in testing environment. I will take a closer look tomorrow.

[humphd]

I filed #275 to improve the situation for local dev and needing a hosted zone in moto.

[humphd]

Agree with @dadolhay's comments, and added some more of my own.

[humphd]

Another thought I have while reading this. We're starting to add a lot of queues. Each queue requires more memory, Redis event handlers, etc. At some point, we'll have to measure. I know in the official docs, the BullMQ talks about queues not being a huge drain.

Related to the above, I also wonder if we should use the same queue or worker for all common tasks. For example: all Route53 operations are going to get made against the same API, and share the same rate limiting. We can set a rate limit on a worker to avoid hitting this.

From https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/DNSLimitations.html, here are the Route53 API limits we need to respect:

All Amazon Route 53 API requests
For the Amazon Route 53 APIs five requests per second per AWS account. If you submit more than five requests per second, Amazon Route 53 returns an HTTP 400 error (Bad request). The response header also includes a Code element with a value of Throttling and a Message element with a value of Rate exceeded.

Note
If your application exceeds this limit, we recommend that you implement exponential backoff for retries. For more information, see Error Retries and Exponential Backoff in AWS in the Amazon Web Services General Reference.

ChangeResourceRecordSets requests
If Route 53 can't process a request before the next request arrives, it will reject subsequent requests for the same hosted zone and return an HTTP 400 error (Bad request). The response header also includes a Code element with a value of PriorRequestNotComplete and a Message element with a value of The request was rejected because Route 53 was still processing a prior request.

[Genne23v]

Thank you for the issue. I will assign myself and work as a next step.

[Genne23v]

Another thought I have while reading this. We're starting to add a lot of queues. Each queue requires more memory, Redis event handlers, etc. At some point, we'll have to measure. I know in the official docs, the BullMQ talks about queues not being a huge drain.

I will take a look at this and write an issue for follow-up. Thank you

[humphd]

I think this looks like something we could take. I've made a few more comments, but in general, we could start with this and improve going forward.

[ghost]

Looks nice!

[Genne23v]

Looks nice!

Thank you for all your valuable feedback. It really helps me a lot!

[humphd]

Looks good, a few small things and it's ready.

[ghost]

Looks nice!

Thank you for all your valuable feedback. It really helps me a lot!

Cool, I'm happy if you find it useful :)

[humphd]

• Create a hostedZone using unit test dns.server.test.ts. If you run all test, it will remove hostedZone. Run partial test.
• Get hostedZoneId from localhost:5053/moto-api and set AWS_ROUTE53_HOSTED_ZONE_ID
• Set ROOT_DOMAIN to starchart.com

I wrote #283 to help with this logic.