HTTP Alias Issue left HTTPSCert and CustomDomainAction stuck in UPDATE_FAILED

[benmkramer]

Hello, and thank you for your work on copilot!

I've ben experimenting with running some workloads via copilot, and have run into an issue that I seem stuck on...

For a load balanced web service, I attempted to add another http.alias value. The deploy failed as there was already a dns record for that subdomain created previously. I attempted two things:

1. changing the existing route53 record to a different subdomain
2. deleting the existing route53 record

Neither of those worked, and the only way to get it to redeploy succesfully was removing the new alias.

What I noticed, is that some components of the parent env stack are now stuck at UPDATE_FAILED:

I tried to use the "continue update rollback" feature in cloudformation, which seemed to resolve for the HTTPSCert, but the CustomDomainAction is still continually failing (even though the env seems to be all deployed and in a healthy state).

I see that the custom domain action is failing because it is trying to delete the dns record that it failed to create in the first place, but it doesn't exist. Received response status [FAILED] from custom resource. Message returned: [Tried to delete resource record set [name='subdomain.example.com.', type='A'] but it was not found]

Is there some way around this? Also, is there any particular procedure I should follow for adding an alias where there might be a recently deleted route53 record? Perhaps I need to delete some other dns related resources for that subdomain, wait for the dns ttl, etc.?

[efekarakus]

Hi @benmkramer !

Just so that I'm on the same page, is the service stack's status UPDATE_ROLLBACK_COMPLETE but the environment stack is on UPDATE_ROLLBACK_FAILED? That's surprising I'd have expected both the environment and service rollback to fail.

because it is trying to delete the dns record that it failed to create in the first place, but it doesn't exist. Received response status [FAILED] from custom resource. Message returned: [Tried to delete resource record set [name='subdomain.example.com.', type='A'] but it was not found]

I wonder if we can get the environment stack unstuck by manually creating back the A-record in the hosted zone for subdomain.example.com with a dummy value.
We can try to continue update rollback to see if the CustomDomainAction can remove the record then

[benmkramer]

Thank you for the reply 👍

I had actually tried to delete the originally offending service and recreate it (without the subdomain that caused the original issue, instead using an alternative to test). So currently the service stack is in "CREATE_COMPLETE", and the environment stack is "UPDATE_ROLLBACK_FAILED".

I will try this on later. It is an active subdomain that I would like to avoid much downtime for, so I was originally trying to remove the original dns record (a cname) for that subdomain just before the deploy happened, where copilot / cfn would try and create the new one.

My next steps then, based on your guidance, will be...

1. Replace current dns record for subdomain.example.com with a dummy a record
2. Continue update rollback and see if it can become unstuck

If that works, I can then try re-deploying with subdomain.example.com in the manifest, with the expectation that it may now be able to create the new record.

Do I need to run a copilot env deploy at some point during this? My understanding is that once the rollback completes succesfully, then a subsequent service deploy should handle triggering the dns updates.

[efekarakus]

Do I need to run a copilot env deploy at some point during this? My understanding is that once the rollback completes succesfully, then a subsequent service deploy should handle triggering the dns updates.

Your understanding is correct 👍 You should only need to run copilot svc deploy.

copilot env deploy is useful if there are any modifications in the Environment manifest under copilot/environments/

[efekarakus]

In the mean time, I'll write a synopsis of the issue and action items for the Copilot team here:

Root Cause

1. The hosted zone that copilot tried to insert an A-record to for subdomain.example.com. already contained a CNAME record with the same name.
2. This resulted in the A-record creation to fail because:

   Error occurred
   Bad request.
   (InvalidChangeBatch 400: RRSet of type CNAME with DNS name subdomain.example.com. is not permitted as it conflicts with other records with the same DNS name in zone example.com.)
   

3. Since the Create operation failed for the CustomDomainAction custom resource, it caused it to enter the Delete operation which tries to delete the A-record.
4. 🚨 However, since the A-record was never created now the CustomDomainAction also fails!

Mitigation

For the CustomDomainAction to rollback and delete itself successfully, we need to unfortunately replace the existing CNAME record with an A-record :(.

I'd recommend taking a screenshot/saving the existing record just in case to make it easy to restore the value back in case something goes wrong.

Action item for Copilot

Copilot should check the error message returned from route53, if the error says "the record set is not found"

Received response status [FAILED] from custom resource. Message returned: [Tried to delete resource record set [name='subdomain.example.com.', type='A'] but it was not found]

Then we should just move on. There is no need to fail the custom resource since the record already doesn't exist.

[benmkramer]

It seems that I was able to get past that first issue by creating the dummy A record. It is now trying to delete the certificate though, which is still used by the default subdomain copilot setup, as well as the other (working) subdomain I had configured in the service's manifest.

The message is like this:

Received response status [FAILED] from custom resource. Message returned: Certificate still in use after checking for 10 attempts. (Log: [path]) (RequestId: dfc80fb4-5ff6-4eec-a8a0-76d5a784663c)

As the cert is used for a couple of other domains, I would expect this. It seems the delete on the cert is now retrying a second time. I will watch and share updates if anything changes.

I could maybe just delete the entire service again, and try spinning it up from scratch, but maybe that won't help since this is in the environment stack and not the service stack... Perhaps deleting the service would free up that cert to be cleaned up though. 🤔 Or I could delete this cert myself, but not sure if it will then fail as it can't find the one to delete as experienced with the dns records.

[benmkramer]

One more update -- ALL CLEAR!

After the above, it retried a few times over a 15-20 minute period and then failed. I ran copilot env deploy --force and then it managed to get everything green again. I then deployed one more time with the new domain that originall got me into this predicament, and it went through smoothly.

Looking back, I think the following sequence of actions led to the issue.

1. Added subdomain as alias to service manifest, for which there was already a record created separately for in route53
2. On deploy of that version of the manifest, the stack got into a stuck state trying to create that new subdomain as it already existed
3. In finding no way to work around that, at the time, I tried manually deleting and deploying again, but it seemed stuck. So I deleted the service, and re-created it ( using the copilot cli ).
4. At this point, the environment that originally failed in step 2, was still in a state where it hadn't completed rolling back, so on deploy of the newly-created version of the service, it had a weird mix of newly created things, and things it still wanted to roll-back and delete

Ultimately, the solution for me ended up being:

1. Create a dummy A record for subdomain
2. Run the "contunue roll back" in the stack (which still had failures on the HTTPSCert deletion)
3. Wait for HTTPSCert deletion to fail 3 times and stop in the delete failed state
4. Run copilot env deploy --force for the broken environment (and at end stack should be all green)
5. Re-deploy the service with alias for subdomain
6. Success, service now available at subdomain

Is there any more information I can provide? I am unblocked now, so otherwise, this issue can be closed.

[efekarakus]

Yay! I'm glad everything is back to a COMPLETE state 🎉

Is there any more information I can provide?

It's all good thank you Ben! I'm not clear how copilot env deploy --force managed to take the environment stack that's in UPDATE_ROLLBACK_FAILED to success 🤔 that's the only piece of the puzzle that's not clear to me.

I am unblocked now, so otherwise, this issue can be closed.

I'm going to keep the issue open until we address the action item in this comment (#4401 (comment)) and then close the issue.

Thank you again for your patience and help with troubleshooting!