Potential service outage if no custom domain used in API Gateway

[szilveszter]

Just finished standing up a new service using the apigateway module, but ran into some strange behavior: after I changed something in the Swagger file, my API stopped working. After some investigation I found out that Terraform will delete the stage during the deployment process:

0. Notices Swagger has changed, so the deployment resource has to be replaced
1. Creates new deployment, pointing to the stage
2. As the first step of getting rid of the old deployment resource, it deletes the stage

The reason why this does not happen to our existing services is thanks to mere luck: if there's an existing base path mapping pointing to the stage, the DeleteStage call fails, Terraform ignores that failure, and skips to the DeleteDeployment step.

Fortunately a patch is already available that fixes this behavior, hopefully it'll be merged upstream soon: hashicorp/terraform-provider-aws#3896

Until this gets fixed, the workaround is to set up a custom domain in the module, which is considered best practice anyway. I just wanted to raise awareness, and track the upstream issue.

[ryandub]

Thanks for documenting this @szilveszter - that's a nasty bug.