AWS::ApiGatewayV2::Route and AWS::ApiGatewayV2::Integration resources are deleted from API Gateway upon stack update

[bensie]

Description:

Route and Integration resources that are newly defined are created without issue. However, when certain parts of the template are updated, those resources are deleted from the API Gateway API even though they still exist in the template and are still resources visible to CloudFormation.

The only way to restore them is by removing them from the template (so CloudFormation registers it as a deleted resource), deploy, then add them back to the template and deploy again.

The template in the steps below has an explicitly defined AWS::Serverless::HttpApi resource and a AWS::Serverless::Function whose Events attribute references that API resource. There is an intentionally commented out event that we'll un-comment in the stack update to reproduce the issue.

Further below, there's a AWS::ApiGatewayV2::Route and AWS::ApiGatewayV2::Integration as a sample direct-to-SQS integration. This is the integration and route that disappears upon stack update.

Steps to reproduce:

Create a stack with the following template. It will create without issue and the expected routes will exist.

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Resources:
  Api:
    Type: AWS::Serverless::HttpApi
    Properties:
      StageName: v1
      CorsConfiguration: true
  Func:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: nodejs12.x
      PackageType: Zip
      InlineCode: |
        exports.handler = function(event, context) {
        }
      Handler: index.handler
      MemorySize: 128
      Events:
        HiGetEndpoint:
          Type: HttpApi
          Properties:
            ApiId: !Ref Api
            Path: /hi
            Method: GET
        # HiPostEndpoint:
        #   Type: HttpApi
        #   Properties:
        #     ApiId: !Ref Api
        #     Path: /hi
        #     Method: POST
      Policies:
        - AWSLambdaBasicExecutionRole
        - AWSXrayWriteOnlyAccess
  MyRoute:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      ApiId: !Ref Api
      RouteKey: POST /myroute
      Target: !Sub integrations/${MyRouteIntegration}
      AuthorizationType: NONE
  MyRouteIntegration:
    Type: AWS::ApiGatewayV2::Integration
    Properties:
      ApiId: !Ref Api
      IntegrationType: AWS_PROXY
      IntegrationSubtype: SQS-SendMessage
      PayloadFormatVersion: "1.0"
      CredentialsArn: !GetAtt MyRouteRole.Arn
      RequestParameters:
        QueueUrl: !Ref Queue
        MessageBody: $request.body
  MyRouteRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: apigateway.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: APIGWToSQS
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              Action: sqs:SendMessage
              Effect: Allow
              Resource: !GetAtt Queue.Arn
  Queue:
    Type: AWS::SQS::Queue

Open the API Gateway console and confirm the 2 expected routes exist.

Now, update the above template, un-commenting these 6 commented lines:

        # HiPostEndpoint:
        #   Type: HttpApi
        #   Properties:
        #     ApiId: !Ref Api
        #     Path: /hi
        #     Method: POST

Update the stack with the updated template, then head over to the API Gateway console. The added POST /hi exists but POST /myroute is gone.

Observed result:

POST /myroute is gone from the API and results in a 404 Not Found response when attempting to hit the URL.

Expected result:

Resources that are defined in CloudFormation should not be deleted when they remain in the template and are valid resources listed in the stack.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Tested in CloudFormation stacks in multiple accounts and in us-west-2 and us-east-1. It is environment-agnostic as the template provided can be directly run in the CloudFormation console.

[moelasmar]

I was able to reproduce this issue.

[moelasmar]

this issue does not exist if I tried to create the Stack with the final template, but the issue mainly happen while updating the stack to add the new Post endpoint.

I also can reproduce this issue using the CFN templates.

[bensie]

Yes, that's correct. It's only an issue on updates, where the route/integration items are deleted from the API during the update. This makes for very surprising 404 errors when previously functional items still defined in the template disappear from the API Gateway definition. Thanks for looking into this!

[mims92]

@moelasmar Does that mean it is related to CF rather than SAM?

I am also having the same issue with ALB integration. Some routes are returning 404 during the Stack update.

Is there a workaround?

[moelasmar]

The issue here is a limitation of API Gateway, which does not support combining OpenAPI definitions (like you have in the AWS::Serverless::HttpApi resource) with resources like AWS::ApiGatewayV2::Route. The current behavior is that this might work on stack creation, but the routes may be deleted on stack update. This is intended behavior by API Gateway and will not be changed. The path forward is to move all of your route definitions into your OpenAPI document, or all as route resources.
Please find this documentation for the openApi Document here

[samaybhavsar]

@bensie - Were you able to resolve this ? I am facing a similar you where I have more than 20 lambda functions in SAM Template. If I move the integration to OpenAPI Document, it creates a new API Gateway. Its not possible to have routes in SAM working with the routes in OpenAPI document.

[bensie]

@samaybhavsar I ended up moving all route definitions into OpenAPI. That unfortunately required dropping the direct-to-SQS integration in the example (and just using Lambda => SQS instead), but that allowed moving on with life.