ApiGatewayBasePathMapping Mixing of REST APIs and HTTP APIs on the same domain name

[paulcruse-syn]

Bug Report

Description

Apigateway base path mapping attachment throws error.

1. What did you do?
   I ran this command serverless deploy --stage uat --aws-profile uat --aws_envs cloud --region us-east-2

2. What happened?
   Apigateway errors out and forces whole stack into auto deletion state; deployment fails.

3. What should've happened?
   The app should've deployed and the base path mapping should've been properly set.

4. What's the content of your serverless.yml file?

app: agriedge
service: ae
package:
    individually: true
    exclude:
        - .*
        - .**/**
        - aws/**
        - docs/**
        - test/**
        - node_modules/aws-sdk/**
        - README.md
        - buildspec.yml
        - serverless.yml
        - package.json
        - package-lock.json
custom:
    security_group: ${file(aws/resources/security-group-rules.yml):groups}
    db_instance_size:
        local: db.t2.small
        dev: db.t2.small
        uat: db.m4.large
        qa: db.m4.large
        prod: db.m4.large
    cors:
        origin: "*"
        allowCredentials: false
        headers:
            - Authorization
            - Content-Type
            - userID
            - web
            - token
    serverless-offline:
        port: 9000
        stage: local
        prefix: ${self:service}
        region: us-east-2
        noAuth: true
provider:
    name: aws
    runtime: nodejs10.x
    logRetentionInDays: 14
    versionFunctions: false
    environment:  ${file(./aws/envs/${opt:aws_envs, 'local'}.yml):environment}
    stage: ${opt:stage, 'local'}
    region: ${opt:region, 'us-east-2'}
    profile: ${opt:stage, 'local'}
    stackName: ${self:provider.stackTags.name}
    apiName: ${self:provider.stackTags.name}
    endpointType: regional
    deploymentPrefix: ${self:provider.stackTags.name}
    deploymentBucket:
        name: '${self:provider.stage}-syngenta-deployments'
    stackTags:
        name: ${self:provider.stage}-${self:app}-${self:service}
        service: ${self:service}
        environment: ${self:provider.stage}
        managedby: serverless
    iamRoleStatements:
        - '${file(aws/iamroles/ssm.yml)}'
resources:
    - '${file(aws/resources/apigateway.yml)}'
    - '${file(aws/resources/rds-mysql.yml)}'
    - '${file(aws/resources/security-group.yml)}'
    - '${file(aws/resources/vpc-rds.yml)}'
functions:
    v1-database-versioner:
        name: ${self:provider.stackTags.name}-v1-database-versioner
        description: Database versioner script
        handler: application/db-versioner.applyVersion
        memorySize: 3008
        timeout: 900
    v1-apigateway-handler:
        name: ${self:provider.stackTags.name}-v1-apigateway-handler
        description: API Router handler
        handler: application/lambda.run
        memorySize: 512
        timeout: 29
        events:
            - http:
                path: /
                method: ANY
                cors: ${self:custom.cors}
            - http:
                path: /{proxy+}
                method: ANY
                cors: ${self:custom.cors}
plugins:
    - serverless-plugin-bind-deployment-id
    - serverless-offline

Here is the ApiGateway cloudformation template:

Resources:
    ApiGatewayStage:
        Type: 'AWS::ApiGateway::Stage'
        Properties:
            StageName: ${self:provider.stage}
            DeploymentId:
                Ref: __deployment__
            RestApiId:
                Ref: ApiGatewayRestApi
            MethodSettings:
                -
                    ResourcePath: /*
                    HttpMethod: '*'
                    LoggingLevel: INFO
                    DataTraceEnabled: true
                    MetricsEnabled: true
    ApiGatewayAppBasePathMapping:
        Type: 'AWS::ApiGateway::BasePathMapping'
        Properties:
            BasePath: ${self:service}
            DomainName: '${self:provider.stage}-api-agriedge-lb.syndpe.com'
            RestApiId:
                Ref: ApiGatewayRestApi
            Stage:
                Ref: ApiGatewayStage

1. What's the output you get when you use the SLS_DEBUG=* environment variable (e.g. SLS_DEBUG=* serverless deploy)

Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command offline
Serverless: Load command offline:start
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: [AWS sts 200 0.249s 0 retries] assumeRole({
  RoleArn: 'arn:aws:iam::052908401056:role/*********',
  RoleSessionName: 'aws-sdk-js-1584056092473',
  SerialNumber: 'arn:aws:iam::169145965798:mfa/***************************',
  TokenCode: '738410'
})
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage
Serverless: Invoke aws:common:validate
Serverless: [AWS s3 200 0.156s 0 retries] getBucketLocation({ Bucket: 'uat-syngenta-deployments' })
Serverless: Invoke aws:deploy:deploy
Serverless: [AWS cloudformation 200 0.19s 0 retries] describeStacks({ StackName: 'uat-agriedge-ae' })
Serverless: [AWS s3 200 0.137s 0 retries] listObjectsV2({
  Bucket: 'uat-syngenta-deployments',
  Prefix: 'uat-agriedge-ae/ae/uat'
})
Serverless: [AWS s3 200 0.145s 0 retries] headObject({
  Bucket: 'uat-syngenta-deployments',
  Key: 'uat-agriedge-ae/ae/uat/1584056058027-2020-03-12T23:34:18.027Z/v1-database-versioner.zip'
})
Serverless: [AWS s3 200 0.146s 0 retries] headObject({
  Bucket: 'uat-syngenta-deployments',
  Key: 'uat-agriedge-ae/ae/uat/1584056058027-2020-03-12T23:34:18.027Z/v1-apigateway-handler.zip'
})
Serverless: [AWS s3 200 0.14s 0 retries] headObject({
  Bucket: 'uat-syngenta-deployments',
  Key: 'uat-agriedge-ae/ae/uat/1584056058027-2020-03-12T23:34:18.027Z/compiled-cloudformation-template.json'
})
Serverless: [AWS lambda 200 0.198s 0 retries] getFunction({ FunctionName: 'uat-agriedge-ae-v1-database-versioner' })
Serverless: [AWS lambda 200 0.109s 0 retries] getFunction({ FunctionName: 'uat-agriedge-ae-v1-apigateway-handler' })
Serverless: [AWS sts 200 0.212s 0 retries] getCallerIdentity({})
Serverless: Uploading CloudFormation file to S3...
Serverless: [AWS s3 200 0.215s 0 retries] putObject({
  Body: <Buffer 7b 22 41 57 53 54 65 6d 70 6c 61 74 65 46 6f 72 6d 61 74 56 65 72 73 69 6f 6e 22 3a 22 32 30 31 30 2d 30 39 2d 30 39 22 2c 22 44 65 73 63 72 69 70 74 ... 11124 more bytes>,
  Bucket: 'uat-syngenta-deployments',
  Key: 'uat-agriedge-ae/ae/uat/1584056107902-2020-03-12T23:35:07.902Z/compiled-cloudformation-template.json',
  ContentType: 'application/json',
  Metadata: { filesha256: '+yGzZR06glBhwi8eZQcdqdacgqmtNOqP9lIroUnoDkc=' }
})
Serverless: Uploading artifacts...
Serverless: Uploading service v1-database-versioner.zip file to S3 (3.28 MB)...
Serverless: Uploading service v1-apigateway-handler.zip file to S3 (3.28 MB)...
Serverless: [AWS s3 200 1.19s 0 retries] putObject({
  Body: <Buffer 50 4b 03 04 14 00 08 00 08 00 00 00 21 00 00 00 00 00 00 00 00 00 00 00 00 00 21 00 00 00 61 70 70 6c 69 63 61 74 69 6f 6e 2f 63 6f 6e 66 69 67 2f 64 ... 3440390 more bytes>,
  Bucket: 'uat-syngenta-deployments',
  Key: 'uat-agriedge-ae/ae/uat/1584056107902-2020-03-12T23:35:07.902Z/v1-apigateway-handler.zip',
  ContentType: 'application/zip',
  Metadata: { filesha256: 'Ps0PZMAVqpRV691Fyxm7+xgNXaDkbQGzGGiNzpKwLNw=' }
})
Serverless: [AWS s3 200 6.543s 0 retries] putObject({
  Body: <Buffer 50 4b 03 04 14 00 08 00 08 00 00 00 21 00 00 00 00 00 00 00 00 00 00 00 00 00 21 00 00 00 61 70 70 6c 69 63 61 74 69 6f 6e 2f 63 6f 6e 66 69 67 2f 64 ... 3440390 more bytes>,
  Bucket: 'uat-syngenta-deployments',
  Key: 'uat-agriedge-ae/ae/uat/1584056107902-2020-03-12T23:35:07.902Z/v1-database-versioner.zip',
  ContentType: 'application/zip',
  Metadata: { filesha256: 'Ps0PZMAVqpRV691Fyxm7+xgNXaDkbQGzGGiNzpKwLNw=' }
})
Serverless: Validating template...
Serverless: [AWS cloudformation 200 0.598s 0 retries] validateTemplate({
  TemplateURL: 'https://s3.amazonaws.com/uat-syngenta-deployments/uat-agriedge-ae/ae/uat/1584056107902-2020-03-12T23:35:07.902Z/compiled-cloudformation-template.json'
})
Serverless: Updating Stack...
Serverless: [AWS cloudformation 200 0.596s 0 retries] updateStack({
  StackName: 'uat-agriedge-ae',
  Capabilities: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', [length]: 2 ],
  Parameters: [ [length]: 0 ],
  TemplateURL: 'https://s3.amazonaws.com/uat-syngenta-deployments/uat-agriedge-ae/ae/uat/1584056107902-2020-03-12T23:35:07.902Z/compiled-cloudformation-template.json',
  Tags: [
    { Key: 'STAGE', Value: 'uat' },
    { Key: 'name', Value: 'uat-agriedge-ae' },
    { Key: 'service', Value: 'ae' },
    { Key: 'environment', Value: 'uat' },
    { Key: 'managedby', Value: 'serverless' },
    [length]: 5
  ]
})
Serverless: Checking Stack update progress...
Serverless: [AWS cloudformation 200 0.31s 0 retries] describeStackEvents({
  StackName: 'arn:aws:cloudformation:us-east-2:052908401056:stack/uat-agriedge-ae/01b09050-64b8-11ea-9c94-025084048856'
})
.....Serverless: [AWS cloudformation 200 0.316s 0 retries] describeStackEvents({
  StackName: 'arn:aws:cloudformation:us-east-2:052908401056:stack/uat-agriedge-ae/01b09050-64b8-11ea-9c94-025084048856'
})
......Serverless: [AWS cloudformation 200 0.793s 0 retries] describeStackEvents({
  StackName: 'arn:aws:cloudformation:us-east-2:052908401056:stack/uat-agriedge-ae/01b09050-64b8-11ea-9c94-025084048856'
})
...
Serverless: Operation failed!
Serverless: View the full error output: https://us-east-2.console.aws.amazon.com/cloudformation/home?region=us-east-2#/stack/detail?stackId=arn%3Aaws%3Acloudformation%3Aus-east-2%3A052908401056%3Astack%2Fuat-agriedge-ae%2F01b09050-64b8-11ea-9c94-025084048856

  Serverless Error ---------------------------------------

  ServerlessError: An error occurred: ApiGatewayAppBasePathMapping - Mixing of REST APIs and HTTP APIs on the same domain name can only be accomplished through API Gateway's V2 DomainName interface. Currently, WebSocket APIs can only be attached to a domain name with other WebSocket APIs. This must also occur through API Gateway's V2 DomainName interface. (Service: AmazonApiGateway; Status Code: 400; Error Code: BadRequestException; Request ID: 61e4032c-780c-4c3d-adda-64e87972e253).
      at /Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:125:33
  From previous event:
      at AwsDeploy.monitorStack (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:28:12)
      at /Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:107:28
  From previous event:
      at AwsDeploy.update (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:107:8)
  From previous event:
      at AwsDeploy.<anonymous> (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:121:35)
  From previous event:
      at AwsDeploy.updateStack (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:117:33)
  From previous event:
      at AwsDeploy.<anonymous> (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/plugins/aws/deploy/index.js:127:39)
  From previous event:
      at Object.aws:deploy:deploy:updateStack [as hook] (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/plugins/aws/deploy/index.js:123:30)
      at /Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/classes/PluginManager.js:490:55
  From previous event:
      at PluginManager.invoke (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/classes/PluginManager.js:490:22)
      at PluginManager.spawn (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/classes/PluginManager.js:510:17)
      at AwsDeploy.<anonymous> (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/plugins/aws/deploy/index.js:93:48)
  From previous event:
      at Object.deploy:deploy [as hook] (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/plugins/aws/deploy/index.js:89:30)
      at /Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/classes/PluginManager.js:490:55
  From previous event:
      at PluginManager.invoke (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/classes/PluginManager.js:490:22)
      at /Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/classes/PluginManager.js:525:24
  From previous event:
      at PluginManager.run (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/classes/PluginManager.js:525:8)
      at /Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/Serverless.js:133:33
      at processImmediate (internal/timers.js:456:21)
      at process.topLevelDomainCallback (domain.js:137:15)
  From previous event:
      at Serverless.run (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/lib/Serverless.js:120:74)
      at /Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/bin/serverless.js:82:30
      at /Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/graceful-fs/graceful-fs.js:136:16
      at /Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/graceful-fs/graceful-fs.js:57:14
      at FSReqCallback.oncomplete (fs.js:154:23)
  From previous event:
      at /Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/bin/serverless.js:82:8
      at processImmediate (internal/timers.js:456:21)
      at process.topLevelDomainCallback (domain.js:137:15)
  From previous event:
      at Object.<anonymous> (/Users/paul.cruse/Developer/syngenta/api-node-agriedge/node_modules/serverless/bin/serverless.js:71:4)
      at Module._compile (internal/modules/cjs/loader.js:1158:30)
      at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)
      at Module.load (internal/modules/cjs/loader.js:1002:32)
      at Function.Module._load (internal/modules/cjs/loader.js:901:14)
      at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:74:12)
      at internal/main/run_main_module.js:18:47

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              12.16.1
     Framework Version:         1.65.0
     Plugin Version:            3.5.0
     SDK Version:               2.3.0
     Components Version:        2.22.3

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! api-node-agriedge@1.0.0 deploy: `SLS_DEBUG=* serverless deploy --stage uat --aws-profile uat --aws_envs cloud --region us-east-2`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the api-node-agriedge@1.0.0 deploy script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/paul.cruse/.npm/_logs/2020-03-12T23_35_33_551Z-debug.log

[paulcruse-syn]

I was able to figure this out.

AWS has just released a new API Gateway V2 Resource Type Reference, see:

If you manually create a custom domain in API Gateway using the AWS web console, you will be opting into using that domain name as V2 resource.

Therefore none of your resources using the old API Gateway Cloudformation Templates will work.

The ONLY way to fix this is by creating the domain in some programmatic fashion, like using the AWS CLI, or to use a cloudformation template for the domain; be advised, building the custom domain through a cloud formation template will tie that domain name to the cloudformation stack that built it. No other service will be able to tie to that domain created by that cloudformation template.

This is not ideal for distributed systems; my suggestion is to use the cli instead if you are build miscroservices/other distributed systems pattern. Using the CLI/SDK/etc will allow other services to tie themselves to the that custom domain without issue.

[zainzafar]

Where are we on this? I'm still getting this error. Any official fixes in pipeline?

[paulcruse-syn]

@zainzafar to my understanding there is no real fix for serverless to do. The fix is on you, because AWS upgraded the apigateway web console.

You have three options:

• delete the custom domain you made through the console and create the custom domain via the cloudformation
• delete the custom domain you made through the console and create the custom domain via the aws cli (or other SDKs/CDKs); example
• upgrade your cloudformation apigateway templates to use V2

Hope that helps,

[zainzafar]

@paulcruse-syn For now, I've added the domain using the aws cli. I'm wondering if, for the last option related to CloudFormation template, it falls under the scope of serverless-domain-manager since that is what I'm using to manage custom domains?

[paulcruse-syn]

@zainzafar lol yea that is a good question, never used that so I wouldn't know. but glad to see you found a workable solution for now

[sthepatel]

@paulcruse-syn helped a lot, thanks!!

For anyone else who might need to do this - here's a quick CLI command:
aws apigateway create-domain-name --domain-name '{{domain-name}}' --regional-certificate-arn '{{cerificate-manager-arn}}' --security-policy 'TLS_1_2' --endpoint-configuration 'types=REGIONAL' --profile {{profile-name}}

Just replace the {{....}}