Serverless Error: InternalFailure with Bitbucket CI/CD

[anuj9196]

This is a Bug Report

Description

I'm using Bitbucket CI/CD to automate deployment. I have the following bitbucket-pipeline.yml content

image: node:11.13.0-alpine

pipelines:
  branches:
    staging:
      - step:
          caches:
            - node
          deployment: staging
          script:
            - apk add python3
            - npm install -g serverless
            - npm install -g @angular/cli
            - npm install
            - serverless config credentials --provider aws --key ${AWS_ACCESS_KEY_ID} --secret ${AWS_SECRET_ACCESS_KEY}
            - serverless deploy --stage ${DEPLOYMENT_STAGE} --no-confirm

and the following is the serverless.yml content

service: myapp
app: myapp
org: org

provider:
  name: aws
  runtime: nodejs10.x
  stage: ${opt:stage, self:custom.variables.stage}
  region: ${opt:region, self:custom.variables.region}
  memorySize: ${opt:memory, self:custom.variables.memorySize}
  timeout: ${opt:timeout, self:custom.variables.timeout}
  logRetentionInDays: 14
  deploymentBucket:
    tags:
      name: ${self:custom.namePrefix}
  deploymentPrefix: ${self:custom.variables.deploymentPrefix}
  versionFunctions: ${self:custom.variables.versionFunctions}
  endpointType: ${self:custom.variables.endpointType}
  tracing:
    apiGateway: ${self:custom.variables.tracing.apiGateway}
    lambda: ${self:custom.variables.tracing.lambda}
  logs:
    restApi:
      accessLogging: ${self:custom.variables.logs.restApi.accessLogging}
      format: 'requestId: $context.requestId'
      executionLogging: ${self:custom.variables.logs.restApi.executionLogging}
      level: ${self:custom.variables.logs.restApi.level}
      fullExecutionData: ${self:custom.variables.logs.restApi.fullExecutionData}
    websocket: ${self:custom.variables.logs.websocket}
    frameworkLambda: ${self:custom.variables.logs.frameworkLambda}

packages:
  exclude:
    - .git/**
    - env.yml
    - readme.md

plugins:
  - fullstack-serverless
  - serverless-offline

custom:
  variables:
    stage: ${env:DEPLOYMENT_STAGE, 'dev'}
    region: ${env:AWS_REGION, 'us-west-2'}
    memorySize: ${env:MEMORY_SIZE, '512'}
    timeout: ${env:TIMEOUT, '30'}
    deploymentPrefix: ${env:DEPLOYMENT_PREFIX, 'serverless'}
    versionFunctions: false # ${env:VERSION_FUNCTIONS, 'false'}
    endpointType: ${env:ENDPOINT_TYPE, 'Edge'}
    tracing:
      apiGateway: true # ${env:TRACING_API_GATEWAY, 'true'}
      lambda: true # ${env:TRACING_LAMBDA, 'true'}
    logs:
      restApi:
        accessLogging: true # ${env:LOGS_ACCESS_LOGGING, 'true'}
        executionLogging: true # ${env:LOGS_EXECUTION_LOGGING, 'true'}
        level: ${env:LOGS_LEVEL, 'INFO'}
        fullExecutionData: true # ${env:LOGS_FULL_EXECUTION_DATA, 'true'}
      websocket: true # ${env:LOGS_WEBSOCKET, 'true'}
      frameworkLambda: true # ${env:LOGS_FRAMEWORK_LAMBDA, 'true'}

  namePrefix: ${self:provider.stage}-${self:provider.region}
  fullstack:
    bucketName: serverless-${self:custom.namePrefix}-frontend
    distributionFolder: webapp/dist/webapp
    singlePageApp: true
    clientCommand: npm install && ng build --aot --prod
    clientSrcPath: webapp

functions:
  hello:
    handler: handler.server
    description: Node Express application handler # Description of the function
    events: # events trigger lambda functions
      - http: # this is an API Gateway HTTP event trigger
          path: /
          method: ANY
          cors: truehttp://slss.io/sg-require-cfn-role

      - http: # all routes get proxied to the Express router
          path: /{proxy+}
          method: ANY
          cors: true

• What went wrong?
  The deployment of AWS services was successful but pipeline failed.

• What did you expect should have happened?
  The deployment should be a success.

• What stacktrace or error message from your provider did you see?

Service Information
service: myapp
stage: qa
region: us-west-2
stack: myapp-qa
resources: 25
api keys:
  None
endpoints:
  ANY - https://as8d7adf7.execute-api.us-west-2.amazonaws.com/qa/
  ANY - https://as8d7adf7.execute-api.us-west-2.amazonaws.com/qa/{proxy+}
functions:
  hello: myapp-qa-hello
layers:
  None
CloudFront domain name
  aas3234s.cloudfront.net (CNAME: -)
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 1 of 4
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 2 of 4
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 3 of 4
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 4 of 4
Serverless: Publishing service to the Serverless Dashboard...
Serverless: Successfully published your service to the Serverless Dashboard: https://dashboard.serverless.com/tenants/org/applications/myapp/services/myapp/stage/qa/region/us-west-2
 
  Serverless Error ---------------------------------------
 
  InternalFailure
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              11.13.0
     Framework Version:         1.52.0
     Plugin Version:            2.0.0
     SDK Version:               2.1.1

Similar or dependent issues:

Additional Data

• Serverless Framework Version you're using: 1.52.0
• Operating System: linux
• Stack Trace:
• Provider Error messages: InternalFailure

[drexler]

@anuj9196 mind publishing with SLS_DEBUG enabled in order to get a more detailed output of what's causing the issue?
Command:
SLS_DEBUG=* serverless deploy

[anuj9196]

@drexler Here is the log.

   <Buffer 66 75 6e 63 74 69 6f 6e 20 5f 64 65 66 69 6e 65 50 72 6f 70 65 72 74 69 65 73 28 74 2c 65 29 7b 66 6f 72 28 76 61 72 20 6e 3d 30 3b 6e 3c 65 2e 6c 65 ... >,
  ContentType: 'application/javascript' })
Serverless: [AWS s3 200 0.331s 0 retries] putObject({ Bucket: 'myapp-qa-serverless-qa-us-west-2-frontend',
  Key: 'runtime-es5.85f895af57b038f1e5b4.js',
  Body:
   <Buffer 21 66 75 6e 63 74 69 6f 6e 28 65 29 7b 66 75 6e 63 74 69 6f 6e 20 72 28 72 29 7b 66 6f 72 28 76 61 72 20 6e 2c 6c 2c 66 3d 72 5b 30 5d 2c 69 3d 72 5b ... >,
  ContentType: 'application/javascript' })
Serverless: [AWS s3 200 0.318s 0 retries] putObject({ Bucket: 'myapp-qa-serverless-qa-us-west-2-frontend',
  Key: 'styles.09e2c710755c8867a460.css',
  Body: <Buffer >,
  ContentType: 'text/css' })
Serverless: Success! Client deployed.
Serverless: Invoke aws:info
Serverless: [AWS cloudformation 200 0.398s 0 retries] describeStacks({ StackName: 'myapp-qa' })
Serverless: [AWS cloudformation 200 0.346s 0 retries] listStackResources({ StackName: 'myapp-qa' })
Service Information
service: myapp
stage: qa
region: us-west-2
stack: myapp-qa
resources: 25
api keys:
  None
endpoints:
  ANY - https://as8d7adf7.execute-api.us-west-2.amazonaws.com/qa/
  ANY - https://as8d7adf7.execute-api.us-west-2.amazonaws.com/qa/{proxy+}
functions:
  hello: myapp-qa-hello
layers:
  None
CloudFront domain name
  aas3232sd.cloudfront.net (CNAME: -)
Serverless: [AWS sts 200 0.024s 0 retries] getCallerIdentity({})
Serverless: [AWS apigateway 200 0.396s 0 retries] getRestApis({ position: undefined, limit: 500 })
Serverless: [AWS apigateway 200 0.358s 0 retries] getStage({ restApiId: 'as8d7adf7', stageName: 'qa' })
Serverless: [AWS apigateway 500 3.297s 3 retries] updateStage({ restApiId: 'as8d7adf7',
  stageName: 'qa',
  patchOperations:
   [ { op: 'replace', path: '/tracingEnabled', value: 'true' },
     { op: 'replace',
       path: '/accessLogSettings/destinationArn',
       value:
        'arn:aws:logs:us-west-2:881210447458:log-group:/aws/api-gateway/myapp-qa' },
     { op: 'replace',
       path: '/accessLogSettings/format',
       value:
        '{"requestTime":"$context.requestTime","requestId":"$context.requestId","apiId":"$context.apiId","resourceId":"$context.resourceId","resourcePath":"$context.resourcePath","path":"$context.path","httpMethod":"$context.httpMethod","status":"$context.status","authLatency":"$context.authorizer.integrationLatency","integrationLatency":"$context.integrationLatency","integrationStatus":"$context.integrationStatus","responseLatency":"$context.responseLatency","responseLength":"$context.responseLength","errorMessage":"$context.error.message","format":"SLS_ACCESS_LOG","version":"1.0.0"}' },
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'true' },
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'INFO' },
     [length]: 5 ] })
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 1 of 4
Serverless: [AWS apigateway 500 2.889s 3 retries] updateStage({ restApiId: 'as8d7adf7',
  stageName: 'qa',
  patchOperations:
   [ { op: 'replace', path: '/tracingEnabled', value: 'true' },
     { op: 'replace',
       path: '/accessLogSettings/destinationArn',
       value:
        'arn:aws:logs:us-west-2:881210447458:log-group:/aws/api-gateway/myapp-qa' },
     { op: 'replace',
       path: '/accessLogSettings/format',
       value:
        '{"requestTime":"$context.requestTime","requestId":"$context.requestId","apiId":"$context.apiId","resourceId":"$context.resourceId","resourcePath":"$context.resourcePath","path":"$context.path","httpMethod":"$context.httpMethod","status":"$context.status","authLatency":"$context.authorizer.integrationLatency","integrationLatency":"$context.integrationLatency","integrationStatus":"$context.integrationStatus","responseLatency":"$context.responseLatency","responseLength":"$context.responseLength","errorMessage":"$context.error.message","format":"SLS_ACCESS_LOG","version":"1.0.0"}' },
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'true' },
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'INFO' },
     [length]: 5 ] })
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 2 of 4
Serverless: [AWS apigateway 500 3.147s 3 retries] updateStage({ restApiId: 'as8d7adf7',
  stageName: 'qa',
  patchOperations:
   [ { op: 'replace', path: '/tracingEnabled', value: 'true' },
     { op: 'replace',
       path: '/accessLogSettings/destinationArn',
       value:
        'arn:aws:logs:us-west-2:881210447458:log-group:/aws/api-gateway/myapp-qa' },
     { op: 'replace',
       path: '/accessLogSettings/format',
       value:
        '{"requestTime":"$context.requestTime","requestId":"$context.requestId","apiId":"$context.apiId","resourceId":"$context.resourceId","resourcePath":"$context.resourcePath","path":"$context.path","httpMethod":"$context.httpMethod","status":"$context.status","authLatency":"$context.authorizer.integrationLatency","integrationLatency":"$context.integrationLatency","integrationStatus":"$context.integrationStatus","responseLatency":"$context.responseLatency","responseLength":"$context.responseLength","errorMessage":"$context.error.message","format":"SLS_ACCESS_LOG","version":"1.0.0"}' },
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'true' },
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'INFO' },
     [length]: 5 ] })
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 3 of 4
Serverless: [AWS apigateway 500 3.388s 3 retries] updateStage({ restApiId: 'as8d7adf7',
  stageName: 'qa',
  patchOperations:
   [ { op: 'replace', path: '/tracingEnabled', value: 'true' },
     { op: 'replace',
       path: '/accessLogSettings/destinationArn',
       value:
        'arn:aws:logs:us-west-2:881210447458:log-group:/aws/api-gateway/myapp-qa' },
     { op: 'replace',
       path: '/accessLogSettings/format',
       value:
        '{"requestTime":"$context.requestTime","requestId":"$context.requestId","apiId":"$context.apiId","resourceId":"$context.resourceId","resourcePath":"$context.resourcePath","path":"$context.path","httpMethod":"$context.httpMethod","status":"$context.status","authLatency":"$context.authorizer.integrationLatency","integrationLatency":"$context.integrationLatency","integrationStatus":"$context.integrationStatus","responseLatency":"$context.responseLatency","responseLength":"$context.responseLength","errorMessage":"$context.error.message","format":"SLS_ACCESS_LOG","version":"1.0.0"}' },
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'true' },
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'INFO' },
     [length]: 5 ] })
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 4 of 4
Serverless: [AWS apigateway 500 2.97s 3 retries] updateStage({ restApiId: 'as8d7adf7',
  stageName: 'qa',
  patchOperations:
   [ { op: 'replace', path: '/tracingEnabled', value: 'true' },
     { op: 'replace',
       path: '/accessLogSettings/destinationArn',
       value:
        'arn:aws:logs:us-west-2:881210447458:log-group:/aws/api-gateway/myapp-qa' },
     { op: 'replace',
       path: '/accessLogSettings/format',
       value:
        '{"requestTime":"$context.requestTime","requestId":"$context.requestId","apiId":"$context.apiId","resourceId":"$context.resourceId","resourcePath":"$context.resourcePath","path":"$context.path","httpMethod":"$context.httpMethod","status":"$context.status","authLatency":"$context.authorizer.integrationLatency","integrationLatency":"$context.integrationLatency","integrationStatus":"$context.integrationStatus","responseLatency":"$context.responseLatency","responseLength":"$context.responseLength","errorMessage":"$context.error.message","format":"SLS_ACCESS_LOG","version":"1.0.0"}' },
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'true' },
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'INFO' },
     [length]: 5 ] })
Serverless: Publishing service to the Serverless Dashboard...
Serverless: [AWS sts 200 0.02s 0 retries] getCallerIdentity({})
Serverless: [AWS cloudformation 200 0.411s 0 retries] describeStacks({ StackName: 'myapp-qa' })
platform-sdk fetching: POST https://api.serverless.com/core/tenants/org/applications/myapp/services/myapp/stages/qa/regions/us-west-2/deployments
Serverless: Successfully published your service to the Serverless Dashboard: https://dashboard.serverless.com/tenants/org/applications/myapp/services/myapp/stage/qa/region/us-west-2
 
  Serverless Error ---------------------------------------
 
  ServerlessError: InternalFailure
      at promise.catch.err (/usr/local/lib/node_modules/serverless/lib/plugins/aws/provider/awsProvider.js:317:27)
      at process._tickCallback (internal/process/next_tick.js:68:7)
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              10.16.3
     Framework Version:         1.52.0
     Plugin Version:            2.0.0
     SDK Version:               2.1.1

Do I need to share full log?

[mcshiz]

I am receiving the same error.
I think it has to do with enabling x-ray and API Gateway.
Whats really odd is that it is only doing it on one AWS account and not the other.
Using AWS CodePipeline/CodeBuild created by the same cloudformation template.

EDIT: See the bottom snippet from the other account where the deployment works sucessfully.

EDIT 2:

Once I turned off tracing for apiGateway it completed successfully again.
see snippet 3 for logs.

tracing:
    lambda: ${self:custom.xrayEnabled.${self:provider.stage}}
    # apiGateway: ${self:custom.xrayEnabled.${self:provider.stage}}

Serverless: [AWS sts 200 0.32s 0 retries] getCallerIdentity({}) 
Serverless: [AWS apigateway 200 0.069s 0 retries] getRestApis({ position: undefined, limit: 500 }) 
Serverless: [AWS apigateway 200 0.06s 0 retries] getStage({ restApiId: 'gz8p6jc4l5', stageName: 'prod' }) 
Serverless: [AWS apigateway 500 1.974s 3 retries] updateStage({ restApiId: 'gz8p6jc4l5', 
  stageName: 'prod', 
  patchOperations:  
   [ { op: 'replace', path: '/tracingEnabled', value: 'true' }, 
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'false' }, 
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'OFF' }, 
     [length]: 3 ] }) 
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 1 of 4 
Serverless: [AWS apigateway 500 2.202s 3 retries] updateStage({ restApiId: 'gz8p6jc4l5', 
  stageName: 'prod', 
  patchOperations:  
   [ { op: 'replace', path: '/tracingEnabled', value: 'true' }, 
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'false' }, 
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'OFF' }, 
     [length]: 3 ] }) 
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 2 of 4 
Serverless: [AWS apigateway 500 1.879s 3 retries] updateStage({ restApiId: 'gz8p6jc4l5', 
  stageName: 'prod', 
  patchOperations:  
   [ { op: 'replace', path: '/tracingEnabled', value: 'true' }, 
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'false' }, 
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'OFF' }, 
     [length]: 3 ] }) 
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 3 of 4 
Serverless: [AWS apigateway 500 2.053s 3 retries] updateStage({ restApiId: 'gz8p6jc4l5', 
  stageName: 'prod', 
  patchOperations:  
   [ { op: 'replace', path: '/tracingEnabled', value: 'true' }, 
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'false' }, 
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'OFF' }, 
     [length]: 3 ] }) 
Serverless: Recoverable error occurred (InternalFailure), sleeping for 5 seconds. Try 4 of 4 
Serverless: [AWS apigateway 500 1.711s 3 retries] updateStage({ restApiId: 'gz8p6jc4l5', 
  stageName: 'prod', 
  patchOperations:  
   [ { op: 'replace', path: '/tracingEnabled', value: 'true' }, 
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'false' }, 
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'OFF' }, 
     [length]: 3 ] }) 
  
  Serverless Error --------------------------------------- 
  
  ServerlessError: InternalFailure 
    at promise.catch.err (/codebuild/output/src809956651/src/back-end/node_modules/serverless/lib/plugins/aws/provider/awsProvider.js:316:27) 
    at <anonymous> 
    at process._tickDomainCallback (internal/process/next_tick.js:229:7) 
  
  Get Support -------------------------------------------- 
     Docs:          docs.serverless.com 
     Bugs:          github.com/serverless/serverless/issues 
     Issues:        forum.serverless.com 
  
  Your Environment Information --------------------------- 
     Operating System:          linux 
     Node Version:              8.16.0 
     Framework Version:         1.50.1 
     Plugin Version:            1.3.11 
     SDK Version:               2.1.1 

Success

Serverless: [AWS sts 200 0.316s 0 retries] getCallerIdentity({}) 
Serverless: [AWS apigateway 200 0.086s 0 retries] getRestApis({ position: undefined, limit: 500 }) 
Serverless: [AWS apigateway 200 0.042s 0 retries] getStage({ restApiId: '07bzrnhwgi', stageName: 'preprod' }) 
Serverless: [AWS apigateway 200 0.227s 0 retries] updateStage({ restApiId: '07bzrnhwgi', 
  stageName: 'preprod', 
  patchOperations:  
   [ { op: 'replace', path: '/tracingEnabled', value: 'true' }, 
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'false' }, 
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'OFF' }, 
     [length]: 3 ] }) 

APIGateway Tracing turned off

Serverless: [AWS sts 200 0.3s 0 retries] getCallerIdentity({}) 
Serverless: [AWS apigateway 200 0.06s 0 retries] getRestApis({ position: undefined, limit: 500 }) 
Serverless: [AWS apigateway 200 0.045s 0 retries] getStage({ restApiId: 'gz8p6jc4l5', stageName: 'prod' }) 
Serverless: [AWS apigateway 200 0.258s 0 retries] updateStage({ restApiId: 'gz8p6jc4l5', 
  stageName: 'prod', 
  patchOperations:  
   [ { op: 'replace', path: '/tracingEnabled', value: 'false' }, 
     { op: 'replace', path: '/*/*/logging/dataTrace', value: 'false' }, 
     { op: 'replace', path: '/*/*/logging/loglevel', value: 'OFF' }, 
     [length]: 3 ] }) 

[anuj9196]

@mcshiz This worked for me as well after disabling APIGateway Tracing