Add support for API Gateway REST API Logs

[pmuens]

What did you implement:

Adds native support for API Gateway logs via the API Gateway stage resource.

Closes #4461

How did you implement it:

Merge in all the resources necessary to setup API Gateway logging via CloudFormation. Set the correct settings in the API Gateway Stage resource.

How can we verify it:

Deploy the following serverless.yml:

service: test

provider:
  name: aws
  runtime: nodejs8.10
  apiGateway:
    logs: true

functions:
  hello:
    handler: handler.hello
    events:
      - http: GET hello

After that curl the endpoint and look into CloudWatch logs for the api-gateway log group.

Todos:

• Write tests
• Write documentation
• Fix linting errors
• Make sure code coverage hasn't dropped
• Provide verification config / commands / resources
• Enable "Allow edits from maintainers" for this PR
• Update the messages below

Is this ready for review?: YES
Is it a breaking change?: NO

[drissamri]

Cool to see native integration of API GW logs are coming soon! Great work.

Should we expect options to customize the format logging afterwards as well? https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html & https://docs.aws.amazon.com/apigateway/api-reference/resource/stage/#format

[dschep]

LGTM & worked as expected

[dschep]

oops. wrong button. approved!

[pmuens]

Thanks @dschep 👍

@drissamri yes, that's a good idea. The initial implementation was planned to be as simple as possible but we can extend this easily later on.

[anjo-swe]

@pmuens is there already a ticket on your backlog to update the implementation to support custom logging?

[gootdude]

@pmuens please expose the following fields for configuration of the log group and log format

• AWS::Logs::LogGroup
  • Properties
    • LogGroupName
    • RetentionInDays
• AWS::ApiGateway::Stage
  • AccessLogSettings
    • Format

Here's our current serverless.yml that we use to create the log group and set retention time as well as the custom format:

resources:
  Resources:
    ABCAccessLogGroup:
      Type: AWS::Logs::LogGroup
      Properties:
        LogGroupName: ABC-${self:provider.stage}
        RetentionInDays: ${file(./config.yml):accessLogRetentionInDays}
    ABCGatewayStage:
      Type: 'AWS::ApiGateway::Stage'
      Properties:
        DeploymentId:
          Ref: __deployment__
        RestApiId:
          Ref: ABCApiGateway
        StageName: ${self:provider.stage}
        TracingEnabled: true
        MethodSettings:
          - HttpMethod: "*"
            ResourcePath: "/*"
            MetricsEnabled: true
            DataTraceEnabled: false
        AccessLogSetting:
          Format: '{"apiId":"$context.apiId","stage":"$context.stage","resourcePath":"$context.resourcePath","requestId":"$context.requestId","awsEndpointRequestId":"$context.awsEndpointRequestId","xrayTraceId":"$context.xrayTraceId","requestTime":"$context.requestTime","requestTimeEpoch":$context.requestTimeEpoch,"httpMethod":"$context.httpMethod","status":"$context.status","path":"$context.path"}'
          DestinationArn:
                Fn::GetAtt:
                  - OneAccessLogGroup
                  - Arn

[pmuens]

@pmuens is there already a ticket on your backlog to update the implementation to support custom logging?

@anjo-swe no, we haven't created such an issue. Feel free to open one! 👍

@gootdude thanks for the feedback. That sounds reasonable. With this iteration we want to implement the most simple way to enable API Gateway logs. Well built upon this implementation in the upcoming versions. Feel free to add your requirements to the API Gateway logs issue (the one @anjo-swe will create).

Thanks again for looking into this and providing feedback 🙏

[pmuens]

@anjo-swe @gootdude I just created #6094 to track proposals for future enhancements.

[davidfarinha]

When can we expect to see this feature released and is there any way we can use this feature in the meantime? Many thanks

[pmuens]

@davidfarinha We've just release the new Serverless Framework version which adds support for this.

Should hit the CDNs in a couple of minutes ⏲

[davidfarinha]

Very nice, great work!

[davidfarinha]

I'm using the following serverless config with version 1.42.1

apiGateway:
  logs: true
  apiKeySourceType: HEADER
  minimumCompressionSize: 1024

And even tried re-deploying however still not getting any logs configured on my API Gateway or can't see any log streams created in CloudWatch. Any ideas?

[sdomagala]

@davidfarinha checking history I found out this feature was changed a bit before a release - 1df8598#diff-7ef39ede04c5da8e81c20fe60e80a9ce

Now, format for logs is:

provider:
  logs:
    restApi: true

and it works for me

[davidfarinha]

Thanks @sdomagala, works for me now.

[ozbillwang]

Any one tested the feature to true first, then to false?

I enable access log with restApi: true, then I change it to false, and deploy again, but the option is still enabled

[ozbillwang]

After enabled the access log, I got the error sometime (2 times), after deploy the change.

CloudFormation - CREATE_FAILED - AWS::Logs::LogGroup - ApiGatewayLogGroup

/aws/api-gateway/serverless-api-dev already exists

I have to manually delete the log group to fix this issue, but that's not right. I will lost old access logs.

[sachinsmc]

After enabled the access log, I got the error sometime (2 times), after deploy the change.

CloudFormation - CREATE_FAILED - AWS::Logs::LogGroup - ApiGatewayLogGroup

/aws/api-gateway/serverless-api-dev already exists

I have to manually delete the log group to fix this issue, but that's not right. I will lost old access logs.

Facing same issue as @ozbillwang , first deployment is fine, second one throws

CloudFormation - CREATE_FAILED - AWS::Logs::LogGroup - ApiGatewayLogGroup

/aws/api-gateway/serverless-api-dev already exists

any workaround?

[toxuin]

After enabled the access log, I got the error sometime (2 times), after deploy the change.

CloudFormation - CREATE_FAILED - AWS::Logs::LogGroup - ApiGatewayLogGroup

/aws/api-gateway/serverless-api-dev already exists

I have to manually delete the log group to fix this issue, but that's not right. I will lost old access logs.

This is happening to us too.

Furthermore, if you set logging to true you get this error on second deploy. If you NOT set logging to anything - it unsets the existing logging settings and stops ALL logging for the API. (See: #6165)

This is pretty bad.

[ozbillwang]

I do recommend to rollback this change now and redo it later

[tommedema]

Where is this feature documented?

[neilh-cogapp]

@tommedema https://www.serverless.com/framework/docs/providers/aws/events/apigateway/#logs