How can I enable logging in API Gateway?

[weixu365]

I want to know the proper way to enable logging in Api Gateway Stage.

AWS CloudFormation support it by using resource type AWS::ApiGateway::Stage, I can define the customized resource, but it requires two parameter "DeploymentId" and "RestApiId" which are dynamically generated in serverless.

Any suggestions on how to enable it?

the CloudFormation doc I'm referencing is http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-stage.html

Thanks

[flomotlik]

@wei-xu-myob Take a look at Custom provider resources: https://github.com/serverless/serverless/blob/master/docs/guide/custom-provider-resources.md#adding-custom-provider-resources

You can overwrite anything we create in the Cloudformation template easily there. I'm going to close this one as its more of a support request and is better suited to our Forum

[nzmkey]

For those who look for a solution here is what I have done:

1. Install the serverless-plugin-stage-variables which will sort out the "DeploymentId" and "RestApiId" for you.
2. Follow custom-provider-resources to add your stage setting.

EG, below is an example of serverless.yml to enable logging:

service: lambda-blah
custom:
  stageVariables:
    env: ${self:provider.stage} 
plugins:
  - serverless-plugin-stage-variables
provider: aws
functions:
  ...
resources:
  Resources:
    ApiGatewayStage:
      Type: AWS::ApiGateway::Stage
      Properties:
        MethodSettings:
          - DataTraceEnabled: true
            HttpMethod: "*"
            LoggingLevel: INFO
            ResourcePath: "/*"
            MetricsEnabled: true

There is a bug in serverless-plugin-stage-variables which will overwrite what custom-provider-resources does. There is a pull request waiting.

[jacob-meacham]

As an alternative to the method described by @nzmkey above, I've created a plugin, https://github.com/jacob-meacham/serverless-plugin-bind-deployment-id, which exposes the randomly generated deployment resource to your custom resources. This lets you write something like this:

PathMapping:
  Type: AWS::ApiGateway::BasePathMapping
  DependsOn: __deployment__
  Properties:
    BasePath: analytics
    DomainName: ${self:provider.domain}
    RestApiId:
      Ref: ApiGatewayRestApi
    Stage: ${self:provider.stage}
__deployment__:
   Properties:
      StageSettings:
          DataTraceEnabled: true
          MetricsEnabled: false
ApiGatewayStage:
  Type: AWS::ApiGateway::Stage
  Properties:
    DeploymentId:
      Ref: __deployment__
    RestApiId:
      Ref: ApiGatewayRestApi
    StageName : ${self:provider.stage}
    MethodSettings:
      - DataTraceEnabled: false
        HttpMethod: "GET"
        LoggingLevel: INFO
        ResourcePath: "/foo"
        MetricsEnabled: false

[rochdev]

@flomotlik After speaking to AWS support center, it seems the correct way to do this is to remove StageName from the deployment and use a separate AWS::ApiGateway::Stage resource instead (which is not compatible with StageName).

Since it's not possible to remove properties using Custom provider resources (to my knowledge), this issue should be reopened.

[rochdev]

I am working on fixing this directly in Serverless. I should have a PR up today or Monday.

[jacob-meacham]

@rochdev You can set the deployment StageName to something like unused, and then create the Stage resource yourself as in my comment above. There will be an extra stage on your API, but it won't do anything and can be safely ignored.

[rochdev]

Yes, but that is a hack caused by a wrong implementation of the stage deployment. The correct way is to forego using StageName completely so you don't have to create a dummy stage. It also removes the need for a plugin to add stage configuration. (great job with your plugin BTW, it's the one we are currently using while this gets resolved :)

[pmuens]

@rochdev thanks for the follow-up regarding your exchange with the AWS team and the PR to resolve this issue! 👍

We'll look into it soon!

[alexcasalboni]

Hey @pmuens, are you planning to implement this as a native feature of the framework? It would be very useful, especially during development.

[pmuens]

Hey @pmuens, are you planning to implement this as a native feature of the framework? It would be very useful, especially during development.

Hey @alexcasalboni thanks for commenting 👍 . Right now there are no plans to get this into core.

Could you estimate how much work / tech-debt such an integration would introduce?

[alexcasalboni]

Based on @nzmkey's solution, it could just be a reusable plugin?

I would expect this to be a config param of API Gateway triggers since you can configure logging on a per-resource fashion too ("Override for this method" on the web Console). Having this option at the provider/service level would just make it more convenient.

Not sure about how much tech-debt this would introduce, but not being able to easily enable logging without manual intervention or custom resources sounds silly to me. If you've ever tried to debug IAM issues, CORS headers, or Auth0 integrations without API Gateway logging, you know what I mean :)

As far as the required work, it's not super trivial, as it requires additional permissions to be granted to API Gateway (you need to set API Gateway's CloudWatch log role arn setting to a role containing the AmazonAPIGatewayPushToCloudWatchLogs policy), and the Stage to be configured after it's created.

I found this guide very useful for manual configuration.

[pmuens]

Based on @nzmkey's solution, it could just be a reusable plugin?

@alexcasalboni that sounds like a good solution for now. This way we still have the possibility to migrate this into core later on.

I would expect this to be a config param of API Gateway triggers since you can configure logging on a per-resource fashion too ("Override for this method" on the web Console). Having this option at the provider/service level would just make it more convenient.

Not sure about how much tech-debt this would introduce, but not being able to easily enable logging without manual intervention or custom resources sounds silly to me. If you've ever tried to debug IAM issues, CORS headers, or Auth0 integrations without API Gateway logging, you know what I mean :)

As far as the required work, it's not super trivial, as it requires additional permissions to be granted to API Gateway (you need to set API Gateway's CloudWatch log role arn setting to a role containing the AmazonAPIGatewayPushToCloudWatchLogs policy), and the Stage to be configured after it's created.

Sounds like the way to go 👍

Maybe we can revisit the implementation (and tech-debt aspect) once it's this is implemented as a plugin to consider if it's worth to move it to core since debugging during API development w/o this feature seems to be pretty hard.

Let us know if you need any help with the plugin and keep us posted here so that we can test-drive this 💪

[alexcasalboni]

@pmuens it seems like @jacob-meacham's solution would be a good starting point for a plugin that would:

• mask the __deployment__ id trick (using this plugin)
• expose a custom StageSettings property that will be mapped to the Stage's MethodSettings configuration (where you can define a list of settings for each method/path, or for all of them)

This would make all of this quite explicit, though.

I was thinking of a simpler way to achieve the same, like a simple LoggingLevel parameter at the provider level. Similarly, we may have a MetricsEnabled parameter. It would be much simpler and it'd just enable logging/metrics globally on the main stage.

This would make sense for the development-phase use case, where global logging always makes sense. Then you can always use @jacob-meacham's manual workaround if you need to customize the stage more granularly.

What do you think?

[sthuber90]

What's the best practice to follow for this use case now?

[yankeeinlondon]

I am fully against "manual configuration" but I'm left quite confused on what to do within Serverless to configure something as basic as logging for API Gateway. @jacob-meacham you're plugin seemed the most straight-forward approach but I must admit that I just couldn't grok it in enough detail to get it to work :(