A Serverless application that parses custom metrics from CloudWatch Logs and sends them to CloudWatch as custom metrics.
This application deploys a single Lambda function with a name prefixed with
serverlessrepo-async-custom-metrics. This function supports both
CloudWatch Logs as well as
Kinesis as event source.
You can subscribe the function to CloudWatch log groups directly. But since you can only have one subscription filter per log group and you probably want to ship your logs elsewhere (maybe to an ELK stack?), most likely you'll subscribe log groups to a Kinesis stream first, then subscribe this
serverlessrepo-async-custom-metrics function to the Kinesis stream.
To help you manage the subscription of your logs, consider using this Serverless application.
Once subscribed, you can record custom metrics by writing to
stdout. The function would parse the custom metrics out of your logs and send them to CloudWatch as metrics.
The format of the custom metric needs to follow the convention:
metric_unit: any of the allowed CloudWatch metric units
string, what you want to call your metric
string, your custom metrics would appear under this namespace
comma separated key value pairs, e.g.
Deploying to your account (via the console)
Go to this page and click the
Deploying via SAM/Serverless framework/CloudFormation
To deploy this via SAM, you need something like this in the CloudFormation template:
AutoDeployMyAwesomeLambdaLayer: Type: AWS::Serverless::Application Properties: Location: ApplicationId: arn:aws:serverlessrepo:us-east-1:374852340823:applications/async-custom-metrics SemanticVersion: <enter latest version> Parameters: EventSourceType: <Lambda or Kinesis> TimeoutSeconds: <defaults to 300> KinesisStreamArn: <ARN for the Kinesis stream if EventSourceType is Kinesis> KinesisStreamBatchSize: <defaults to 100> RecordLambdaUsageMetrics: <"true" or "false"> RecordLambdaCostMetric: <"true" or "false">
To do the same via CloudFormation or the Serverless framework, you need to first add the following
For more details, read this post.
EventSourceType: What's the event source you will intend to use? If logs are pushed from CloudWatch Logs to Lambda directly, then use
Lambda. If logs are pushed to a Kinesis Stream first, then use
Kinesis. Other event sources are not supported (yet).
TimeoutSeconds: (optional) Timeout for the Lambda function that would ship metrics to CloudWatch. Defaults to 30s.
KinesisStreamArn: (optional) Only relevant to the Kinesis event source type. The ARN to the Kinesis stream to subscribe the function to.
KinesisStreamBatchSize: (optional) Only relevant to the Kinesis event source type. The batch size to use for the subscription.
RecordLambdaUsageMetrics: (optional) Whether to collect Lambda usage metrics (i.e. billed duration, memory size and max memory used) from the logs and turn them into metrics. Allowed values are
RecordLambdaCostMetric: (optional) Whether to report estimated cost for Lambda functions as metrics. Allowed values are