Skip to content
A Serverless application that parses custom metrics from CloudWatch Logs and sends them to CloudWatch as custom metrics
JavaScript
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci
examples feat: support raw Kinesis events Sep 9, 2019
functions feat: improve error handling and logging Sep 11, 2019
.editorconfig feat: init commit Jul 23, 2019
.eslintignore feat: init commit Jul 23, 2019
.eslintrc.json feat: init commit Jul 23, 2019
.gitignore build: added npm scripts for publishing sar app Jul 23, 2019
LICENSE chore: updated license and bumped version Sep 4, 2019
README.md build: bumped version Sep 11, 2019
commitlint.config.js feat: init commit Jul 23, 2019
jest.config.js feat: init commit Jul 23, 2019
package-lock.json feat: improve error handling and logging Sep 11, 2019
package.json feat: improve error handling and logging Sep 11, 2019
template.yml build: bumped version Sep 11, 2019

README.md

SAR-async-lambda-metrics

Version License CircleCI Greenkeeper badge

A Serverless application that parses custom metrics from CloudWatch Logs and sends them to CloudWatch as custom metrics.

This application deploys a single Lambda function with a name prefixed with serverlessrepo-async-custom-metrics. This function supports both CloudWatch Logs as well as Kinesis as event source.

You can subscribe the function to CloudWatch log groups directly. But since you can only have one subscription filter per log group and you probably want to ship your logs elsewhere (maybe to an ELK stack?), most likely you'll subscribe log groups to a Kinesis stream first, then subscribe this serverlessrepo-async-custom-metrics function to the Kinesis stream.

To help you manage the subscription of your logs, consider using this Serverless application.

Once subscribed, you can record custom metrics by writing to stdout. The function would parse the custom metrics out of your logs and send them to CloudWatch as metrics.

The format of the custom metric needs to follow the convention:

MONITORING|<metric_value>|<metric_unit>|<metric_name>|<namespace>|<dimensions>

where:

  • metric_value: float
  • metric_unit: any of the allowed CloudWatch metric units
  • metric_name: string, what you want to call your metric
  • namespace: string, your custom metrics would appear under this namespace
  • dimensions: comma separated key value pairs, e.g. service=content-item,region=eu-west-1

e.g.

MONITORING|1|Count|request_count|theburningmonk.com|service=content-item,region=eu-west-1 MONITORING|42.7|Milliseconds|latency|theburningmonk.com|service=content-item,region=eu-west-1

Deploying to your account (via the console)

Go to this page and click the Deploy button.

Deploying via SAM/Serverless framework/CloudFormation

To deploy this via SAM, you need something like this in the CloudFormation template:

AutoDeployMyAwesomeLambdaLayer:
  Type: AWS::Serverless::Application
  Properties:
    Location:
      ApplicationId: arn:aws:serverlessrepo:us-east-1:374852340823:applications/async-custom-metrics
      SemanticVersion: <enter latest version>
    Parameters:
      EventSourceType: <Lambda or Kinesis>
      TimeoutSeconds: <defaults to 300>
      KinesisStreamArn: <ARN for the Kinesis stream if EventSourceType is Kinesis>
      KinesisStreamBatchSize: <defaults to 100>
      RecordLambdaUsageMetrics: <"true" or "false">
      RecordLambdaCostMetric: <"true" or "false">

To do the same via CloudFormation or the Serverless framework, you need to first add the following Transform:

Transform: AWS::Serverless-2016-10-31

For more details, read this post.

Parameters

EventSourceType: What's the event source you will intend to use? If logs are pushed from CloudWatch Logs to Lambda directly, then use Lambda. If logs are pushed to a Kinesis Stream first, then use Kinesis. Other event sources are not supported (yet).

TimeoutSeconds: (optional) Timeout for the Lambda function that would ship metrics to CloudWatch. Defaults to 30s.

KinesisStreamArn: (optional) Only relevant to the Kinesis event source type. The ARN to the Kinesis stream to subscribe the function to.

KinesisStreamBatchSize: (optional) Only relevant to the Kinesis event source type. The batch size to use for the subscription.

RecordLambdaUsageMetrics: (optional) Whether to collect Lambda usage metrics (i.e. billed duration, memory size and max memory used) from the logs and turn them into metrics. Allowed values are "true" or "false".

RecordLambdaCostMetric: (optional) Whether to report estimated cost for Lambda functions as metrics. Allowed values are "true" or "false".

You can’t perform that action at this time.