AWS CloudWatch log group alerting.
Sniffles is a collection of CloudFormation templates and lambdas that you can compose together to suit your needs.
There are two ways to monitor log groups:
- A lambda looks through every log entry and sends you the logs you care about, or
- CloudWatch alarm whenever a log you care about appears
Lambda solution pros:
- You receive the log entry in the alert
CloudWatch alarm solution pros:
- You can alarm only after a threshold during a timeperiod has been reached
- Lambda-less (no maintenance)
Log groups can of course be monitored with both solutions.
Prerequisites:
- If you want to use the Chatbot integration, set up chatbot with slack
- If you want to use the Slack lambda integration, set up a Slack app with chat:write and files:write permissions in your workspace
- Clone this repo
- Assume into the AWS account you want to deploy to
cp config.{template,sh}- Edit
config.shwith your favourite editor, enter the required config ./deploy-all.sh ./config.sh
This will set up:
- Kinesis stream
- SNS topic
- Assets S3 bucket
- Lambda to filter the logs from the Kinesis stream, publishes to SNS topic
- Slack lambda that consumes from SNS topic and sends to Slack
- SNS topic to Opsgenie integration
- Automatic log subscriber (CloudWatch logs to Kinesis stream)
- CloudWatch alarms for Sniffles lambda errors
- Chatbot to send CloudWatch alarms to Slack
- CloudWatch to Opsgenie integration
To deploy individual pieces you simply deploy the corresponding module.
Sniffles assets storage (an S3 bucket).
BucketName - String
Name of the S3 bucket to create
aws cloudformation deploy \
--stack-name "<stackname>" \
--template-file ./cloudformation-templates/assets.yml \
--parameter-overrides <parameters>Sets up an SNS topic for the core lambda or CloudWatch to publish to.
Nope.
aws cloudformation deploy \
--stack-name "<stackname>" \
--template-file ./cloudformation-templates/topic.yml
ExistingKinesisStream - ARN
If you supply this, the Kinesis Stream and the CloudWatch role won’t be created, the lambda will instead be subscribed to the existing Kinesis Stream
SnsTopic - ARN
The SNS Topic to publish to
WhitelistParameterStorePath - String
Where in Parameter Store the lambda should look for the whitelist. The whitelist is a comma separated string of regular expressions. Any logs that match any of the regular expressions in the whitelist will be passed on the the SNS Topic. Any non-matching logs will be discarded.
ProjectKey - String
Will appear in the Subject of the SNS Message
ErrorMessage - String
Will appear in the Subject of the SNS Message
S3Bucket - String
Assets bucket
S3Key - String
Location of the lambda code
A comma separated string of either strings, regular expressions, or jspaths.
Examples:
ERROR
Any log that contains ERROR
/\serror\s/i
Any log that contains error surrounded by white space, case-insensitive.
ERROR, { .level === "error" }
Any log that contains ERROR or a JSON object with a property named level that has a value of error.
aws cloudformation deploy \
--stack-name "<stackname>" \
--template-file ./cloudformation-templates/core.yml \
--parameter-overrides <parameters> \
--capabilities CAPABILITY_NAMED_IAMLambda code in slack-lambda.
SnsTopic - ARN
The SNS Topic to subscribe to
SlackParameterStorePath - ARN
The Parameter Store path that holds the Slack token
SlackParameterStoreKey - ARN
The KMS Key ID/Alias with which the Slack token is encrypted
SlackChannel - String
The Slack channel to notify in
S3Bucket - String
Assets bucket
S3Key - String
Location of the lambda code
aws cloudformation deploy \
--stack-name "<stackname>" \
--template-file ./cloudformation-templates/slack.yml \
--parameter-overrides <parameters> \
--capabilities CAPABILITY_IAM
SnsTopic - ARN
The SNS Topic to subscribe to
OpsgenieEndpoint - URL
The Opsgenie endpoint, e.g. https://api.opsgenie.com/v1/json/amazonsns?apiKey=XXXXXX
aws cloudformation deploy \
--stack-name "<stackname>" \
--template-file ./cloudformation-templates/opsgenie.yml \
--parameter-overrides <parameters>Automatically subscribes Kinesis to CloudWatch Log Groups based on regular expressions.
KinesisStream - ARN
The Kinesis Stream that logs should be forwarded to
CloudwatchRole - ARN
The IAM role that CloudWatch should use to send logs to the Kinesis Stream.
LogGroupPatternsParameterStorePath - String
The Parameter Store path that holds the CloudWatch Log Group patterns that should be set up with subscriptions to Kinesis. e.g. ^/aws/lambda/.-prod-.$, ^/aws/lambda/.-test-.$
A comma separated string of regular expressions.
Examples:
^/aws/lambda/.*-prod-.*$, ^/aws/lambda/.*-test-.*$
aws cloudformation deploy \
--stack-name "<stackname>" \
--template-file ./cloudformation-templates/log-subscriber.yml \
--parameter-overrides <parameters> \
--capabilities CAPABILITY_IAM
SnsTopic - ARN
The SNS Topic to subscribe to
SlackChannelId - String
The Slack channel ID to Chatbot will post in
SlackWorkspaceId - String
The Slack workspace ID the channel resides in
aws cloudformation deploy \
--stack-name "<stackname>" \
--template-file ./cloudformation-templates/chatbot.yml \
--parameter-overrides <parameters> \
--capabilities CAPABILITY_IAM
SnsTopic - ARN
The SNS Topic to publish alarms to
LogGroupName - String
e.g. /aws/lambda/some-function
FunctionName - String
e.g. MyFunction-Test
ProjectKey - String
Will appear in the title of the alarm
EvaluationPeriods - String
How many evaluation periods to consider
DatapointsToAlarm - String
How many datapoints in the evaluation period need to be over the threshold to trigger the alarm
aws cloudformation deploy \
--stack-name "<stackname>" \
--template-file ./cloudformation-templates/lambda-cloudwatch-alarm.yml \
--parameter-overrides <parameters>Sniffles is licensed under the terms of the MIT license.