Skip to content
Branch: master
Find file History
Latest commit 017cb89 Feb 19, 2020

Files

Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
tests Implement code review feedback Oct 4, 2019
.gitignore v3.0.0 Jan 22, 2020
README.md Update the default CF stack name Feb 18, 2020
enhanced_lambda_metrics.py
lambda_function.py [Lambda]: Use default context for SSL Feb 17, 2020
release.sh Update release script with new S3 folder structure Feb 18, 2020
template.yaml Bump version from 3.1.0 to 3.2.0 Feb 11, 2020

README.md

Datadog Forwarder

AWS Lambda function to ship logs from S3 and CloudWatch, custom metrics and traces from Lambda functions to Datadog.

Features

  • Forward CloudWatch, ELB, S3, CloudTrail, VPC and CloudFront logs to Datadog
  • Forward S3 events to Datadog
  • Forward Kinesis data stream events to Datadog, only CloudWatch logs are supported
  • Forward custom metrics from AWS Lambda functions via CloudWatch logs
  • Forward traces from AWS Lambda functions via CloudWatch logs
  • Generate and submit enhanced Lambda metrics (aws.lambda.enhanced.*) parsed from the AWS REPORT log: duration, billed_duration, max_memory_used, and estimated_cost

Installation

Launch Stack

  1. Log into your admin AWS account/role and deploy the CloudFormation Stack with the button above.
  2. Fill in DdApiKey and select the appropriate DdSite. All other parameters are optional.
  3. Click Create stack.
  4. Set up triggers to the installed Forwarder either automatically or manually.
  5. Repeat the above steps in another region if you operate in multiple AWS regions.

Note: you can find the installed Forwarder under the stack's "Resources" tab.

Updating

Upgrade to a new version

  1. Find the datadog-forwarder (if you didn't rename it) CloudFormation stack.
  2. Update the stack using template https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml. You can also replace latest with a specific version, e.g., 3.0.2.yaml, if needed.

Upgrade an older version to +3.0.0

Since version 3.0.0, the forwarder Lambda function is managed by CloudFormation. To upgrade an older forwarder installation to 3.0.0 and above, follow the steps below.

  1. Install a new forwarder following the installation steps.
  2. Manually migrate a few triggers on the old forwarder to the new one.
  3. Ensure the new forwarder is working as expected.
  4. Update all the triggers on the old forwarder to the new one, following these steps.
  5. Delete the old forwarder Lambda function when you feel comfortable.

Adjusting forwarder settings

  1. Find the datadog-forwarder (if you didn't rename it) CloudFormation stack.
  2. Update the stack using the current template.
  3. Adjust parameter values.

Note: It's recommended to adjust forwarder settings through CloudFormation rather than directly editing the Lambda function. The description of settings can be found in the template.yaml and the CloudFormation stack creation user interface when you launch the stack. Feel free to submit a pull request to make additional settings adjustable through the template.

Settings

To view all the adjustable settings of the forwarder, click "Launch Stack" from the Installation section and you will be prompted with a CloudFormation user interface with all the adjustable settings (you do not have to complete the installation).

The technical definition of the settings can be found in the "Parameters" section of template.yaml.

Troubleshooting

Set the environment variable DD_LOG_LEVEL to debug on the Forwarder Lambda function to enable detailed logging temporarily (don't forget to remove it). If the debug logs don't help, please contact Datadog support.

Notes

  • For S3 logs, there may be some latency between the time a first S3 log file is posted and the Lambda function wakes up.
  • Currently, the forwarder has to be deployed manually to GovCloud and China, and supports only log forwarding.
    1. Create a Lambda function using aws-dd-forwarder-<VERSION>.zip from the latest releases.
    2. Save your Datadog API key in AWS Secrets Manager, and set environment variable DD_API_KEY_SECRET_ARN with the secret ARN on the Lambda function.
    3. Configure triggers.
You can’t perform that action at this time.