CloudWatch to Logsene AWS Lambda function
This tutorial shows how to send CloudWatch logs to a Logsene application. The code in this repository can be used to send any CloudWatch logs to Logsene. To illustrate how to do that we'll use AWS VPC logs, for which this Lambda function happens to have built-in parsing. If you're using a type of CloudWatch logs that isn't supported yet, feel free to edit pattern.yml or to open an issue.
The main steps are: 0. Create a Flow Log for your VPC, if there isn't one already. If you're looking to ship other CloudWatch logs, just skip this step and go through the rest.
- Create a new Lambda Function
- Clone this repository and fill in your Logsene Application Token, create a ZIP file with the contents of the cloned repository, and configure the new Lambda function to use the created ZIP file as code
- Decide on the maximum memory to allocate for this function and the timeout for its execution
- Explore your logs in Logsene :)
Create a Flow Log
Then you'll need to set up a IAM role that's able to push VPC logs to your CloudWatch account (if you don't have one already) and then choose a name for this flow. You'll use the name later on in the lambda function.
Create a new AWS Lambda function
The next step is to select a source. Here you'd make sure the source type is CloudWatch Logs and select the flow you just created. You can filter only certain logs, but you'd normally leave the Filter Pattern empty to process all of them. Nevertheless, you need to give this filter a name:
Then you have to specify the code.
Add the code to your Lambda function
First you'd need to clone this repository:
Optionally: Edit pattern.yml (see logagent parser) for additional parser rules, depending on the structure of your logs. Note: The "sourceName"" in the pattern definition should match the AWS "logGroup".
Now your code is ready, so you need to make a zip file out of it. Note: make sure you zip only the contents of the repository, not the directory containing the repository. The correct way to do it is something like this:
pwd # /tmp/cloned-repos/logsene-aws-lambda-cloudwatch zip -r logsene.zip *
Set the Logsene application token in LOGSENE_TOKEN environment variable. To find the Logsene Application Token, go to your Sematext Account, then in the Services menu select Logsene, and then the Logsene application you want to send your logs to. Once you're in that application, click the Integration button and you'll see the application token:
Finalize the function configuration
After the code, leave the handler to the default index.handler and select a role that allows this function to execute. You can create a new Basic execution role to do that (from the drop-down) or select a basic execution role that you've already created:
Then, you need to decide on how much memory you allow for the function and how long you allow it to run. This depends on the log throughput (more logs will need more processing resources) and will influence costs (i.e. like keeping the equivalent general-purpose instance up for that time). Normally, runtime is very short so even large resources shouldn't generate significant costs. 256MB and 30 second timeout should be enough for most use-cases:
Exploring CloudTrail logs with Logsene