Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LOG-4540 - Rewritten lambda to use more efficient API calls #2

Merged
merged 3 commits into from Jan 30, 2020
Merged
Changes from 1 commit
Commits
File filter...
Filter file types
Jump to…
Jump to file
Failed to load files.

Always

Just for now

Updated script to use environment variable to get customer token, upd…

…ated README
  • Loading branch information
Jiri Tomek
Jiri Tomek committed May 23, 2019
commit 382d2537d35549816fb3c6b5d7f510b727a955e5
@@ -5,56 +5,58 @@ Sends AWS Cloudwatch metrics to Loggly
* http://aws.amazon.com/lambda/
* https://www.loggly.com/

## Get the code and prepare it for the uploading to AWS
* Clone the git repo
```bash
git clone https://github.com/psquickitjayant/cloudwatch-metrics-to-loggly.git
cd cloudwatch-metrics-to-loggly
```
* Install required npm packages.
```
npm install
```
## Getting started with AWS Lambda
Getting started documentation for AWS lambda can be found in [this
article](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html). There is also [this blog post](http://alestic.com/2014/11/aws-lambda-cli) on how to set things up using the command line tools.

* zip up your code
```bash
zip -r cloudwatchMetrics2Loggly.zip index.js node_modules
```

The resulting zip (cloudwatchMetrics2Loggly.zip) is what you will upload to AWS.

## Setting up AWS
For all of the AWS setup, I used the AWS console following [this
example](http://docs.aws.amazon.com/lambda/latest/dg/getting-started-amazons3-events.html). Below, you will find a high-level
description of how to do this. I also found [this blog post](http://alestic.com/2014/11/aws-lambda-cli) on how to set things up
using the command line tools.

### Create and upload the cloudwatchMetrics2Loggly function in the AWS Console
## Creating lambda function in AWS
1. Build lambda function code
1. Clone the Git repository:
`git clone https://github.com/psquickitjayant/cloudwatch-metrics-to-loggly.git`
1. Go to cloned folder:
`cd cloudwatch-metrics-to-loggly`
1. Install dependencies:
`npm install`
1. Create a .zip file for upload to AWS console later:
`zip -r cloudwatchMetrics2Loggly.zip index.js node_modules`
1. Create Role
1. Sign in to your AWS account and open IAM console https://console.aws.amazon.com/iam/
2. In your IAM console create a new Role say, 'cloudwatch-full-access'
3. Select Role Type as 'AWS Lambda'
4. Apply policy 'CloudWatchFullAccess' and save.
2. Create KMS Key
1. Create a KMS key - http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html
2. Encrypt the Loggly Customer Token using the AWS CLI - **aws kms encrypt --key-id alias/<your KMS key arn> --plaintext "<your loggly customer token>"**
3. Copy the base-64 encoded, encrypted token from step 2's CLI output (CiphertextBlob attribute) and replace it with the "your KMS encypted key" in the script at line no 13
3. Create lambda function
1. https://console.aws.amazon.com/lambda/home
2. Click "Create a Lambda function" button. *(Choose "Upload a .ZIP file")*
* **Name:** *cloudwatchMetrics2Loggly*
* Upload lambda function (zip file you made above.)
* **Handler*:** *index.handler*
* Set Role : *cloudwatch-full-access*
* Set Timeout to 2 minutes
3. Go to your Lamda function and select the "Event sources" tab
* Click on **Add Event Source**
* Event Source Type : *Cloudwatch Events - Schedule*
* Name : Provide any customized name. e.g. cloudwatchMetrics2Loggly Event Source
* Description: Invokes Lambda function in every 5 minutes
* Schedule expression : *rate(5 minutes)*
* Enable Event Source : *Enable Now*
Now click on submit and wait for the events to occur in Loggly
1. Sign in to your AWS account and open IAM console https://console.aws.amazon.com/iam/
2. In your IAM console create a new Role say, `cloudwatch-full-access`
3. Select Role Type as **AWS Lambda**
4. Apply policy **CloudWatchFullAccess** and save.
1. Create KMS Key
1. Create a KMS key - http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html
1. Encrypt the Loggly Customer Token using the AWS CLI
`aws kms encrypt --key-id alias/<your KMS key alias> --plaintext "<your loggly customer token>"`
1. Copy or keep `CiphertextBlob` attribute for furhter usage in the next step.
1. Create lambda function
1. Go to https://console.aws.amazon.com/lambda/home
1. Click **Create a Lambda function** button.
1. Select **Author from scratch** option
1. Set **Function name** for example to `cloudwatchMetrics2Loggly`
1. Set **Runtime** to `Node.js 10.x`
1. Under **Permissions** click on **Choose or create an execution role**
1. Select **Use an existing role** and select **cloudwatch-full-access** role created above in step 1
1. Click on **Create function** button
1. Scroll to **Function code** section
1. Select **Upload a .zip file** in **Code entry type** dropdown
1. Upload lambda function (zip file `cloudwatchMetrics2Loggly.zip` you made above)
1. Go to **Environment variables** section
1. Define new environment variable **kmsEncryptedCustomerToken** and set it to `CiphertextBlob` value from step 3 above (Create KMS Key)
1. Scroll to **Basic settings** section
1. Set **Memory (MB)** to **512 MB**
1. Set Timeout to **2** minutes
1. Scroll up to **Designer** section (expand if it's collapsed)
1. Click on **CloudWatch Events** in **Add triggers** section to add the trigger item
1. Click on added **CloudWatch Events** trigger and scroll down to trigger settings
1. Select **Create a new rule** in **Rule** dropdown
1. **Name:** Provide any customized name. e.g. cloudwatchMetrics2Loggly Event Source
1. **Description:** Invokes Lambda function in every 5 minutes
1. **Schedule expression:** rate(5 minutes)
1. Make sure the checkbox **Enable trigger** is checked
1. Click on **Add**
1. Click on **Save** to save the whole lambda function.
1. Wait for the events to occur in Loggly

**NOTE**: Always use latest version of **AWSCLI**. Some features like KMS may not work on older versions of AWSCLI. To upgrade, use the command given below

@@ -13,9 +13,6 @@ var logglyConfiguration = {
tags: 'CloudwatchMetrics'
};

var encryptedLogglyToken = "your KMS encrypted key";
var encryptedLogglyTokenBuffer = Buffer.from(encryptedLogglyToken, "base64");

var kms = new AWS.KMS({
apiVersion: '2014-11-01'
});
@@ -42,21 +39,27 @@ exports.handler = function (event, context) {
decryptLogglyToken().then(function () {
getMetricsListFromAWSCloudwatch().then(function () {
sendRemainingStatistics().then(function () {
context.done('all statistics are sent to Loggly');
context.done();
}, function () {
context.done();
});
}, function () {
context.done();
});
}, function () {
context.done();
}, function (reason) {
context.done(reason);
});

//decrypts your Loggly Token from your KMS key
function decryptLogglyToken() {

return Q.Promise(function (resolve, reject) {
if (!process.env.kmsEncryptedCustomerToken) {
reject("Environment variable 'kmsEncryptedCustomerToken' is not defined. Define 'kmsEncryptedCustomerToken' "
+ "environment variable and set it to KMS encrypted customer token for Loggly.");
}

var encryptedLogglyTokenBuffer = Buffer.from(process.env.kmsEncryptedCustomerToken, "base64");
var params = {
CiphertextBlob: encryptedLogglyTokenBuffer
};
@@ -1,14 +1,14 @@
{
"name": "cloudwatch-metrics-to-loggly",
"version": "1.0.0",
"version": "2.0.0",
"description": "A NodeJS library to send Cloudwatch Metrics to Loggly",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": {
"type": "git",
"url": "https://github.com/psquickitjayant/cloudwatch-metrics-to-loggly.git"
"url": "https://github.com/loggly/cloudwatch-metrics-to-loggly.git"
},
"keywords": [
"Loggly",
@@ -18,9 +18,9 @@
"author": "psquickitjayant",
"license": "MIT",
"bugs": {
"url": "https://github.com/psquickitjayant/cloudwatch-metrics-to-loggly/issues"
"url": "https://github.com/loggly/cloudwatch-metrics-to-loggly/issues"
},
"homepage": "https://github.com/psquickitjayant/cloudwatch-metrics-to-loggly",
"homepage": "https://github.com/loggly/cloudwatch-metrics-to-loggly",
"dependencies": {
"aws-sdk": "^2.2.25",
"q": "^1.4.1",
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.