Skip to content
A Beat for shipping AWS Lambda metrics.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
beater
config
LICENSE
Makefile
README.org
glide.yaml
lambdabeat.template.json
lambdabeat.yml
main.go
main_test.go

README.org

Lambdabeat

Overview

Lambdabeat allows you to ship AWS Lambda metrics (by way of the Cloudwatch API) to Elasticsearch.

Here is an example of a document produced by Lambdabeat:

POST http://localhost:9200/lambdabeat-2016.06.14/_search?pretty
{
  "size": 1,
  "query": {
    "match_all": {}
  }
}
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 79,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "lambdabeat-2016.06.14",
      "_type" : "metric",
      "_id" : "AVVQbTk8fZl27BsUqkOv",
      "_score" : 1.0,
      "_source" : {
        "@timestamp" : "2016-06-14T01:03:00.000Z",
        "beat" : {
          "hostname" : "baamonde-ThinkPad-T460s",
          "name" : "baamonde-ThinkPad-T460s"
        },
        "code-size" : 3564954,
        "description" : "my lambda function",
        "duration-average" : 2718.6,
        "duration-maximum" : 2718.6,
        "duration-minimum" : 2718.6,
        "duration-sample-count" : 1,
        "duration-sum" : 2718.6,
        "duration-unit" : "Milliseconds",
        "errors-average" : 0,
        "errors-maximum" : 0,
        "errors-minimum" : 0,
        "errors-sample-count" : 1,
        "errors-sum" : 0,
        "errors-unit" : "Count",
        "function" : "my-lambda-function",
        "handler" : "my-lambda-function.handler",
        "invocations-average" : 1,
        "invocations-maximum" : 1,
        "invocations-minimum" : 1,
        "invocations-sample-count" : 1,
        "invocations-sum" : 1,
        "invocations-unit" : "Count",
        "last-modified" : "2016-04-04T02:46:48.557+0000",
        "memory-size" : 128,
        "runtime" : "python2.7",
        "throttles-average" : 0,
        "throttles-maximum" : 0,
        "throttles-minimum" : 0,
        "throttles-sample-count" : 1,
        "throttles-sum" : 0,
        "throttles-unit" : "Count",
        "timeout" : 60,
        "type" : "metric",
        "version" : "$LATEST"
      }
    } ]
  }
}

Installation

Binaries are available here.

Released packages contain:

  1. The lambdabeat binary.
  2. An example =lambdabeat.yml” configuration file.
  3. The lambdabeat.template.json mapping template.

Configuration and Usage

AWS: Credentials and Region

Lambdabeat fetches data from the Cloudwatch API. As such, it does not need to be deployed anywhere in particular, but it does require valid AWS credentials to be specified either:

  1. In an AWS configuration file located at ~/.aws/credentials.
  2. As environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.)

See here for further details regarding credentials.

Additionally, Lambdabeat requires that a default AWS region be set in the lambdabeat.yml configuration file.

Functions

Simply specify an array of Lambda functions for which you would like to retrieve metrics in lambdabeat.yml like so:

lambdabeat:
  functions: ["foo", "bar", "baz", "qux"]

Metrics

Cloudwatch maintains data for these metrics with respect to Lambda:

  • Invocations
  • Errors
  • Duration
  • Throttles

These are largely self-explanatory, but see here for details and subtleties.

Lambdabeat fetches and indexes all of these.

Period and Interval

Unlike many (really, most) other Beats, Lambdabeat’s notion of a period does not necessarily determine the timestamps of the events being indexed. Rather, the Cloudwatch API provides data between two points in time at a given, user-specified interval. This is configured via the interval key in lambdabeat.yml.

Separately, the period key specifies how often Lambdabeat itself should run. As such, the value of the @timestamp field for each event is not the time at which Lambdabeat fetched the data; it’s the time for which Cloudwatch has returned data. At query time, this is an unimportant implementation detail. What matters is that the events indexed into Elasticsearch are timestamped in a meaningful matter, which they are. It is important, however, to understand the distinction at configuration time.

Note that the value of interval must be a multiple of 60. Note also that it only really makes sense for period to be greater than or equal to interval.

Backfilling data

If you would like to index data that predates the first invocation of Lambdabeat, you may do so by passing a the option -backfill-date when invoking lambdabeat. This will fetch and index all data between the given backfill-date and the time at which Lambdabeat begins running.

For example:

lambdabeat -e -v -backfill-date="2016-06-08T00:00:00.000Z"

Outputs

Lambdabeat currently supports Elasticsearch as an output. Simply provide an array of hosts in lambdabeat.yml like so:

output:
  elasticsearch:
    hosts: ["http://localhost:9200"]

Lambdabeat introduces no novel infrastructure with respect to outputs.

Development

Requirements

Build

To build the binary for Lambdabeat run the command below. This will generate a binary in the same directory with the name lambdabeat.

make

Run

To run Lambdabeat with debugging output enabled, run:

./lambdabeat -c lambdabeat.yml -e -d "*"

Package

To produce a versioned tarball containing the lambdabeat binary, lambdabeat.yml, and lambdabeat index template, run:

make release

Cleanup

To clean up the build directory and generated artifacts, run:

make clean

Clone

To clone Lambdabeat from the git repository, run the following commands:

mkdir -p ${GOPATH}/github.com/michaelbaamonde
cd ${GOPATH}/github.com/michaelbaamonde
git clone https://github.com/michaelbaamonde/lambdabeat

For further development, check out the beat developer guide.

You can’t perform that action at this time.