Logstash Plugin for AWS Kinesis Input
Latest commit da80360 Dec 15, 2016 @codekitchen codekitchen version 2.0.3


Logstash AWS Kinesis Input Plugin

Build Status

This is a AWS Kinesis input plugin for Logstash. Under the hood uses the Kinesis Client Library.


This plugin requires Logstash >= 2.0, and can be installed by Logstash itself.

bin/logstash-plugin install logstash-input-kinesis


input {
  kinesis {
    kinesis_stream_name => "my-logging-stream"
    codec => json { }

Using with CloudWatch Logs

If you are looking to read a CloudWatch Logs subscription stream, you'll also want to install and configure the CloudWatch Logs Codec.


This are the properties you can configure and what are the default values:

  • application_name: The name of the application used in DynamoDB for coordination. Only one worker per unique stream partition and application will be actively consuming messages.
    • required: false
    • default value: logstash
  • kinesis_stream_name: The Kinesis stream name.
    • required: true
  • region: The AWS region name for Kinesis, DynamoDB and Cloudwatch (if enabled)
    • required: false
    • default value: us-east-1
  • checkpoint_interval_seconds: How many seconds between worker checkpoints to DynamoDB. A low value ussually means lower message replay in case of node failure/restart but it increases CPU+network ussage (which increases the AWS costs).
    • required: false
    • default value: 60
  • metrics: Worker metric tracking. By default this is disabled, set it to "cloudwatch" to enable the cloudwatch integration in the Kinesis Client Library.
    • required: false
    • default value: nil


This plugin uses the default AWS SDK auth chain, DefaultAWSCredentialsProviderChain, to determine which credentials the client will use.

The default chain follows this order trying to read the credentials:

  • AWS_ACCESS_KEY_ID / AWS_SECRET_KEY environment variables
  • ~/.aws/credentials credentials file
  • EC2 instance profile

The credentials will need access to the following services:

  • AWS Kinesis
  • AWS DynamoDB: the client library stores information for worker coordination in DynamoDB (offsets and active worker per partition)
  • AWS CloudWatch: if the metrics are enabled the credentials need CloudWatch update permisions granted.

Look at the documentation for deeper information.


  1. https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md#contribution-steps
  2. Fork it ( https://github.com/logstash-plugins/logstash-input-kinesis/fork )
  3. Create your feature branch (git checkout -b my-new-feature)
  4. Commit your changes (git commit -am 'Add some feature')
  5. Push to the branch (git push origin my-new-feature)
  6. Create a new Pull Request