Skip to content
Writes Snowplow enriched events from Kinesis to Elasticsearch
Branch: develop
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
core/src
elasticsearch/src Sharded indices by date (closes #82) Sep 24, 2018
examples
project
.gitignore
.release.yml
.scalafmt.conf
.travis.yml
CHANGELOG
LICENSE-2.0.txt
README.md Bump sbt to 1.1.5 (closes #91) Aug 6, 2018
build.sbt

README.md

Snowplow Elasticsearch Loader

Introduction

The Snowplow Elasticsearch Loader consumes Snowplow enriched events or failed events from an Amazon Kinesis stream or NSQ topic, transforms them to JSON, and writes them to Elasticsearch. Events which cannot be transformed or which are rejected by Elasticsearch are written to a separate Kinesis stream.

Building

Assuming you already have SBT 1.1.5 installed:

$ git clone git://github.com/snowplow/snowplow-elasticsearch-loader.git
$ sbt compile

Usage

The Snowplow Elasticsearch Loader has the following command-line interface:

snowplow-elasticsearch-loader 0.10.2

Usage: snowplow-elasticsearch-loader [options]

  --config <filename>

Running

Create your own config file:

$ cp examples/config.hocon.sample my.conf

Update the configuration to fit your needs like modifying the AWS credentials:

aws {
  access-key: "default"
  secret-key: "default"
}

Next, start the loader, making sure to specify your new config file:

$ java -jar snowplow-elasticsearch-loader-http-0.10.2.jar --config my.conf

Find out more

Technical Docs Setup Guide Roadmap & Contributing
i1 i2 i3
Technical Docs Setup Guide coming soon

Copyright and license

Copyright 2014-2017 Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

You can’t perform that action at this time.