Ruby Java Shell HTML PowerShell Batchfile Other
Switch branches/tags
Latest commit a355d9f Jun 23, 2017 @jordansissel jordansissel Use plugin name instead of metadata because sometimes Bundler-provide…
…d Gem::Specifications have nil metadata.

Fixes #7530
Permalink
Failed to load latest commit information.
.github Move Contributors for more visibility May 20, 2016
bin use jruby 9.1.9.0 Jun 12, 2017
ci Remove *.lock and Gemfile before running any acceptance test Jun 16, 2017
config Feature flag for string escape sequences (#7442) Jun 20, 2017
data add data dir back May 30, 2017
docs Remove IDs from common files to fix build errors (#7523) Jun 23, 2017
gradle/wrapper MINOR: Gradle 4.0 Jun 21, 2017
lib Update default license (#7502) Jun 22, 2017
logstash-core-plugin-api bump master version to 6.0.0-alpha3 (#7343) Jun 6, 2017
logstash-core Fix for: Unreliable logstash-core/spec/logstash/timestamp_spec under … Jun 21, 2017
modules Fixed folder layout (#7501) Jun 22, 2017
pkg Only glob *.conf by default in conf.d folder (#7130) May 19, 2017
qa use jruby 9.1.9.0 Jun 12, 2017
rakelib Use plugin name instead of metadata because sometimes Bundler-provide… Jun 23, 2017
spec use jruby 9.1.9.0 Jun 12, 2017
tools Enhance ingest convert to add input/output (#7456) Jun 15, 2017
.gitignore use jruby 9.1.9.0 Jun 12, 2017
.ruby-version MINOR: Add Ruby Version File Jun 16, 2017
.travis.yml Use docker-ce instead of docker engine Jun 22, 2017
CHANGELOG move changelog to markdown and added link Jun 15, 2015
CHANGELOG.md Spelling fixes (#6806) Mar 30, 2017
CONTRIBUTING.md Doc: update contributing guide to include link for IntelliJ setup Fixes Jun 19, 2017
CONTRIBUTORS Doc: update contributing guide to include link for IntelliJ setup Fixes Jun 19, 2017
Gemfile.template MINOR: Update docker-api gem to latest version to improve stabilitiy Jun 15, 2017
LICENSE Generate notice file during artifact build (#6538) Jan 16, 2017
README.md Readme missing update for JRuby 9k Jun 12, 2017
ROADMAP.md New roadmap page May 20, 2016
Rakefile Rakefile: update help with test tasks. Mar 31, 2017
STYLE.md use jruby 9.1.9.0 Jun 12, 2017
build.gradle MINOR: Dry up gradle settings Jun 3, 2017
dripmain.rb remove unused logstash/program file Jun 14, 2017
gradlew Migrate Logstash to Log4j2 Logging (#5651) Aug 25, 2016
gradlew.bat Migrate Logstash to Log4j2 Logging (#5651) Aug 25, 2016
settings.gradle #7128 ingest json to grok js converter May 27, 2017
versions.yml Centralize the JRuby version Jun 13, 2017

README.md

Logstash

Logstash is part of the Elastic Stack along with Beats, Elasticsearch and Kibana. Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash." (Ours is Elasticsearch, naturally.). Logstash has over 200 plugins, and you can write your own very easily as well.

The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

For more info, see https://www.elastic.co/products/logstash

Documentation and Getting Started

You can find the documentation and getting started guides for Logstash on the elastic.co site

Downloads

You can download officially released Logstash binaries, as well as debian/rpm packages for the supported platforms, from downloads page.

Snapshot Builds

For the daring, snapshot builds from master branch are available. These builds are created nightly and have undergone no formal QA, so they should never be run in production.

artifact
tar
zip
deb
rpm

Need Help?

Logstash Plugins

Logstash plugins are hosted in separate repositories under the logstash-plugins github organization. Each plugin is a self-contained Ruby gem which gets published to RubyGems.org.

Writing your own Plugin

Logstash is known for its extensibility. There are hundreds of plugins for Logstash and you can write your own very easily! For more info on developing and testing these plugins, please see the working with plugins section

Plugin Issues and Pull Requests

Please open new issues and pull requests for plugins under its own repository

For example, if you have to report an issue/enhancement for the Elasticsearch output, please do so here.

Logstash core will continue to exist under this repository and all related issues and pull requests can be submitted here.

Developing Logstash Core

Prerequisites

  • Install JDK version 8. Make sure to set the JAVA_HOME environment variable to the path to your JDK installation directory. For example set JAVA_HOME=<JDK_PATH>
  • Install JRuby 9.1.x It is recommended to use a Ruby version manager such as RVM or rbenv.
  • Install rake and bundler tool using gem install rake and gem install bundler respectively.

RVM install (optional)

If you prefer to use rvm (ruby version manager) to manage Ruby versions on your machine, follow these directions:

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
\curl -sSL https://get.rvm.io | bash -s stable --ruby=jruby-9.1.10.0

Check Ruby version

Before you proceed, please check your ruby version by:

$ ruby -v
jruby 9.1.10.0 (2.3.3) 2017-05-25 b09c48a Java HotSpot(TM) 64-Bit Server VM 25.131-b11 on 1.8.0_131-b11 +jit [darwin-x86_64]

Building Logstash

  • To run Logstash from the repo you must first bootstrap the environment:
rake bootstrap
  • You can then use bin/logstash to start Logstash, but there are no plugins installed. To install default plugins, you can run:
rake plugin:install-default

This will install the 80+ default plugins which makes Logstash ready to connect to multiple data sources, perform transformations and send the results to Elasticsearch and other destinatins.

To verify your environment, run the following to send your first event:

bin/logstash -e 'input { stdin { } } output { stdout {} }'

This should start Logstash with stdin input waiting for you to enter an event

hello world
2016-11-11T01:22:14.405+0000 0.0.0.0 hello world

Advanced: Drip Launcher

Drip is a tool that solves the slow JVM startup problem while developing Logstash. The drip script is intended to be a drop-in replacement for the java command. We recommend using drip during development, in particular for running tests. Using drip, the first invocation of a command will not be faster but the subsequent commands will be swift.

To tell logstash to use drip, either set the USE_DRIP=1 environment variable or set JAVACMD=`which drip`.

Example:

USE_DRIP=1 bin/rspec

Caveats

Drip does not work with STDIN. You cannot use drip for running configs which use the stdin plugin.

Testing

Most of the unit tests in Logstash are written using rspec for the Ruby parts. For the Java parts, we use junit. For testing you can use the test rake tasks and the bin/rspec command, see instructions below:

Core tests

1- In order to run the core tests, a small set of plugins must first be installed:

rake test:install-core

2- To run the core tests you can use the rake task:

rake test:core

or use the rspec tool to run all tests or run a specific test:

bin/rspec
bin/rspec spec/foo/bar_spec.rb

3- To run the subset of tests covering the Java codebase only run:

./gradlew test

Plugins tests

To run the tests of all currently installed plugins:

rake test:plugin

You can install the default set of plugins included in the logstash package or all plugins:

rake test:install-default
rake test:install-all

Note that if a plugin is installed using the plugin manager bin/logstash-plugin install ... do not forget to also install the plugins development dependencies using the following command after the plugin installation:

bin/logstash-plugin install --development

Building Artifacts

You can build a Logstash snapshot package as tarball or zip file

rake artifact:tar
rake artifact:zip

This will create the artifact LS_HOME/build directory

You can also build .rpm and .deb, but the fpm tool is required.

rake artifact:rpm
rake artifact:deb

Project Principles

  • Community: If a newbie has a bad time, it's a bug.
  • Software: Make it work, then make it right, then make it fast.
  • Technology: If it doesn't do a thing today, we can make it do it tomorrow.

Contributing

All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.

Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.

It is more important to me that you are able to contribute.

For more information about contributing, see the CONTRIBUTING file.