Skip to content
Logstash - transport and process your logs, events, or other data
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Add section for security vulnerabilities to new issue template. Mar 23, 2018
bin Java 11 support (#10279) Feb 5, 2019
ci Update logstash_releases.json May 28, 2019
config rename config option .url and .ca to .hosts and .certificate_authority ( Feb 5, 2019
data Add back missing gitkeep in data dir Jan 23, 2018
docker generate tarballs for docker images May 24, 2019
docs remove the beta designation from the docs for Java plugins Jun 25, 2019
gradle/wrapper Java 11 support (#10279) Feb 5, 2019
lib avoid clashes between Environment class methods Jun 14, 2019
licenses Harmonize license with apache.org and other Elastic repos Jun 19, 2018
logstash-core-plugin-api Add VERSION_QUALIFIER support for use by release manager (#10117) Nov 7, 2018
logstash-core Remove debug code for p2p plus formatting Jun 25, 2019
modules bump to 8.0.0 (#10400) Feb 6, 2019
pkg Unify logging properties across distributions. Mar 29, 2018
qa Rename filebeat.prospectors to filebeat.inputs (#10711) Apr 23, 2019
rakelib Add Logstash JMS input to the list of default plugins Jun 17, 2019
spec Java 11 support (#10279) Feb 5, 2019
tools Enhancements to Logstash Benchmarking Tool (#10253) Feb 28, 2019
x-pack Enhance GET _node/stats/pipelines API for Metricbeat monitoring (#10576) Jun 22, 2019
.dockerignore Introduce docker for CI builds. Sep 14, 2017
.gitignore build docker images from logstash repo (#10603) Apr 4, 2019
.ruby-version fix building of deb and rpm Feb 6, 2019
CONTRIBUTING.md documentation for the dependency license audit tool Aug 21, 2018
CONTRIBUTORS Doc: update contributing guide to include link for IntelliJ setup Fixes Jun 19, 2017
COPYING.csv Add tool to generate dependency details in csv Dec 22, 2017
Dockerfile moved to openjdk 11 in docker testing (#10563) Apr 15, 2019
Dockerfile.base moved to openjdk 11 in docker testing (#10563) Apr 15, 2019
Gemfile.template cleanup many rakelib tasks that aren't used anymore Mar 15, 2019
LICENSE.txt Migrate x-pack-logstash source to logstash Apr 24, 2018
NOTICE.TXT manually generate NOTICE.TXT from licenses in dependencies-report Aug 10, 2018
README.md Update JRuby and Java prerequisites Mar 28, 2019
ROADMAP.md remove overly-broad statements about licensing from docs Mar 15, 2018
Rakefile Multiple spelling corrections (#9782) Jul 4, 2018
STYLE.md use jruby 9.1.9.0 Jun 12, 2017
build.gradle moved to openjdk 11 in docker testing (#10563) Apr 15, 2019
gradle.properties Test fix: disable the gradle daemon by default (as recommended for CI) Oct 5, 2017
gradlew Java 11 support (#10279) Feb 5, 2019
gradlew.bat Java 11 support (#10279) Feb 5, 2019
rubyUtils.gradle * Adds a java_generator input with jdots codec to facilitate testing,… May 3, 2019
settings.gradle Load tests via JUnit May 15, 2018
versions.yml Bump JrJackson to 0.4.8 May 7, 2019

README.md

Logstash

Logstash is part of the Elastic Stack along with Beats, Elasticsearch and Kibana. Logstash is a server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash." (Ours is Elasticsearch, naturally.). Logstash has over 200 plugins, and you can write your own very easily as well.

For more info, see https://www.elastic.co/products/logstash

Documentation and Getting Started

You can find the documentation and getting started guides for Logstash on the elastic.co site

For information about building the documentation, see the README in https://github.com/elastic/docs

Downloads

You can download officially released Logstash binaries, as well as debian/rpm packages for the supported platforms, from downloads page.

Snapshot Builds

For the daring, snapshot builds are available. These builds are created nightly and have undergone no formal QA, so they should never be run in production.

Complete, with X-Pack Apache 2.0 licensed
tar-complete tar-oss
zip-complete zip-oss
deb-complete deb-oss
rpm-complete rpm-oss

Need Help?

Logstash Plugins

Logstash plugins are hosted in separate repositories under the logstash-plugins github organization. Each plugin is a self-contained Ruby gem which gets published to RubyGems.org.

Writing your own Plugin

Logstash is known for its extensibility. There are hundreds of plugins for Logstash and you can write your own very easily! For more info on developing and testing these plugins, please see the working with plugins section

Plugin Issues and Pull Requests

Please open new issues and pull requests for plugins under its own repository

For example, if you have to report an issue/enhancement for the Elasticsearch output, please do so here.

Logstash core will continue to exist under this repository and all related issues and pull requests can be submitted here.

Developing Logstash Core

Prerequisites

  • Install JDK version 8 or 11. Make sure to set the JAVA_HOME environment variable to the path to your JDK installation directory. For example set JAVA_HOME=<JDK_PATH>
  • Install JRuby 9.2.x It is recommended to use a Ruby version manager such as RVM or rbenv.
  • Install rake and bundler tool using gem install rake and gem install bundler respectively.

RVM install (optional)

If you prefer to use rvm (ruby version manager) to manage Ruby versions on your machine, follow these directions. In the Logstash folder:

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
\curl -sSL https://get.rvm.io | bash -s stable --ruby=$(cat .ruby-version)

Check Ruby version

Before you proceed, please check your ruby version by:

$ ruby -v

The printed version should be the same as in the .ruby-version file.

Building Logstash

The Logstash project includes the source code for all of Logstash, including the Elastic-Licensed X-Pack features and functions; to run Logstash from source using only the OSS-licensed code, export the OSS environment variable with a value of true:

export OSS=true
  • To run Logstash from the repo you must first bootstrap the environment:
rake bootstrap
  • You can then use bin/logstash to start Logstash, but there are no plugins installed. To install default plugins, you can run:
rake plugin:install-default

This will install the 80+ default plugins which makes Logstash ready to connect to multiple data sources, perform transformations and send the results to Elasticsearch and other destinations.

To verify your environment, run the following to send your first event:

bin/logstash -e 'input { stdin { } } output { stdout {} }'

This should start Logstash with stdin input waiting for you to enter an event

hello world
2016-11-11T01:22:14.405+0000 0.0.0.0 hello world

Advanced: Drip Launcher

Drip is a tool that solves the slow JVM startup problem while developing Logstash. The drip script is intended to be a drop-in replacement for the java command. We recommend using drip during development, in particular for running tests. Using drip, the first invocation of a command will not be faster but the subsequent commands will be swift.

To tell logstash to use drip, set the environment variable JAVACMD=`which drip`.

Example (but see the Testing section below before running rspec for the first time):

JAVACMD=`which drip` bin/rspec

Caveats

Drip does not work with STDIN. You cannot use drip for running configs which use the stdin plugin.

Building Logstash Documentation

To build the Logstash Reference (open source content only) on your local machine, clone the following repos:

logstash - contains main docs about core features

logstash-docs - contains generated plugin docs

docs - contains doc build files

Make sure you have the same branch checked out in logstash and logstash-docs. Check out master in the docs repo.

Run the doc build script from within the docs repo. For example:

./build_docs.pl --doc ../logstash/docs/index.asciidoc --chunk=1 -open

Testing

Most of the unit tests in Logstash are written using rspec for the Ruby parts. For the Java parts, we use junit. For testing you can use the test rake tasks and the bin/rspec command, see instructions below:

Core tests

1- To run the core tests you can use the Gradle task:

./gradlew test

or use the rspec tool to run all tests or run a specific test:

bin/rspec
bin/rspec spec/foo/bar_spec.rb

Note that before running the rspec command for the first time you need to set up the RSpec test dependencies by running:

./gradlew bootstrap

2- To run the subset of tests covering the Java codebase only run:

./gradlew javaTests

3- To execute the complete test-suite including the integration tests run:

./gradlew check

Sometimes you might find a change to a piece of Logstash code causes a test to hang. These can be hard to debug.

If you set LS_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" you can connect to a running Logstash with your IDEs debugger which can be a great way of finding the issue.

Plugins tests

To run the tests of all currently installed plugins:

rake test:plugin

You can install the default set of plugins included in the logstash package:

rake test:install-default

Note that if a plugin is installed using the plugin manager bin/logstash-plugin install ... do not forget to also install the plugins development dependencies using the following command after the plugin installation:

bin/logstash-plugin install --development

Building Artifacts

Built artifacts will be placed in the LS_HOME/build directory, and will create the directory if it is not already present.

You can build a Logstash snapshot package as tarball or zip file

./gradlew assembleTarDistribution
./gradlew assembleZipDistribution

OSS-only artifacts can similarly be built with their own gradle tasks:

./gradlew assembleOssTarDistribution
./gradlew assembleOssZipDistribution

You can also build .rpm and .deb, but the fpm tool is required.

rake artifact:rpm
rake artifact:deb

and:

rake artifact:rpm_oss
rake artifact:deb_oss

Using a Custom JRuby Distribution

If you want the build to use a custom JRuby you can do so by setting a path to a custom JRuby distribution's source root via the custom.jruby.path Gradle property.

E.g.

./gradlew clean test -Pcustom.jruby.path="/path/to/jruby"

Project Principles

  • Community: If a newbie has a bad time, it's a bug.
  • Software: Make it work, then make it right, then make it fast.
  • Technology: If it doesn't do a thing today, we can make it do it tomorrow.

Contributing

All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.

Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.

It is more important to me that you are able to contribute.

For more information about contributing, see the CONTRIBUTING file.

You can’t perform that action at this time.