Building and running logstash from source

jordansissel edited this page Feb 2, 2012 · 2 revisions

I've seen this come up on the IRC channel quite a few times so I thought it deserved a wiki entry


  • Java 6 (Sun or OpenJDK)

Building logstash from source

It is very easy to build logstash from source (though it's much slower to test this way).

Since I frequently fork projects, I like to keep the original in a different directory. I typically follow the same pattern as github does for naming - author-project.

$ git clone logstash-logstash
$ cd logstash-logstash

At this point you have two options. If you're testing custom changes, it's much quicker to run from the source than build a new jar file.

Building a new jar file

Jordan has been kind enough to provide a (now fixed) Makefile that goes through all the heavy lifting. It should just work out-of-the-box but if it doesn't, please open a ticket. It should not matter but previously I saw issues when attempting to do the build while jruby was my default ruby. It's best if that's not the case. You don't need JRuby to build the jar.

$ cd logstash-logstash
$ make jar

At this point lots of stuff will happen. The general flow of the process that happens is this:

  • Downloads jruby source
  • Builds JRuby
  • Installs JRuby to build/jruby/jruby-<version>
  • Downloads ElasticSearch
  • Pulls jars out of ElasticSearch source
  • Runs JRuby to install the bundler gem
  • Does a bundle install to vendor/bundle
  • Extracts the contents of any third-party jar files (from gems or things like ElasticSearch)
  • Runs a build to create the final jar file
  • Final jar is in build/logstash--monolithic.jar

Building a rubygem

From the root of the source tree, run

make gem

Then you can 'gem install' that gem.

Running from source

However, depending on your machine specs, building a jar can take a while. It's quicker to test running from source.

To do this, you'll need to have jruby installed. The best way to do this is install rvm and then run: rvm install jruby-1.6.5 && rvm use jruby@logstash --create The first command installs jruby. The second command makes jruby your current ruby and also creates a new gemset for it. The reason for creating the gemset is because logstash installs a boatload of gems (for all the different inputs and outputs).

Install bundler (this will be the only gem you need): gem install bundler --no-ri --no-rdoc then bundle install

Once bundler is done pretending to be Maven, let's create a quick test file:

input { stdin { type => 'test' } }
output { stdout { debug => true }}

and save it as logstash.conf

Now run the following: bundle exec ruby --1.9 bin/logstash -f logstash.conf

It will appear as if nothing is happening. Type something and you should see a pretty print of the JSON format of your message:

         "@source" => "stdin://jvx64/",
           "@type" => "test",
           "@tags" => [],
         "@fields" => {},
      "@timestamp" => "2011-12-07T04:08:33.556000Z",
    "@source_host" => "jvx64",
    "@source_path" => "/",
        "@message" => "test"

Now you can stop and start logstash at will to test the agent.

Running the web interface from source

This is a tad bit trickier. You see, the monolithic jar file wraps everything together with a nice Runner main class that handles being both agent and web. Since you're running from source, you don't have that luxury. Let's modify our logstash.conf and start the agent with the embedded ElasticSearch server:

input { stdin { type => "test" } }
output { stdout { debug => true } elasticsearch { embedded => true } }

Now start logstash up again (with minor debug output): bundle exec ruby --1.9 bin/logstash -f logstash.conf -v

You should see something along the lines of this in the output:

[2011-12-06T23:15:17.432000 #3984]  INFO -- : Output registered {"timestamp":"2011-12-06T23:15:17.431000 -0500","plugin":"LogStash::Outputs::ElasticSearch: {\"embedded\"=>true, \"type\"=>\"\", \"tags\"=>[], \"index\"=>\"logstash-%{+YYYY.MM.dd}\", \"index_type\"=>\"%{@type}\", \"port\"=>9300, \"embedded_http_port\"=>\"9200-9300\", \"max_inflight_requests\"=>50}","message":"Output registered","level":"info"}

If you look in the source directory, you'll now have a directory called data. This is created by ES. Now type a few things to get some messages into ES. You should see some debug messages about inserting into ES along with the pretty-print output.

So now we need to test the web interface. Remember when I said that the jar file does some magic for you? One of those things is using intra-JVM communication for running logstash in both agent and web mode. Since we'll be running two distinct processes, we don't have that luxury.

Open a new terminal and change to the source directory. We'll need to "activate" our jruby install:

rvm use jruby@logstash

Now run the following to start the web interface:

bundle exec ruby --1.9 bin/logstash-web --backend "elasticsearch://localhost:9300/"

But that won't work. Here's the thing. ElasticSearch has two protocols it supports. One is the HTTP interface on port 9200. The other is a Java-only protocol called Transport on 9300. Unfortunately, logstash doesn't start the embedded ES server with 9300 enabled. You have two options here.

Option 1 - Disable local mode

  • Edit lib/logstash/outputs/elasticsearch.rb
  • Search the file for builder.local(true)
  • Comment out that line

Now restart your agent and you should be listening on port 9300 as well as 9200.

Option 2 - Run a distinct ES instance

I won't walk through these steps except to say, you need to run the version of ES that logstash is built against. That version is listed at the top of the Makefile.

  • Stop your agent (since it has ports bound) and start elasticsearch in the foreground with the -f option.
  • Edit the logstash.conf ES output to look like so and restart your agent:
elasticsearch { cluster => 'logstash' host => '' port => 9300}

Running the web interface (finally)

Whichever route you decide to go, your web interface will have its backend point to elasticsearch://

At this point, I'm unable to get the logstash-web cli to run reliably via bundler. It's best if you just install the gems it needs manually:

gem install --no-ri --no-rdoc jruby-elasticsearch cabin sinatra mizuno haml

Then start the web interface with:

`bin/logstash-web --backend "elasticsearch://"

Now if you go to http://localhost:9292 in a browser, you should see the test messages you created in the web interface!