Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
src
README.md
pom.xml

README.md

Integrate logstash ruby transform script tests with maven

logstash allows to transform events before sending to destination using filters. One of the powerful filter is the ruby filter. You can write the transformation using the ruby script into a separate file. On top of that you can write tests which gets executed when logstash starts. However, there is no direct way documented to achive the following:

  • Use existing logstash filters like mutate, date etc. in the ruby script.
  • Write tests for ruby script and execute those as part of maven build.

Using existing logstash filters in the ruby script

The logstash filters are mostly writen in ruby itself and can be eaily referenced from the ruby script. If the filter does not have any logstash specific dependency then it can be instantiated and used. Refer to es_transform.rb which uses the mutate and date filter plugins. It simple includes the filter files as:

require "logstash/filters/mutate"
require "logstash/filters/date"
require "logstash/logging/logger"

Then you can create the instance of the filter as:

mutate_filter_config = {}

# Add configuration here like
# mutate_filter_config['rename'] = {'old_field' => 'new_field'}

instance = LogStash::Filters::Mutate.new(mutate_filter_config)
instance.register()

The configuration required per filter is mentioned in the documentation. Make sure to call register on the filter instance as some plugins may do required initialization in this call. You need to then just call filter(event) method on the filter instance.

Writing tests for ruby filter script and integrating with the build

logstash ruby filter allows writing inline tests to validate the transformation. These tests are run when logstash starts. While this suffices in most of the cases there are few issues:

  • The tests are part of the ruby script which may become unmanageable as they grow.
  • You need to install or run logstash to validate the tests.
  • Becomes difficult to integrate with the build.

Approach for running tests outside logstash

The ruby filter script is just like any other ruby script. Only thing required to run it outside logstash is to take care of the dependencies. The dependencies are in terms of other ruby script files (or gems) or even jar files. To see how these dependencies have been handled let's refer to pom.xml. Here, we are using the exec-maven-plugin to run the ruby script. Look at the plugin with id Execute ES Data Transform Tests.

ruby dependencies

This can be handled by setting the GEM_HOME environment variable. To get the same dependencies as logstash we copy the logstash folder to maven project location and refer to the gems path. The gems path will be <logstash_folder>/vendor/bundle/jruby/2.3.0. The logstash gem file uses local path for logstash-core and logstash-core-plugin-api which was not getting resolved even after specify the gem file for logstash installation. Hence, as a workaround we download the logstash-core-plugin-api gems which also pulls logstash-core.

jar dependencies

The jar dependencies needs to be taken care because core classes like logstash event etc. are in the logstash-core jar file. Also, we will need other dependencies like fasterxml jars, jruby-complete jar etc. all of these jars are available in logstash deployment folder. For executing the java application we can specify the jar file paths but java does not search for the sub-folders in the path. So the option is to specify each jar file path explicitly or have all of them in a single folder. We went ahead with second option. Refer to plugin with ID as Copy jars in pom.xml. Here, we are using groovy script to copy the jar from two locations into local_jars folder. Once the jars are available in local_jars folder we use following command to execute the ruby script:

java -cp ${basedir}/local_jars/* org.jruby.Main ${basedir}/src/test/ruby/core/es_transform_tests.rb

Above will execute the mentioned ruby script i.e. es_transform_tests.rb. This is the main script which includes other tests files and dynamically run them. The script uses following conventions to detect and run the tests:

Running the tests

Since we have used the exec-maven-plugin the tests will now run as part of the maven build using following command:

mvn install

If there are test failures then the maven build will fail. Only thing missing here is that we won't get an xml or html test execution report but that is not hard to implement.

You can’t perform that action at this time.