Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
[META] Native support for Java plugins #9215
This tracks the phases of the Native support for Java plugins Project.
Beta phase (#10232):
referenced this issue
Mar 12, 2018
referenced this issue
Jun 12, 2018
I will need it to production in 4-5 months.
Where can I find documentation about this new java plugin?
When it should be in production ready stage?
Thanks a lot!
In addition I would to contribute from my knowledge to convert some of the plugin from ruby to java.
@zivziv77, the current plan for this work is to release the Java Plugin API in a series of phases. The first phase is the experimental phase in which we'll solicit feedback on its basic functionality. After that, it would progress to the beta phase where we'd evaluate its stability and then it would be released as GA. I would hope that we would have something within your target timeframe, but I cannot commit to any specific release date.
Right now, that work is happening in the
It may also be worth mentioning that we do already have a number of hybrid Ruby/Java plugins in which the bulk of the code is Java using Ruby shims to interface with the Ruby plugin API in Logstash. The TCP, date, and DLQ plugins are examples of hybrid Ruby/Java plugins if you'd like some examples of how those work.
Thanks for your detailed answer.
Where can I find documentation about how the java api works?
I just think it doesn’t make sense writing hybrid plugin java and ruby if there’s going to be support for full java plugin without ruby.
@zivziv77, we don't have docs for the Java API, yet. It's pretty simple, so perhaps a brief explanation will provide what you need. (Note that these APIs are subject to change, though it is unlikely that they will change significantly.)
Your Redis input will need to implement the
You will also need to provide a constructor that takes two parameters of type
Let me know if that helps or if you have additional questions.
Oh, right. You'll need to add the
I wanted to try to create a filter for two of my (Java) projects (Yauaa and Logparser) so I decided to start by simply trying to build the java filter example.
So I download logstash 6.6.0 and hardcoded the path into the build.gradle file.
It would really help me if you can point me a bit more into the right direction (preferably by updating the readme in the example project).
@nielsbasjes, thank you for experimenting with the Java plugin API. You have noted some deficiencies in the existing documentation which have also been reported here: logstash-plugins/logstash-filter-java_filter_example#1
I am working on improving that documentation. Let me know if my explanation in that issue doesn't get you back up and running.
Ok, So as a first step I tried to simply "get the example running".
In an empty working directory:
First get a really clean logstash
Then get the filter example
I had to create this (documented but missing) file: lib/logstash-filter-java_filter_example_jars.rb
Then build the filter gem
Install the filter
Create test input file
Test config file run-test.conf
When I run this using
@nielsbasjes, thank you for trying out the Java plugin API and leaving feedback on your experience. The problem you encountered when attempting to run Logstash with the example filter is that you were not running Logstash with the Java execution engine (with the
The rest of the steps you took look good. In the beta version of the Java plugin API that will be released with Logstash 6.7.0, many of the manual steps and all of the boilerplate Ruby source files will be automated.
Yes, that does the trick!!
FYI: I have been playing around with the example filter and now I have it working being built fully by Maven
I wanted to create a first 'real' filter and ran into something I do not understand.
My starting point (This tag in my git repo) works on my machine.
using this filter config
The output I get on a file containing "Niels Basjes" is
which is what I expected.
Now I change this code in only 1 single aspect: The javacode no longer lives under
When I repeat the same steps I now get this output:
So to me the underlying error seems to be this:
The place where I find this is in the just about the only piece of Ruby code:
Where the working variant only has one line different: The one near the end
@nielsbasjes, congrats on the first working third-party Java filter!
As for the change to the package of the filter, that does have to be updated in some of the Ruby source files as you noted above. I wasn't sure if that resolved your problem or if you needed something else.
Also, in the beta release of this API which will land in Logstash 6.7.0, all the Ruby source files will be auto-generated so no manual editing of them will be required.
The example filter clearly states (as expected):
The question I have: What is the proper way of failing in case of bad configuration so that a descriptive error message is passed on to the user?
I have tried throwing an IllegalArgumentException but that wasn't shown in the console.