[META] Native support for Java plugins #9215

andrewvc · 2018-03-08T13:25:37Z

zivziv77 · 2018-12-01T10:33:11Z

Hi,
Sorry for the lake of knowledge. I need to write a Java input and output for logstash. I wonder if writing it directly with the new java plugin api.

I will need it to production in 4-5 months.

Where can I find documentation about this new java plugin?

When it should be in production ready stage?

Thanks a lot!

In addition I would to contribute from my knowledge to convert some of the plugin from ruby to java.

Thanks

Ziv

danhermann · 2018-12-03T13:14:47Z

@zivziv77, the current plan for this work is to release the Java Plugin API in a series of phases. The first phase is the experimental phase in which we'll solicit feedback on its basic functionality. After that, it would progress to the beta phase where we'd evaluate its stability and then it would be released as GA. I would hope that we would have something within your target timeframe, but I cannot commit to any specific release date.

Right now, that work is happening in the java-api branch of the Logstash repository. You can see the current state of the Java Plugin API in the org.logstash.plugins.api package here.

It may also be worth mentioning that we do already have a number of hybrid Ruby/Java plugins in which the bulk of the code is Java using Ruby shims to interface with the Ruby plugin API in Logstash. The TCP, date, and DLQ plugins are examples of hybrid Ruby/Java plugins if you'd like some examples of how those work.

danhermann · 2018-12-03T13:47:33Z

@zivziv77, I should also add that we'd be happy to have any contributions you might make in converting plugins from Ruby to Java!

zivziv77 · 2018-12-03T14:34:55Z

Hi

Thanks for your detailed answer.
The main plugin that I need is Input Redis with cluster support with one of the java Redis client. I thought writing it myself and give it to you also.

Where can I find documentation about how the java api works?

I just think it doesn’t make sense writing hybrid plugin java and ruby if there’s going to be support for full java plugin without ruby.

Thanks

danhermann · 2018-12-03T15:11:10Z

@zivziv77, we don't have docs for the Java API, yet. It's pretty simple, so perhaps a brief explanation will provide what you need. (Note that these APIs are subject to change, though it is unlikely that they will change significantly.)

Your Redis input will need to implement the Input interface. Within the start method, you should create a loop that retrieves messages from Redis and writes them to the supplied QueueWriter instance as a Map<String, Object> that will be later converted to an Event within the Logstash event processing pipeline. In your stop method, you should disconnect from Redis and perform whatever cleanup actions might be required. You can see a very simple example of this basic pattern in the StreamInput class in that same file.

You will also need to provide a constructor that takes two parameters of type Configuration and Context. The Configuration instance should provide your input with configuration values such as the connection parameters for your Redis cluster. You probably won't need to do anything with the provided Context instance.

Let me know if that helps or if you have additional questions.

zivziv77 · 2018-12-03T15:16:44Z

Thank you!
That sounds pretty straight forward.
One question I didn’t notice where should I register my plugin to be loaded ?
How can I work with logstash on the java api nad mark to load for example Redis-input from java and not from ruby?

danhermann · 2018-12-03T15:22:56Z

Oh, right. You'll need to add the @LogstashPlugin(name = "myRedisPluginName") annotation to your class. If you then add input { myRedisPluginName {} } (or whatever you call it) to your Logstash config, it will run your Java plugin. Note that you will need to do this with Logstash built from the java-api branch as that code has not yet been merged into the master branch.

nielsbasjes · 2019-02-06T20:32:28Z

I wanted to try to create a filter for two of my (Java) projects (Yauaa and Logparser) so I decided to start by simply trying to build the java filter example.
Apparently the current documentation (yes I know, experimental) assumes more inside knowledge than I currently have.
So the first attempt failed miserably:

* What went wrong:
A problem occurred evaluating root project 'logstash-filter-java_filter_example'.
> Could not get unknown property 'LOGSTASH_CORE_PATH' for object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

So I download logstash 6.6.0 and hardcoded the path into the build.gradle file.
Now the build runs a bit better but fails with lots of errors indicating it cannot find anything it needs

error: package co.elastic.logstash.api does not exist
error: cannot find symbol
import org.logstash.Event;

It would really help me if you can point me a bit more into the right direction (preferably by updating the readme in the example project).
Note that I do not have any experience in Ruby and/or Gradle yet.

Thanks.

danhermann · 2019-02-06T21:11:31Z

@nielsbasjes, thank you for experimenting with the Java plugin API. You have noted some deficiencies in the existing documentation which have also been reported here: logstash-plugins/logstash-filter-java_filter_example#1

I am working on improving that documentation. Let me know if my explanation in that issue doesn't get you back up and running.

nielsbasjes · 2019-02-09T09:47:43Z

Ok, So as a first step I tried to simply "get the example running".
Important note: I'm pretty experienced in Java, my Ruby/Gradle skills are close to none.
In my mind this is exactly the group you are trying to enable with this Java api.
This is as accurately as I could the description what I have tried.

In an empty working directory:

First get a really clean logstash

git clone --branch 6.6 --single-branch https://github.com/elastic/logstash.git
cd  logstash
./gradlew assemble
rake bootstrap
cd ..

Then get the filter example

git clone https://github.com/logstash-plugins/logstash-filter-java_filter_example
cd logstash-filter-java_filter_example
echo LOGSTASH_CORE_PATH=../logstash/logstash-core/ > gradle.properties

I had to create this (documented but missing) file: lib/logstash-filter-java_filter_example_jars.rb

require 'jar_dependencies'
require_jar('org.logstash.javaapi', 'logstash-filter-java_filter_example', '0.0.1')

Then build the filter gem

./gradlew vendor
gem build *.gemspec
cd ..

Install the filter

./logstash/bin/logstash-plugin install ./logstash-filter-java_filter_example/logstash-filter-java_filter_example-0.0.1.gem

Validating ./logstash-filter-java_filter_example/logstash-filter-java_filter_example-0.0.1.gem
Installing logstash-filter-java_filter_example
Installation successful

Create test input file

echo "Hello World" > /tmp/helloworld.txt

Test config file run-test.conf

input {
  file {
    path => "/tmp/helloworld.txt"
    start_position => "beginning"
  }
}

filter {
  java_filter_example {
    source => "message"
  }
}

output {
  stdout {}
}

When I run this using

./logstash/bin/logstash -f run-test.conf

I get

Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties
[2019-02-09T10:45:46,337][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/queue"}
[2019-02-09T10:45:46,343][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/dead_letter_queue"}
[2019-02-09T10:45:46,618][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-09T10:45:46,627][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-09T10:45:46,653][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"af5f5856-a671-49b8-bc88-de53df75ef7e", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/uuid"}
[2019-02-09T10:45:51,414][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"java.lang.reflect.InvocationTargetException", :backtrace=>["org.logstash.plugins.PluginFactoryExt$Plugins.plugin(PluginFactoryExt.java:284)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(PluginFactoryExt.java:181)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$plugin$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:71)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_EVAL(Interpreter.java:122)", "org.jruby.ir.interpreter.Interpreter.evalCommon(Interpreter.java:176)", "org.jruby.ir.interpreter.Interpreter.evalWithBinding(Interpreter.java:200)", "org.jruby.RubyKernel.evalCommon(RubyKernel.java:1027)", "org.jruby.RubyKernel.eval19(RubyKernel.java:994)", "org.jruby.RubyKernel$INVOKER$s$0$3$eval19.call(RubyKernel$INVOKER$s$0$3$eval19.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$initialize$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:49)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:983)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:974)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$initialize$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:90)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.RubyClass.newInstance(RubyClass.java:1022)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$block$execute$1(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call19(RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$exclusive$1(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:94)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:80)", "org.jruby.runtime.Block.yieldSpecific(Block.java:134)", "org.jruby.ext.thread.Mutex.synchronize(Mutex.java:148)", "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$method$exclusive$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:94)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$method$exclusive$0$VARARGS(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"]}
[2019-02-09T10:45:51,467][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:103:in create'", "org/logstash/execution/ConvergeResultExt.java:34:in add'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:340:in `block in converge_state'"]}
[2019-02-09T10:45:51,531][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

danhermann · 2019-02-11T16:07:40Z

@nielsbasjes, thank you for trying out the Java plugin API and leaving feedback on your experience. The problem you encountered when attempting to run Logstash with the example filter is that you were not running Logstash with the Java execution engine (with the --java-execution command-line flag) which is required for Java plugins. That requirement is not well-documented and I will fix that.

The rest of the steps you took look good. In the beta version of the Java plugin API that will be released with Logstash 6.7.0, many of the manual steps and all of the boilerplate Ruby source files will be automated.

nielsbasjes · 2019-02-11T20:04:19Z

Yes, that does the trick!!

./logstash/bin/logstash --java-execution -f run-test.conf

Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties
[2019-02-11T21:03:04,121][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-11T21:03:04,145][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-11T21:03:08,826][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x2387b22b run>"}
[2019-02-11T21:03:09,069][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/home/nbasjes/workspace/LogStash/logstash/data/plugins/inputs/file/.sincedb_e2e76eb1ecc8816fd37224075296c425", :path=>["/tmp/helloworld.txt"]}
[2019-02-11T21:03:09,097][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-02-11T21:03:09,161][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-02-11T21:03:09,164][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-11T21:03:09,461][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
{
"@timestamp" => 2019-02-11T20:03:09.692Z,
"path" => "/tmp/helloworld.txt",
"host" => "lw7-06766",
"@Version" => "1",
"message" => "dlroW olleH"
}

nielsbasjes · 2019-02-11T22:30:46Z

FYI: I have been playing around with the example filter and now I have it working being built fully by Maven
https://github.com/nielsbasjes/logstash-filter-java_filter_example/tree/BuildUsingMaven

nielsbasjes · 2019-02-13T09:37:18Z

I wanted to create a first 'real' filter and ran into something I do not understand.
So if you guys can help I really appreciate it.

My starting point (This tag in my git repo) works on my machine.
I can do mvn clean package and I then do the steps mentioned before to install and run this filter.

using this filter config

filter {
  reverse_string {
    source => "message"
  }
}

The output I get on a file containing "Niels Basjes" is

{
      "@version" => "1",
    "@timestamp" => 2019-02-13T09:05:40.911Z,
          "host" => "niels-laptop",
       "message" => "sejsaB sleiN",
          "path" => "/tmp/helloworld.txt"
}

which is what I expected.

Now I change this code in only 1 single aspect: The javacode no longer lives under org.example but under nl.example instead.

That is the only change! (Tag in my git repo)

When I repeat the same steps I now get this output:

$ date; ./reload.sh ; date ; ./run.sh
wo 13 feb 2019 10:10:15 CET
Successfully removed logstash-filter-reverse_string
Validating /home/nbasjes/workspace/Prive/logstash/logstash-filter-java_filter_example/target/logstash-filter-reverse_string-0.0.1.gem
Installing logstash-filter-reverse_string
Installation successful
wo 13 feb 2019 10:10:50 CET
Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties
[2019-02-13T10:11:05,597][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-13T10:11:05,611][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-13T10:11:09,685][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"reverse_string", :path=>"logstash/filters/reverse_string", :error_message=>"undefined local variable or method nl' for LogStash::Filters::ReverseString:Class", :error_class=>NameError, :error_backtrace=>["org/jruby/RubyBasicObject.java:1657:in method_missing'", "/home/nbasjes/workspace/LogStash/logstash/vendor/local_gems/30a7a257/logstash-filter-reverse_string-0.0.1/lib/logstash/filters/reverse_string.rb:13:in javaClass'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:271:in add_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:216:in lazy_add'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:189:in legacy_lookup'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:156:in block in lookup'", "org/jruby/ext/thread/Mutex.java:148:in synchronize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:152:in lookup'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:206:in lookup_pipeline_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'", "org/logstash/plugins/PluginFactoryExt.java:222:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:149:in buildFilter'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:37:in execute'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'"]} [2019-02-13T10:11:09,704][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any filter plugin named 'reverse_string'. Are you sure this is correct? Trying to load the reverse_string filter plugin resulted in this error: Problems loading the requested plugin named reverse_string of type filter. Error: NameError undefined local variable or method nl' for LogStash::Filters::ReverseString:Class", :backtrace=>["/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:211:in lookup_pipeline_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'", "org/logstash/plugins/PluginFactoryExt.java:222:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:149:in buildFilter'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:37:in execute'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'"]}
[2019-02-13T10:11:10,002][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

So to me the underlying error seems to be this:

:error_message=>"undefined local variable or method `nl' for LogStash::Filters::ReverseString:Class",

The place where I find this is in the just about the only piece of Ruby code:

$ cat ./bad/lib/logstash/filters/reverse_string.rb

#
# AUTOGENERATED DURING BUILD. DO NOT EDIT.
#
# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"
require "logstash-filter-reverse_string_jars"
require "java"

class LogStash::Filters::ReverseString < LogStash::Filters::Base
  config_name "reverse_string"

  def self.javaClass() nl.example.logstash.ReverseString.java_class; end
end

Where the working variant only has one line different: The one near the end

$ diff ./{bad,good}/lib/logstash/filters/reverse_string.rb

13c13
<   def self.javaClass() nl.example.logstash.ReverseString.java_class; end
---
>   def self.javaClass() org.example.logstash.ReverseString.java_class; end

nielsbasjes · 2019-02-13T11:05:01Z

For now I'm using an ugly workaround.
First working third-party filter?
https://yauaa.basjes.nl/UDF-Logstash.html

danhermann · 2019-02-13T15:01:20Z

@nielsbasjes, congrats on the first working third-party Java filter!

As for the change to the package of the filter, that does have to be updated in some of the Ruby source files as you noted above. I wasn't sure if that resolved your problem or if you needed something else.

Also, in the beta release of this API which will land in Logstash 6.7.0, all the Ruby source files will be auto-generated so no manual editing of them will be required.

nielsbasjes · 2019-03-10T10:29:52Z

The example filter clearly states (as expected):

constructors should validate configuration options

The question I have: What is the proper way of failing in case of bad configuration so that a descriptive error message is passed on to the user?

I have tried throwing an IllegalArgumentException but that wasn't shown in the console.

danhermann · 2019-03-11T11:30:18Z

@nielsbasjes, I typically throw an IllegalArgumentException or IllegalStateException as you did. In the beta release of Java plugin support in the upcoming Logstash 6.7 release, that exception is displayed in the console as you would expect.

nielsbasjes · 2019-03-12T12:32:56Z

Thanks.
Just out of curiosity: What kind of timeline do you guys expect to have this feature released?

danhermann · 2019-03-12T17:42:40Z

The beta release of the Java plugin API will be in the 6.7 release (and is currently committed in the 6.7 and master branches of this repo). We cannot commit to a specific date for the GA release though it is currently targeted for about two months after that.

ylasri · 2019-04-11T12:25:40Z

I tried to compile a new codec plugin, but failed getting the execution right
Could you help on this ?

[2019-04-11T14:23:35,906][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"codec", :name=>"asc_codec", :path=>"logstash/codecs/asc_codec", :error_message=>"\n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command\n\nno such file to load -- org/logstash/javaapi/asn-plugin/0.2.0/asn-plugin-0.2.0 (LoadError)", :error_class=>RuntimeError, :error_backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:356:in do_require'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:265:in block in require_jar'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:307:in require_jar_with_block'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:264:in require_jar'", "C:/es7/logstash/lib/bootstrap/patches/jar_dependencies.rb:6:in require_jar'", "C:/es7/logstash/lib/logstash-codec-asc_codec_jars.rb:2:in

'", "org/jruby/RubyKernel.java:984:in require'", "C:/es7/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require'", "C:/es7/logstash/logstash-core/lib/logstash/codecs/asc_codec.rb:1:in <main>'", "org/jruby/RubyKernel.java:984:in require'", "C:/es7/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:181:in legacy_lookup'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:156:in block in lookup'", "org/jruby/ext/thread/Mutex.java:165:in synchronize'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:152:in lookup'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:206:in lookup_pipeline_plugin'", "C:/es7/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'", "org/logstash/plugins/PluginFactoryExt.java:200:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:308:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:117:in buildInput'", "org/logstash/execution/JavaBasePipelineExt.java:50:in initialize'", "C:/es7/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in initialize'", "C:/es7/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "C:/es7/logstash/logstash-core/lib/logstash/agent.rb:325:in block in converge_state'"]}

Example

encoding: utf-8

require "logstash/codecs/base"
require "logstash/namespace"
require "logstash-codec-asc_codec_jars"
require "java"

class LogStash::Codecs::AsnCodec < LogStash::Codecs::Base
config_name "asc_codec"

def self.javaClass() org.logstash.javaapi.AsnCodec.java_class; end
end

require 'jar_dependencies'
require_jar('org.logstash.javaapi', 'asn-plugin', '0.2.0')

jsvd · 2019-04-11T12:43:13Z

@synapticiel can you please move this comment to a new issue? there is no need to add debugging discussions to this one as it was created to track the progress of implementation. If necessary you can always link that new issue to this one.

nielsbasjes · 2019-05-25T10:13:44Z

The beta support for this was released with version 6.7 several weeks ago. So far the dependencies needed to build a plugin haven't been released to maven central yet.

Can you indicate when this will be done?

Right now I'm unable to automate my plugin build.

danhermann · 2019-05-28T18:17:03Z

@nielsbasjes, we would like to get to the point where we are publishing a minimal jar containing the Java plugin API to Maven, but that involves some non-trivial refactoring of dependencies in Logstash core and changes to our build and deployment processes. We will be releasing the Java plugin API as GA in the upcoming 7.2.0 release of Logstash without publishing those artifacts because we did not want to hold up the overall effort for that.

Until we are able to complete that work, plugin builds can be automated with the new gem gradle task that will perform all necessary steps to compile and package a Java plugin so long as you have copies of both the Logstash repository and your plugin's repository available locally.

nielsbasjes · 2019-05-29T06:10:31Z

Thanks for the update.
Just wondering: Why don't you simply push the current "much too large" dependency to Maven Central as a part of the beta?

danhermann · 2019-05-29T13:01:35Z

Just wondering: Why don't you simply push the current "much too large" dependency to Maven Central as a part of the beta?

@nielsbasjes, that was an option but a big part of the reason we did not do that is that not all of the public classes in the logstash-core jar are part of the plugin API and are therefore not subject to the backward compatibility guarantees within major versions that we make for the plugin API itself. We did not want to give that impression by publishing a jar that contains nearly all of the Java source code for Logstash.

ghost · 2019-07-05T01:47:18Z

I want to use third-party jars in the java-plugins, I want to know if java-plugins supports this feature.Can you give me some help?

danhermann · 2019-07-06T04:21:50Z

@YSZYCF, yes, third-party jars are definitely supported. The example Java plugins use those in several places. For example, see the Apache Commons and Log4j2 dependencies in the Java filter:

https://github.com/logstash-plugins/logstash-filter-java_filter_example/blob/master/build.gradle

ghost · 2019-07-08T03:19:50Z

@danhermann OK

dkm199 · 2019-07-12T23:35:22Z

@danhermann hi.. i am working on a filter plugin in java. I chose to write some part of it in scala classes(for ease of coding) and used that code in java code. Unit tests works fine..configured gradle to compile scala first and then java code.. gem file generated has only one single jar. I was able to install the gem successfully into logstash 7.2 locally.
But while testing the java filter plugin, I noticed, that java part of filter plugin code executes fine, but
scala part of plugin code is not executed at all(I tracked this by setting debug fields into event in java code and scala code)... Do you know what might be going on here? any workarounds are appreciated. Thanks
Java code accesses scala code(ExpressionEvaluator is a scala class) like below and it works fine, in the unit tests that I wrote.
Boolean result = ExpressionEvaluator$.MODULE$.evaluateExpression(event, someString);
also tried accessing scala class like below as well, it didn't help
Boolean result = ExpressionEvaluator.evaluateExpression(event, someString);

dkm199 · 2019-07-13T00:19:17Z

nevermind, I figured out the issue, it needed scala library packaged into the jar as well, once that included, it worked fine.

damozhizhou · 2019-07-25T07:19:36Z

@danhermann ，hello，I used the example you supplyed in “https://www.elastic.co/guide/en/logstash/current/java-codec-plugin.html”，and success get the result by the java_stdin。However，when i set input from kafka，I got the error：

here is my config：

Need your help, thank you very much~

danhermann · 2019-07-25T11:11:29Z

@damozhizhou, if you'll post your question in the Logstash forums and tag me (@danhermann) on it, I'll reply to you there.

danhermann · 2019-07-25T11:13:14Z

Closing this as Java plugins went GA in Logstash 7.2.0. Any additional work on Java plugins will be tracked in new issues.

For help with developing your own Java plugins for Logstash, please post in the Logstash forums. You may tag me (@danhermann) on your post there and I'll try to respond. If you've encountered a bug related to Java plugins, you can create a GitHub issue here.

nielsbasjes · 2019-07-29T08:09:52Z

@danhermann
Congratulations this is now GA.
What is the issue I can track about stripping and deploying the API library to maven?

The current process cannot be ran in a CI environment because of this missing jar.

danhermann · 2019-07-29T10:50:11Z

@nielsbasjes, could you open an issue here so we can track that effort separately?

karthikdeiva · 2020-12-18T07:21:38Z

Hi,

I built a custom JAVA Input plugin using the example JAVA plugin. I am able to install the GEM in the Logstash in my local. When I tried to install the custom plugin -the generated GEM file in my Dev server, getting the below exception,

Received fatal alert: handshake_failure

Could anyone brief on the process of installing the plugin developed in one machine in another Logstash instance.

Thanks,
KarthikDeiva

nielsbasjes · 2020-12-18T08:49:13Z

Have a look at how I did it here https://www.github.com/nielsbasjes/yauaa/tree/master/udfs/logstash/logstash-filter/src/test/docker

I simply install everything in a docker image and check if the output meets my expectations.

Jesse-so-cool · 2021-09-02T03:30:28Z

Hi，
I want to get the @metadata（like %{[@metadata][kafka][topic]} in my custom java output plugin，how can i do？

andrewvc added the discuss label Mar 8, 2018

andrewvc assigned original-brownbear and andrewvc Mar 8, 2018

andrewvc mentioned this issue Mar 12, 2018

[META] Java Plugin API Dependency Management, Compat, and Artifacts #9227

Closed

danhermann mentioned this issue Mar 12, 2018

[WIP] stdin input original-brownbear/logstash#2

Open

andrewvc mentioned this issue Mar 28, 2018

Invalid cast in Accessors#setChild #9295

Closed

andrewvc added the Java API label Apr 10, 2018

andrewvc mentioned this issue May 3, 2018

[META] Java/Maven Plugin Packaging #9521

Closed

4 tasks

andrewvc mentioned this issue May 21, 2018

6.2.4 release broke plugin #9592

Closed

guyboertje mentioned this issue Jun 12, 2018

Ensure that support for custom config setting validation is included in Java API design. #9733

Open

jsvd assigned danhermann and unassigned andrewvc and original-brownbear Dec 1, 2018

danhermann mentioned this issue Dec 16, 2018

[Feature Branch] Java API #9342

Closed

danhermann mentioned this issue Feb 27, 2019

[DISCUSS] Enable pure-Java plugins #7986

Closed

danhermann changed the title ~~[META] Java API~~ [META] Native support for Java plugins Mar 13, 2019

nielsbasjes mentioned this issue Apr 22, 2019

Finalize Logstash plugin nielsbasjes/yauaa#146

Closed

danhermann removed the discuss label Apr 25, 2019

danhermann closed this as completed Jul 25, 2019

nielsbasjes mentioned this issue Jul 29, 2019

Publish Logstash Java dependencies to Maven central #11002

Open

danhermann mentioned this issue Oct 18, 2019

where is rubyUtils.gradle? logstash-plugins/logstash-filter-java_filter_example#13

Open

[META] Native support for Java plugins #9215

[META] Native support for Java plugins #9215

Comments

andrewvc commented Mar 8, 2018 • edited by danhermann

zivziv77 commented Dec 1, 2018

danhermann commented Dec 3, 2018

danhermann commented Dec 3, 2018

zivziv77 commented Dec 3, 2018

danhermann commented Dec 3, 2018

zivziv77 commented Dec 3, 2018

danhermann commented Dec 3, 2018

nielsbasjes commented Feb 6, 2019

danhermann commented Feb 6, 2019

nielsbasjes commented Feb 9, 2019

danhermann commented Feb 11, 2019

nielsbasjes commented Feb 11, 2019

nielsbasjes commented Feb 11, 2019

nielsbasjes commented Feb 13, 2019

nielsbasjes commented Feb 13, 2019

danhermann commented Feb 13, 2019

nielsbasjes commented Mar 10, 2019

danhermann commented Mar 11, 2019

nielsbasjes commented Mar 12, 2019

danhermann commented Mar 12, 2019

ylasri commented Apr 11, 2019

encoding: utf-8

jsvd commented Apr 11, 2019

nielsbasjes commented May 25, 2019

danhermann commented May 28, 2019

nielsbasjes commented May 29, 2019

danhermann commented May 29, 2019

ghost commented Jul 5, 2019

danhermann commented Jul 6, 2019

ghost commented Jul 8, 2019

dkm199 commented Jul 12, 2019 • edited

dkm199 commented Jul 13, 2019

damozhizhou commented Jul 25, 2019

danhermann commented Jul 25, 2019

danhermann commented Jul 25, 2019 • edited

nielsbasjes commented Jul 29, 2019

danhermann commented Jul 29, 2019

karthikdeiva commented Dec 18, 2020

nielsbasjes commented Dec 18, 2020

Jesse-so-cool commented Sep 2, 2021

andrewvc commented Mar 8, 2018 •

edited by danhermann

dkm199 commented Jul 12, 2019 •

edited

danhermann commented Jul 25, 2019 •

edited