Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[META] Native support for Java plugins #9215

Closed
18 of 20 tasks
andrewvc opened this issue Mar 8, 2018 · 39 comments
Closed
18 of 20 tasks

[META] Native support for Java plugins #9215

andrewvc opened this issue Mar 8, 2018 · 39 comments
Assignees
Labels

Comments

@andrewvc
Copy link
Contributor

@andrewvc andrewvc commented Mar 8, 2018

This tracks the phases of the Native support for Java plugins Project.

Experimental phase:

Beta phase (#10232):

  • Plugin config validation
  • Java codecs
    • API consistency between encode and decode methods?
  • Support flush method for Java filters
  • Create Logstash API package with interfaces ('logstash-api') under co.elastic.logstash.api
    • Extract Java interface from Event class
    • Ensure that implementation is dependent on API interfaces, not concrete classes

GA (#10620):

  • Developer guide for Java plugins
  • Gradle task wrappers
  • Implement classloader isolation (beta): #9521
  • Support for publishing Java plugins packaged as gems via Jarvis

Beyond GA?

  • Implement classloader isolation (GA): #9521
  • Port or implement more complex filters: Beats Input, ES Filter, ES Output
@zivziv77
Copy link

@zivziv77 zivziv77 commented Dec 1, 2018

Hi,
Sorry for the lake of knowledge. I need to write a Java input and output for logstash. I wonder if writing it directly with the new java plugin api.

I will need it to production in 4-5 months.

Where can I find documentation about this new java plugin?

When it should be in production ready stage?

Thanks a lot!

In addition I would to contribute from my knowledge to convert some of the plugin from ruby to java.

Thanks

Ziv

@jsvd jsvd assigned danhermann and unassigned andrewvc and original-brownbear Dec 1, 2018
@danhermann
Copy link
Member

@danhermann danhermann commented Dec 3, 2018

@zivziv77, the current plan for this work is to release the Java Plugin API in a series of phases. The first phase is the experimental phase in which we'll solicit feedback on its basic functionality. After that, it would progress to the beta phase where we'd evaluate its stability and then it would be released as GA. I would hope that we would have something within your target timeframe, but I cannot commit to any specific release date.

Right now, that work is happening in the java-api branch of the Logstash repository. You can see the current state of the Java Plugin API in the org.logstash.plugins.api package here.

It may also be worth mentioning that we do already have a number of hybrid Ruby/Java plugins in which the bulk of the code is Java using Ruby shims to interface with the Ruby plugin API in Logstash. The TCP, date, and DLQ plugins are examples of hybrid Ruby/Java plugins if you'd like some examples of how those work.

@danhermann
Copy link
Member

@danhermann danhermann commented Dec 3, 2018

@zivziv77, I should also add that we'd be happy to have any contributions you might make in converting plugins from Ruby to Java!

@zivziv77
Copy link

@zivziv77 zivziv77 commented Dec 3, 2018

Hi

Thanks for your detailed answer.
The main plugin that I need is Input Redis with cluster support with one of the java Redis client. I thought writing it myself and give it to you also.

Where can I find documentation about how the java api works?

I just think it doesn’t make sense writing hybrid plugin java and ruby if there’s going to be support for full java plugin without ruby.

Thanks

@danhermann
Copy link
Member

@danhermann danhermann commented Dec 3, 2018

@zivziv77, we don't have docs for the Java API, yet. It's pretty simple, so perhaps a brief explanation will provide what you need. (Note that these APIs are subject to change, though it is unlikely that they will change significantly.)

Your Redis input will need to implement the Input interface. Within the start method, you should create a loop that retrieves messages from Redis and writes them to the supplied QueueWriter instance as a Map<String, Object> that will be later converted to an Event within the Logstash event processing pipeline. In your stop method, you should disconnect from Redis and perform whatever cleanup actions might be required. You can see a very simple example of this basic pattern in the StreamInput class in that same file.

You will also need to provide a constructor that takes two parameters of type Configuration and Context. The Configuration instance should provide your input with configuration values such as the connection parameters for your Redis cluster. You probably won't need to do anything with the provided Context instance.

Let me know if that helps or if you have additional questions.

@zivziv77
Copy link

@zivziv77 zivziv77 commented Dec 3, 2018

Thank you!
That sounds pretty straight forward.
One question I didn’t notice where should I register my plugin to be loaded ?
How can I work with logstash on the java api nad mark to load for example Redis-input from java and not from ruby?

@danhermann
Copy link
Member

@danhermann danhermann commented Dec 3, 2018

Oh, right. You'll need to add the @LogstashPlugin(name = "myRedisPluginName") annotation to your class. If you then add input { myRedisPluginName {} } (or whatever you call it) to your Logstash config, it will run your Java plugin. Note that you will need to do this with Logstash built from the java-api branch as that code has not yet been merged into the master branch.

@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented Feb 6, 2019

I wanted to try to create a filter for two of my (Java) projects (Yauaa and Logparser) so I decided to start by simply trying to build the java filter example.
Apparently the current documentation (yes I know, experimental) assumes more inside knowledge than I currently have.
So the first attempt failed miserably:

* What went wrong:
A problem occurred evaluating root project 'logstash-filter-java_filter_example'.
> Could not get unknown property 'LOGSTASH_CORE_PATH' for object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

So I download logstash 6.6.0 and hardcoded the path into the build.gradle file.
Now the build runs a bit better but fails with lots of errors indicating it cannot find anything it needs

error: package co.elastic.logstash.api does not exist
error: cannot find symbol
import org.logstash.Event;

It would really help me if you can point me a bit more into the right direction (preferably by updating the readme in the example project).
Note that I do not have any experience in Ruby and/or Gradle yet.

Thanks.

@danhermann
Copy link
Member

@danhermann danhermann commented Feb 6, 2019

@nielsbasjes, thank you for experimenting with the Java plugin API. You have noted some deficiencies in the existing documentation which have also been reported here: logstash-plugins/logstash-filter-java_filter_example#1

I am working on improving that documentation. Let me know if my explanation in that issue doesn't get you back up and running.

@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented Feb 9, 2019

Ok, So as a first step I tried to simply "get the example running".
Important note: I'm pretty experienced in Java, my Ruby/Gradle skills are close to none.
In my mind this is exactly the group you are trying to enable with this Java api.
This is as accurately as I could the description what I have tried.

In an empty working directory:

First get a really clean logstash

git clone --branch 6.6 --single-branch https://github.com/elastic/logstash.git
cd  logstash
./gradlew assemble
rake bootstrap
cd ..

Then get the filter example

git clone https://github.com/logstash-plugins/logstash-filter-java_filter_example
cd logstash-filter-java_filter_example
echo LOGSTASH_CORE_PATH=../logstash/logstash-core/ > gradle.properties

I had to create this (documented but missing) file: lib/logstash-filter-java_filter_example_jars.rb

require 'jar_dependencies'
require_jar('org.logstash.javaapi', 'logstash-filter-java_filter_example', '0.0.1')

Then build the filter gem

./gradlew vendor
gem build *.gemspec
cd ..

Install the filter

./logstash/bin/logstash-plugin install ./logstash-filter-java_filter_example/logstash-filter-java_filter_example-0.0.1.gem

Validating ./logstash-filter-java_filter_example/logstash-filter-java_filter_example-0.0.1.gem
Installing logstash-filter-java_filter_example
Installation successful

Create test input file

echo "Hello World" > /tmp/helloworld.txt

Test config file run-test.conf

input {
  file {
    path => "/tmp/helloworld.txt"
    start_position => "beginning"
  }
}

filter {
  java_filter_example {
    source => "message"
  }
}

output {
  stdout {}
}

When I run this using

./logstash/bin/logstash -f run-test.conf

I get

Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties
[2019-02-09T10:45:46,337][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/queue"}
[2019-02-09T10:45:46,343][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/dead_letter_queue"}
[2019-02-09T10:45:46,618][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-09T10:45:46,627][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-09T10:45:46,653][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"af5f5856-a671-49b8-bc88-de53df75ef7e", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/uuid"}
[2019-02-09T10:45:51,414][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"java.lang.reflect.InvocationTargetException", :backtrace=>["org.logstash.plugins.PluginFactoryExt$Plugins.plugin(PluginFactoryExt.java:284)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(PluginFactoryExt.java:181)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$plugin$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:71)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_EVAL(Interpreter.java:122)", "org.jruby.ir.interpreter.Interpreter.evalCommon(Interpreter.java:176)", "org.jruby.ir.interpreter.Interpreter.evalWithBinding(Interpreter.java:200)", "org.jruby.RubyKernel.evalCommon(RubyKernel.java:1027)", "org.jruby.RubyKernel.eval19(RubyKernel.java:994)", "org.jruby.RubyKernel$INVOKER$s$0$3$eval19.call(RubyKernel$INVOKER$s$0$3$eval19.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$initialize$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:49)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:983)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:974)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$initialize$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:90)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.RubyClass.newInstance(RubyClass.java:1022)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$block$execute$1(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call19(RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$exclusive$1(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:94)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:80)", "org.jruby.runtime.Block.yieldSpecific(Block.java:134)", "org.jruby.ext.thread.Mutex.synchronize(Mutex.java:148)", "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$method$exclusive$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:94)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$method$exclusive$0$VARARGS(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"]}
[2019-02-09T10:45:51,467][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:103:in create'", "org/logstash/execution/ConvergeResultExt.java:34:in add'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:340:in `block in converge_state'"]}
[2019-02-09T10:45:51,531][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

@danhermann
Copy link
Member

@danhermann danhermann commented Feb 11, 2019

@nielsbasjes, thank you for trying out the Java plugin API and leaving feedback on your experience. The problem you encountered when attempting to run Logstash with the example filter is that you were not running Logstash with the Java execution engine (with the --java-execution command-line flag) which is required for Java plugins. That requirement is not well-documented and I will fix that.

The rest of the steps you took look good. In the beta version of the Java plugin API that will be released with Logstash 6.7.0, many of the manual steps and all of the boilerplate Ruby source files will be automated.

@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented Feb 11, 2019

Yes, that does the trick!!

./logstash/bin/logstash --java-execution -f run-test.conf 

Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties
[2019-02-11T21:03:04,121][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-11T21:03:04,145][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-11T21:03:08,826][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x2387b22b run>"}
[2019-02-11T21:03:09,069][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/home/nbasjes/workspace/LogStash/logstash/data/plugins/inputs/file/.sincedb_e2e76eb1ecc8816fd37224075296c425", :path=>["/tmp/helloworld.txt"]}
[2019-02-11T21:03:09,097][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-02-11T21:03:09,161][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-02-11T21:03:09,164][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-11T21:03:09,461][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
{
"@timestamp" => 2019-02-11T20:03:09.692Z,
"path" => "/tmp/helloworld.txt",
"host" => "lw7-06766",
"@Version" => "1",
"message" => "dlroW olleH"
}

@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented Feb 11, 2019

FYI: I have been playing around with the example filter and now I have it working being built fully by Maven
https://github.com/nielsbasjes/logstash-filter-java_filter_example/tree/BuildUsingMaven

@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented Feb 13, 2019

I wanted to create a first 'real' filter and ran into something I do not understand.
So if you guys can help I really appreciate it.

My starting point (This tag in my git repo) works on my machine.
I can do mvn clean package and I then do the steps mentioned before to install and run this filter.

using this filter config

filter {
  reverse_string {
    source => "message"
  }
}

The output I get on a file containing "Niels Basjes" is

{
      "@version" => "1",
    "@timestamp" => 2019-02-13T09:05:40.911Z,
          "host" => "niels-laptop",
       "message" => "sejsaB sleiN",
          "path" => "/tmp/helloworld.txt"
}

which is what I expected.

Now I change this code in only 1 single aspect: The javacode no longer lives under org.example but under nl.example instead.

That is the only change! (Tag in my git repo)

When I repeat the same steps I now get this output:

$ date; ./reload.sh ; date ; ./run.sh
wo 13 feb 2019 10:10:15 CET
Successfully removed logstash-filter-reverse_string
Validating /home/nbasjes/workspace/Prive/logstash/logstash-filter-java_filter_example/target/logstash-filter-reverse_string-0.0.1.gem
Installing logstash-filter-reverse_string
Installation successful
wo 13 feb 2019 10:10:50 CET
Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties
[2019-02-13T10:11:05,597][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-13T10:11:05,611][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-13T10:11:09,685][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"reverse_string", :path=>"logstash/filters/reverse_string", :error_message=>"undefined local variable or method nl' for LogStash::Filters::ReverseString:Class", :error_class=>NameError, :error_backtrace=>["org/jruby/RubyBasicObject.java:1657:in method_missing'", "/home/nbasjes/workspace/LogStash/logstash/vendor/local_gems/30a7a257/logstash-filter-reverse_string-0.0.1/lib/logstash/filters/reverse_string.rb:13:in javaClass'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:271:in add_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:216:in lazy_add'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:189:in legacy_lookup'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:156:in block in lookup'", "org/jruby/ext/thread/Mutex.java:148:in synchronize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:152:in lookup'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:206:in lookup_pipeline_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'", "org/logstash/plugins/PluginFactoryExt.java:222:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:149:in buildFilter'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:37:in execute'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'"]} [2019-02-13T10:11:09,704][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any filter plugin named 'reverse_string'. Are you sure this is correct? Trying to load the reverse_string filter plugin resulted in this error: Problems loading the requested plugin named reverse_string of type filter. Error: NameError undefined local variable or method nl' for LogStash::Filters::ReverseString:Class", :backtrace=>["/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:211:in lookup_pipeline_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'", "org/logstash/plugins/PluginFactoryExt.java:222:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:149:in buildFilter'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:37:in execute'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'"]}
[2019-02-13T10:11:10,002][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

So to me the underlying error seems to be this:

:error_message=>"undefined local variable or method `nl' for LogStash::Filters::ReverseString:Class",

The place where I find this is in the just about the only piece of Ruby code:

$ cat ./bad/lib/logstash/filters/reverse_string.rb

#
# AUTOGENERATED DURING BUILD. DO NOT EDIT.
#
# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"
require "logstash-filter-reverse_string_jars"
require "java"

class LogStash::Filters::ReverseString < LogStash::Filters::Base
  config_name "reverse_string"

  def self.javaClass() nl.example.logstash.ReverseString.java_class; end
end

Where the working variant only has one line different: The one near the end

$ diff ./{bad,good}/lib/logstash/filters/reverse_string.rb

13c13
<   def self.javaClass() nl.example.logstash.ReverseString.java_class; end
---
>   def self.javaClass() org.example.logstash.ReverseString.java_class; end

@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented Feb 13, 2019

For now I'm using an ugly workaround.
First working third-party filter?
https://yauaa.basjes.nl/UDF-Logstash.html

@danhermann
Copy link
Member

@danhermann danhermann commented Feb 13, 2019

@nielsbasjes, congrats on the first working third-party Java filter!

As for the change to the package of the filter, that does have to be updated in some of the Ruby source files as you noted above. I wasn't sure if that resolved your problem or if you needed something else.

Also, in the beta release of this API which will land in Logstash 6.7.0, all the Ruby source files will be auto-generated so no manual editing of them will be required.

@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented Mar 10, 2019

The example filter clearly states (as expected):

constructors should validate configuration options

The question I have: What is the proper way of failing in case of bad configuration so that a descriptive error message is passed on to the user?

I have tried throwing an IllegalArgumentException but that wasn't shown in the console.

@danhermann
Copy link
Member

@danhermann danhermann commented Mar 11, 2019

@nielsbasjes, I typically throw an IllegalArgumentException or IllegalStateException as you did. In the beta release of Java plugin support in the upcoming Logstash 6.7 release, that exception is displayed in the console as you would expect.

@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented Mar 12, 2019

Thanks.
Just out of curiosity: What kind of timeline do you guys expect to have this feature released?

@danhermann
Copy link
Member

@danhermann danhermann commented Mar 12, 2019

The beta release of the Java plugin API will be in the 6.7 release (and is currently committed in the 6.7 and master branches of this repo). We cannot commit to a specific date for the GA release though it is currently targeted for about two months after that.

@danhermann danhermann changed the title [META] Java API [META] Native support for Java plugins Mar 13, 2019
@ylasri
Copy link

@ylasri ylasri commented Apr 11, 2019

I tried to compile a new codec plugin, but failed getting the execution right
Could you help on this ?

[2019-04-11T14:23:35,906][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"codec", :name=>"asc_codec", :path=>"logstash/codecs/asc_codec", :error_message=>"\n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command\n\nno such file to load -- org/logstash/javaapi/asn-plugin/0.2.0/asn-plugin-0.2.0 (LoadError)", :error_class=>RuntimeError, :error_backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:356:in do_require'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:265:in block in require_jar'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:307:in require_jar_with_block'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:264:in require_jar'", "C:/es7/logstash/lib/bootstrap/patches/jar_dependencies.rb:6:in require_jar'", "C:/es7/logstash/lib/logstash-codec-asc_codec_jars.rb:2:in

'", "org/jruby/RubyKernel.java:984:in require'", "C:/es7/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require'", "C:/es7/logstash/logstash-core/lib/logstash/codecs/asc_codec.rb:1:in <main>'", "org/jruby/RubyKernel.java:984:in require'", "C:/es7/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:181:in legacy_lookup'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:156:in block in lookup'", "org/jruby/ext/thread/Mutex.java:165:in synchronize'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:152:in lookup'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:206:in lookup_pipeline_plugin'", "C:/es7/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'", "org/logstash/plugins/PluginFactoryExt.java:200:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:308:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:117:in buildInput'", "org/logstash/execution/JavaBasePipelineExt.java:50:in initialize'", "C:/es7/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in initialize'", "C:/es7/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "C:/es7/logstash/logstash-core/lib/logstash/agent.rb:325:in block in converge_state'"]}

Example

encoding: utf-8

require "logstash/codecs/base"
require "logstash/namespace"
require "logstash-codec-asc_codec_jars"
require "java"

class LogStash::Codecs::AsnCodec < LogStash::Codecs::Base
config_name "asc_codec"

def self.javaClass() org.logstash.javaapi.AsnCodec.java_class; end
end

require 'jar_dependencies'
require_jar('org.logstash.javaapi', 'asn-plugin', '0.2.0')

@jsvd
Copy link
Member

@jsvd jsvd commented Apr 11, 2019

@synapticiel can you please move this comment to a new issue? there is no need to add debugging discussions to this one as it was created to track the progress of implementation. If necessary you can always link that new issue to this one.

@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented May 25, 2019

The beta support for this was released with version 6.7 several weeks ago. So far the dependencies needed to build a plugin haven't been released to maven central yet.

Can you indicate when this will be done?

Right now I'm unable to automate my plugin build.

@danhermann
Copy link
Member

@danhermann danhermann commented May 28, 2019

@nielsbasjes, we would like to get to the point where we are publishing a minimal jar containing the Java plugin API to Maven, but that involves some non-trivial refactoring of dependencies in Logstash core and changes to our build and deployment processes. We will be releasing the Java plugin API as GA in the upcoming 7.2.0 release of Logstash without publishing those artifacts because we did not want to hold up the overall effort for that.

Until we are able to complete that work, plugin builds can be automated with the new gem gradle task that will perform all necessary steps to compile and package a Java plugin so long as you have copies of both the Logstash repository and your plugin's repository available locally.

@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented May 29, 2019

Thanks for the update.
Just wondering: Why don't you simply push the current "much too large" dependency to Maven Central as a part of the beta?

@danhermann
Copy link
Member

@danhermann danhermann commented May 29, 2019

Just wondering: Why don't you simply push the current "much too large" dependency to Maven Central as a part of the beta?

@nielsbasjes, that was an option but a big part of the reason we did not do that is that not all of the public classes in the logstash-core jar are part of the plugin API and are therefore not subject to the backward compatibility guarantees within major versions that we make for the plugin API itself. We did not want to give that impression by publishing a jar that contains nearly all of the Java source code for Logstash.

@ghost
Copy link

@ghost ghost commented Jul 5, 2019

I want to use third-party jars in the java-plugins, I want to know if java-plugins supports this feature.Can you give me some help?

@danhermann
Copy link
Member

@danhermann danhermann commented Jul 6, 2019

@YSZYCF, yes, third-party jars are definitely supported. The example Java plugins use those in several places. For example, see the Apache Commons and Log4j2 dependencies in the Java filter:

https://github.com/logstash-plugins/logstash-filter-java_filter_example/blob/master/build.gradle

@ghost
Copy link

@ghost ghost commented Jul 8, 2019

@dkm199
Copy link

@dkm199 dkm199 commented Jul 12, 2019

@danhermann hi.. i am working on a filter plugin in java. I chose to write some part of it in scala classes(for ease of coding) and used that code in java code. Unit tests works fine..configured gradle to compile scala first and then java code.. gem file generated has only one single jar. I was able to install the gem successfully into logstash 7.2 locally.
But while testing the java filter plugin, I noticed, that java part of filter plugin code executes fine, but
scala part of plugin code is not executed at all(I tracked this by setting debug fields into event in java code and scala code)... Do you know what might be going on here? any workarounds are appreciated. Thanks
Java code accesses scala code(ExpressionEvaluator is a scala class) like below and it works fine, in the unit tests that I wrote.
Boolean result = ExpressionEvaluator$.MODULE$.evaluateExpression(event, someString);
also tried accessing scala class like below as well, it didn't help
Boolean result = ExpressionEvaluator.evaluateExpression(event, someString);

@dkm199
Copy link

@dkm199 dkm199 commented Jul 13, 2019

nevermind, I figured out the issue, it needed scala library packaged into the jar as well, once that included, it worked fine.

@damozhizhou
Copy link

@damozhizhou damozhizhou commented Jul 25, 2019

@danhermann ,hello,I used the example you supplyed in “https://www.elastic.co/guide/en/logstash/current/java-codec-plugin.html”,and success get the result by the java_stdin。However,when i set input from kafka,I got the error:
image
here is my config:
image

Need your help, thank you very much~

@danhermann
Copy link
Member

@danhermann danhermann commented Jul 25, 2019

@damozhizhou, if you'll post your question in the Logstash forums and tag me (@danhermann) on it, I'll reply to you there.

@danhermann
Copy link
Member

@danhermann danhermann commented Jul 25, 2019

Closing this as Java plugins went GA in Logstash 7.2.0. Any additional work on Java plugins will be tracked in new issues.

For help with developing your own Java plugins for Logstash, please post in the Logstash forums. You may tag me (@danhermann) on your post there and I'll try to respond. If you've encountered a bug related to Java plugins, you can create a GitHub issue here.

@danhermann danhermann closed this Jul 25, 2019
@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented Jul 29, 2019

@danhermann
Congratulations this is now GA.
What is the issue I can track about stripping and deploying the API library to maven?

The current process cannot be ran in a CI environment because of this missing jar.

@danhermann
Copy link
Member

@danhermann danhermann commented Jul 29, 2019

@nielsbasjes, could you open an issue here so we can track that effort separately?

@karthikdeiva
Copy link

@karthikdeiva karthikdeiva commented Dec 18, 2020

Hi,

I built a custom JAVA Input plugin using the example JAVA plugin. I am able to install the GEM in the Logstash in my local. When I tried to install the custom plugin -the generated GEM file in my Dev server, getting the below exception,

Received fatal alert: handshake_failure

Could anyone brief on the process of installing the plugin developed in one machine in another Logstash instance.

Thanks,
KarthikDeiva

@nielsbasjes
Copy link

@nielsbasjes nielsbasjes commented Dec 18, 2020

Have a look at how I did it here https://www.github.com/nielsbasjes/yauaa/tree/master/udfs/logstash/logstash-filter/src/test/docker

I simply install everything in a docker image and check if the output meets my expectations.

@Jesse-so-cool
Copy link

@Jesse-so-cool Jesse-so-cool commented Sep 2, 2021

Hi,
I want to get the @metadata(like %{[@metadata][kafka][topic]} in my custom java output plugin,how can i do?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet