Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[META] Native support for Java plugins #9215

Open
andrewvc opened this Issue Mar 8, 2018 · 20 comments

Comments

Projects
None yet
5 participants
@andrewvc
Copy link
Contributor

andrewvc commented Mar 8, 2018

This tracks the phases of the Native support for Java plugins Project.

Experimental phase:

Beta phase (#10232):

  • Plugin config validation
  • Java codecs
    • API consistency between encode and decode methods?
  • Support flush method for Java filters
  • Create Logstash API package with interfaces ('logstash-api') under co.elastic.logstash.api
    • Extract Java interface from Event class
    • Ensure that implementation is dependent on API interfaces, not concrete classes

GA:

  • Developer guide for Java plugins
  • Gradle task wrappers

Beyond GA:

  • Implement classloader isolation: #9521
  • Port or implement more complex filters: Beats Input, ES Filter, ES Output
@zivziv77

This comment has been minimized.

Copy link

zivziv77 commented Dec 1, 2018

Hi,
Sorry for the lake of knowledge. I need to write a Java input and output for logstash. I wonder if writing it directly with the new java plugin api.

I will need it to production in 4-5 months.

Where can I find documentation about this new java plugin?

When it should be in production ready stage?

Thanks a lot!

In addition I would to contribute from my knowledge to convert some of the plugin from ruby to java.

Thanks

Ziv

@jsvd jsvd assigned danhermann and unassigned andrewvc and original-brownbear Dec 1, 2018

@danhermann

This comment has been minimized.

Copy link
Member

danhermann commented Dec 3, 2018

@zivziv77, the current plan for this work is to release the Java Plugin API in a series of phases. The first phase is the experimental phase in which we'll solicit feedback on its basic functionality. After that, it would progress to the beta phase where we'd evaluate its stability and then it would be released as GA. I would hope that we would have something within your target timeframe, but I cannot commit to any specific release date.

Right now, that work is happening in the java-api branch of the Logstash repository. You can see the current state of the Java Plugin API in the org.logstash.plugins.api package here.

It may also be worth mentioning that we do already have a number of hybrid Ruby/Java plugins in which the bulk of the code is Java using Ruby shims to interface with the Ruby plugin API in Logstash. The TCP, date, and DLQ plugins are examples of hybrid Ruby/Java plugins if you'd like some examples of how those work.

@danhermann

This comment has been minimized.

Copy link
Member

danhermann commented Dec 3, 2018

@zivziv77, I should also add that we'd be happy to have any contributions you might make in converting plugins from Ruby to Java!

@zivziv77

This comment has been minimized.

Copy link

zivziv77 commented Dec 3, 2018

Hi

Thanks for your detailed answer.
The main plugin that I need is Input Redis with cluster support with one of the java Redis client. I thought writing it myself and give it to you also.

Where can I find documentation about how the java api works?

I just think it doesn’t make sense writing hybrid plugin java and ruby if there’s going to be support for full java plugin without ruby.

Thanks

@danhermann

This comment has been minimized.

Copy link
Member

danhermann commented Dec 3, 2018

@zivziv77, we don't have docs for the Java API, yet. It's pretty simple, so perhaps a brief explanation will provide what you need. (Note that these APIs are subject to change, though it is unlikely that they will change significantly.)

Your Redis input will need to implement the Input interface. Within the start method, you should create a loop that retrieves messages from Redis and writes them to the supplied QueueWriter instance as a Map<String, Object> that will be later converted to an Event within the Logstash event processing pipeline. In your stop method, you should disconnect from Redis and perform whatever cleanup actions might be required. You can see a very simple example of this basic pattern in the StreamInput class in that same file.

You will also need to provide a constructor that takes two parameters of type Configuration and Context. The Configuration instance should provide your input with configuration values such as the connection parameters for your Redis cluster. You probably won't need to do anything with the provided Context instance.

Let me know if that helps or if you have additional questions.

@zivziv77

This comment has been minimized.

Copy link

zivziv77 commented Dec 3, 2018

Thank you!
That sounds pretty straight forward.
One question I didn’t notice where should I register my plugin to be loaded ?
How can I work with logstash on the java api nad mark to load for example Redis-input from java and not from ruby?

@danhermann

This comment has been minimized.

Copy link
Member

danhermann commented Dec 3, 2018

Oh, right. You'll need to add the @LogstashPlugin(name = "myRedisPluginName") annotation to your class. If you then add input { myRedisPluginName {} } (or whatever you call it) to your Logstash config, it will run your Java plugin. Note that you will need to do this with Logstash built from the java-api branch as that code has not yet been merged into the master branch.

@nielsbasjes

This comment has been minimized.

Copy link

nielsbasjes commented Feb 6, 2019

I wanted to try to create a filter for two of my (Java) projects (Yauaa and Logparser) so I decided to start by simply trying to build the java filter example.
Apparently the current documentation (yes I know, experimental) assumes more inside knowledge than I currently have.
So the first attempt failed miserably:

* What went wrong:
A problem occurred evaluating root project 'logstash-filter-java_filter_example'.
> Could not get unknown property 'LOGSTASH_CORE_PATH' for object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

So I download logstash 6.6.0 and hardcoded the path into the build.gradle file.
Now the build runs a bit better but fails with lots of errors indicating it cannot find anything it needs

error: package co.elastic.logstash.api does not exist
error: cannot find symbol
import org.logstash.Event;

It would really help me if you can point me a bit more into the right direction (preferably by updating the readme in the example project).
Note that I do not have any experience in Ruby and/or Gradle yet.

Thanks.

@danhermann

This comment has been minimized.

Copy link
Member

danhermann commented Feb 6, 2019

@nielsbasjes, thank you for experimenting with the Java plugin API. You have noted some deficiencies in the existing documentation which have also been reported here: logstash-plugins/logstash-filter-java_filter_example#1

I am working on improving that documentation. Let me know if my explanation in that issue doesn't get you back up and running.

@nielsbasjes

This comment has been minimized.

Copy link

nielsbasjes commented Feb 9, 2019

Ok, So as a first step I tried to simply "get the example running".
Important note: I'm pretty experienced in Java, my Ruby/Gradle skills are close to none.
In my mind this is exactly the group you are trying to enable with this Java api.
This is as accurately as I could the description what I have tried.

In an empty working directory:

First get a really clean logstash

git clone --branch 6.6 --single-branch https://github.com/elastic/logstash.git
cd  logstash
./gradlew assemble
rake bootstrap
cd ..

Then get the filter example

git clone https://github.com/logstash-plugins/logstash-filter-java_filter_example
cd logstash-filter-java_filter_example
echo LOGSTASH_CORE_PATH=../logstash/logstash-core/ > gradle.properties

I had to create this (documented but missing) file: lib/logstash-filter-java_filter_example_jars.rb

require 'jar_dependencies'
require_jar('org.logstash.javaapi', 'logstash-filter-java_filter_example', '0.0.1')

Then build the filter gem

./gradlew vendor
gem build *.gemspec
cd ..

Install the filter

./logstash/bin/logstash-plugin install ./logstash-filter-java_filter_example/logstash-filter-java_filter_example-0.0.1.gem

Validating ./logstash-filter-java_filter_example/logstash-filter-java_filter_example-0.0.1.gem
Installing logstash-filter-java_filter_example
Installation successful

Create test input file

echo "Hello World" > /tmp/helloworld.txt

Test config file run-test.conf

input {
  file {
    path => "/tmp/helloworld.txt"
    start_position => "beginning"
  }
}

filter {
  java_filter_example {
    source => "message"
  }
}

output {
  stdout {}
}

When I run this using

./logstash/bin/logstash -f run-test.conf

I get

Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties
[2019-02-09T10:45:46,337][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/queue"}
[2019-02-09T10:45:46,343][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/dead_letter_queue"}
[2019-02-09T10:45:46,618][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-09T10:45:46,627][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-09T10:45:46,653][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"af5f5856-a671-49b8-bc88-de53df75ef7e", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/uuid"}
[2019-02-09T10:45:51,414][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"java.lang.reflect.InvocationTargetException", :backtrace=>["org.logstash.plugins.PluginFactoryExt$Plugins.plugin(PluginFactoryExt.java:284)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(PluginFactoryExt.java:181)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$plugin$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:71)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_EVAL(Interpreter.java:122)", "org.jruby.ir.interpreter.Interpreter.evalCommon(Interpreter.java:176)", "org.jruby.ir.interpreter.Interpreter.evalWithBinding(Interpreter.java:200)", "org.jruby.RubyKernel.evalCommon(RubyKernel.java:1027)", "org.jruby.RubyKernel.eval19(RubyKernel.java:994)", "org.jruby.RubyKernel$INVOKER$s$0$3$eval19.call(RubyKernel$INVOKER$s$0$3$eval19.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$initialize$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:49)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:983)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:974)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$initialize$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:90)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.RubyClass.newInstance(RubyClass.java:1022)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$block$execute$1(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call19(RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$exclusive$1(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:94)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:80)", "org.jruby.runtime.Block.yieldSpecific(Block.java:134)", "org.jruby.ext.thread.Mutex.synchronize(Mutex.java:148)", "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$method$exclusive$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:94)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$method$exclusive$0$VARARGS(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"]}
[2019-02-09T10:45:51,467][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:103:in create'", "org/logstash/execution/ConvergeResultExt.java:34:in add'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:340:in `block in converge_state'"]}
[2019-02-09T10:45:51,531][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

@danhermann

This comment has been minimized.

Copy link
Member

danhermann commented Feb 11, 2019

@nielsbasjes, thank you for trying out the Java plugin API and leaving feedback on your experience. The problem you encountered when attempting to run Logstash with the example filter is that you were not running Logstash with the Java execution engine (with the --java-execution command-line flag) which is required for Java plugins. That requirement is not well-documented and I will fix that.

The rest of the steps you took look good. In the beta version of the Java plugin API that will be released with Logstash 6.7.0, many of the manual steps and all of the boilerplate Ruby source files will be automated.

@nielsbasjes

This comment has been minimized.

Copy link

nielsbasjes commented Feb 11, 2019

Yes, that does the trick!!

./logstash/bin/logstash --java-execution -f run-test.conf 

Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties
[2019-02-11T21:03:04,121][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-11T21:03:04,145][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-11T21:03:08,826][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x2387b22b run>"}
[2019-02-11T21:03:09,069][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/home/nbasjes/workspace/LogStash/logstash/data/plugins/inputs/file/.sincedb_e2e76eb1ecc8816fd37224075296c425", :path=>["/tmp/helloworld.txt"]}
[2019-02-11T21:03:09,097][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-02-11T21:03:09,161][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-02-11T21:03:09,164][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-11T21:03:09,461][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
{
"@timestamp" => 2019-02-11T20:03:09.692Z,
"path" => "/tmp/helloworld.txt",
"host" => "lw7-06766",
"@Version" => "1",
"message" => "dlroW olleH"
}

@nielsbasjes

This comment has been minimized.

Copy link

nielsbasjes commented Feb 11, 2019

FYI: I have been playing around with the example filter and now I have it working being built fully by Maven
https://github.com/nielsbasjes/logstash-filter-java_filter_example/tree/BuildUsingMaven

@nielsbasjes

This comment has been minimized.

Copy link

nielsbasjes commented Feb 13, 2019

I wanted to create a first 'real' filter and ran into something I do not understand.
So if you guys can help I really appreciate it.

My starting point (This tag in my git repo) works on my machine.
I can do mvn clean package and I then do the steps mentioned before to install and run this filter.

using this filter config

filter {
  reverse_string {
    source => "message"
  }
}

The output I get on a file containing "Niels Basjes" is

{
      "@version" => "1",
    "@timestamp" => 2019-02-13T09:05:40.911Z,
          "host" => "niels-laptop",
       "message" => "sejsaB sleiN",
          "path" => "/tmp/helloworld.txt"
}

which is what I expected.

Now I change this code in only 1 single aspect: The javacode no longer lives under org.example but under nl.example instead.

That is the only change! (Tag in my git repo)

When I repeat the same steps I now get this output:

$ date; ./reload.sh ; date ; ./run.sh
wo 13 feb 2019 10:10:15 CET
Successfully removed logstash-filter-reverse_string
Validating /home/nbasjes/workspace/Prive/logstash/logstash-filter-java_filter_example/target/logstash-filter-reverse_string-0.0.1.gem
Installing logstash-filter-reverse_string
Installation successful
wo 13 feb 2019 10:10:50 CET
Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties
[2019-02-13T10:11:05,597][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-13T10:11:05,611][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-13T10:11:09,685][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"reverse_string", :path=>"logstash/filters/reverse_string", :error_message=>"undefined local variable or method nl' for LogStash::Filters::ReverseString:Class", :error_class=>NameError, :error_backtrace=>["org/jruby/RubyBasicObject.java:1657:in method_missing'", "/home/nbasjes/workspace/LogStash/logstash/vendor/local_gems/30a7a257/logstash-filter-reverse_string-0.0.1/lib/logstash/filters/reverse_string.rb:13:in javaClass'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:271:in add_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:216:in lazy_add'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:189:in legacy_lookup'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:156:in block in lookup'", "org/jruby/ext/thread/Mutex.java:148:in synchronize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:152:in lookup'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:206:in lookup_pipeline_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'", "org/logstash/plugins/PluginFactoryExt.java:222:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:149:in buildFilter'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:37:in execute'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'"]} [2019-02-13T10:11:09,704][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any filter plugin named 'reverse_string'. Are you sure this is correct? Trying to load the reverse_string filter plugin resulted in this error: Problems loading the requested plugin named reverse_string of type filter. Error: NameError undefined local variable or method nl' for LogStash::Filters::ReverseString:Class", :backtrace=>["/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:211:in lookup_pipeline_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'", "org/logstash/plugins/PluginFactoryExt.java:222:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:149:in buildFilter'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:37:in execute'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'"]}
[2019-02-13T10:11:10,002][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

So to me the underlying error seems to be this:

:error_message=>"undefined local variable or method `nl' for LogStash::Filters::ReverseString:Class",

The place where I find this is in the just about the only piece of Ruby code:

$ cat ./bad/lib/logstash/filters/reverse_string.rb

#
# AUTOGENERATED DURING BUILD. DO NOT EDIT.
#
# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"
require "logstash-filter-reverse_string_jars"
require "java"

class LogStash::Filters::ReverseString < LogStash::Filters::Base
  config_name "reverse_string"

  def self.javaClass() nl.example.logstash.ReverseString.java_class; end
end

Where the working variant only has one line different: The one near the end

$ diff ./{bad,good}/lib/logstash/filters/reverse_string.rb

13c13
<   def self.javaClass() nl.example.logstash.ReverseString.java_class; end
---
>   def self.javaClass() org.example.logstash.ReverseString.java_class; end
@nielsbasjes

This comment has been minimized.

Copy link

nielsbasjes commented Feb 13, 2019

For now I'm using an ugly workaround.
First working third-party filter?
https://yauaa.basjes.nl/UDF-Logstash.html

@danhermann

This comment has been minimized.

Copy link
Member

danhermann commented Feb 13, 2019

@nielsbasjes, congrats on the first working third-party Java filter!

As for the change to the package of the filter, that does have to be updated in some of the Ruby source files as you noted above. I wasn't sure if that resolved your problem or if you needed something else.

Also, in the beta release of this API which will land in Logstash 6.7.0, all the Ruby source files will be auto-generated so no manual editing of them will be required.

@nielsbasjes

This comment has been minimized.

Copy link

nielsbasjes commented Mar 10, 2019

The example filter clearly states (as expected):

constructors should validate configuration options

The question I have: What is the proper way of failing in case of bad configuration so that a descriptive error message is passed on to the user?

I have tried throwing an IllegalArgumentException but that wasn't shown in the console.

@danhermann

This comment has been minimized.

Copy link
Member

danhermann commented Mar 11, 2019

@nielsbasjes, I typically throw an IllegalArgumentException or IllegalStateException as you did. In the beta release of Java plugin support in the upcoming Logstash 6.7 release, that exception is displayed in the console as you would expect.

@nielsbasjes

This comment has been minimized.

Copy link

nielsbasjes commented Mar 12, 2019

Thanks.
Just out of curiosity: What kind of timeline do you guys expect to have this feature released?

@danhermann

This comment has been minimized.

Copy link
Member

danhermann commented Mar 12, 2019

The beta release of the Java plugin API will be in the 6.7 release (and is currently committed in the 6.7 and master branches of this repo). We cannot commit to a specific date for the GA release though it is currently targeted for about two months after that.

@danhermann danhermann changed the title [META] Java API [META] Native support for Java plugins Mar 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.