Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CompiledScript throws weird exception that doesn't happen with Invocable #6230

Closed
MikeThomsen opened this issue May 18, 2020 · 7 comments
Closed
Assignees

Comments

@MikeThomsen
Copy link

MikeThomsen commented May 18, 2020

Posted this on the mailing list earlier, but posting here since this appears to get a lot more traffic:

I'm working on updating the Apache NiFi ExecuteScript processor to use CompiledScript for performance reasons, and I'm getting this error message with our unit test. I cannot reproduce it with similar examples using CompiledScript, but as it appears to be related to the internals of JRuby I was wondering if someone could explain to me what JRuby is doing to help debug our issue:

pool-1-thread-1] ERROR org.apache.nifi.processors.script.ExecuteScript - ExecuteScript[id=6824ad59-36a1-49c1-a359-5e83a3ce206c] ExecuteScript[id=6824ad59-36a1-49c1-a359-5e83a3ce206c] failed to process due to java.lang.AssertionError: Setting 1 to org.apache.nifi.util.MockProcessSession@255780a8, O: Static Type[598969490]: block [flowFile=org.apache.nifi.util.MockComponentLog@b9bab94]
  Static Type[1288793622]: local []; rolling back session: java.lang.AssertionError: Setting 1 to org.apache.nifi.util.MockProcessSession@255780a8, O: Static Type[598969490]: block [flowFile=org.apache.nifi.util.MockComponentLog@b9bab94]
  Static Type[1288793622]: local []

java.lang.AssertionError: java.lang.AssertionError: Setting 1 to org.apache.nifi.util.MockProcessSession@255780a8, O: Static Type[598969490]: block [flowFile=org.apache.nifi.util.MockComponentLog@b9bab94]
  Static Type[1288793622]: local []

at org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:220)
at org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:179)
at org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:174)
at org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:169)
at org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:164)
at org.apache.nifi.processors.script.TestExecuteJRuby.testReadFlowFileContentAndStoreInFlowFileAttribute(TestExecuteJRuby.java:53)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
Caused by: java.lang.AssertionError: Setting 1 to org.apache.nifi.util.MockProcessSession@255780a8, O: Static Type[598969490]: block [flowFile=org.apache.nifi.util.MockComponentLog@b9bab94]
  Static Type[1288793622]: local []
at org.jruby.runtime.scope.ManyVarsDynamicScope.assertSetValue(ManyVarsDynamicScope.java:283)
at org.jruby.runtime.scope.ManyVarsDynamicScope.setValueVoid(ManyVarsDynamicScope.java:164)
at org.jruby.runtime.DynamicScope.setValue(DynamicScope.java:349)
at org.jruby.embed.variable.VariableInterceptor.inject(VariableInterceptor.java:137)
at org.jruby.embed.internal.BiVariableMap.inject(BiVariableMap.java:384)
at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:109)
at org.jruby.embed.jsr223.JRubyCompiledScript.eval(JRubyCompiledScript.java:86)
at org.jruby.embed.jsr223.JRubyCompiledScript.eval(JRubyCompiledScript.java:77)
at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:275)
at org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:270)
at org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:264)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

This is the test Ruby code:

java_import Java::org.apache.nifi.processors.script.ExecuteScript
flowFile = session.get()
if flowFile.nil?
    return
end
flowFile = session.putAttribute(flowFile, "from-content", "test content")
session.transfer(flowFile, ExecuteScript::REL_SUCCESS)

This is how we set up the bindings:

Bindings bindings = scriptEngine.getBindings(ScriptContext.ENGINE_SCOPE);
if (bindings == null) {
    bindings = new SimpleBindings();
}
bindings.put("session", session);
bindings.put("context", context);
bindings.put("log", log);
bindings.put("REL_SUCCESS", REL_SUCCESS);
bindings.put("REL_FAILURE", REL_FAILURE);
@MikeThomsen
Copy link
Author

MikeThomsen commented May 22, 2020

@headius @eregon if one of you could take a look at this, I'd appreciate it. I'm trying to update to NiFi's scripting capabilities and don't want to have to leave out compilation support for JRuby unless I have to.

@headius
Copy link
Member

headius commented Jun 11, 2020

This is in my queue for the week.

@headius headius self-assigned this Jun 11, 2020
@headius
Copy link
Member

headius commented Jun 13, 2020

FWIW I'm not sure what performance issue you're trying to deal with, but I'm unsure if CompiledScript will help. At present it will remove some of the overhead of reparsing the script, but it will not go all the way to JVM bytecode, the fastest way to execute.

That probably should be filed as a bug and fixed too.

@MikeThomsen
Copy link
Author

MikeThomsen commented Jun 13, 2020

Thanks for that info. I'll just conditionally use invocable in that code so that it doesn't impact our Groovy users who would see a benefit from that.

@headius
Copy link
Member

headius commented Jun 18, 2020

It's possible this is the same problem as in #6196, which is fixed for 9.2.12. @MikeThomsen Do you think you could push a small repository that shows the problem? I don't want to try to figure out how to run your code snippits and end up running it the wrong way.

I'll mark this for 9.2.12 optimistically.

@headius headius added this to the JRuby 9.2.12.0 milestone Jun 18, 2020
@headius
Copy link
Member

headius commented Jun 26, 2020

We'll call this fixed since we need to get 9.2.12.0 out next week.

@MikeThomsen If you get a chance to confirm, please do.

@headius headius closed this as completed Jun 26, 2020
@MikeThomsen
Copy link
Author

MikeThomsen commented Jun 26, 2020

I grabbed your latest code the other day and started playing with it. I'll let you know if I run into anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants