8.0 build #428

worleydl · 2022-05-29T23:41:43Z

This PR brings the plugin up to 8 however we've ran into build issues due to the ES update here: elastic/elasticsearch#67127

…g-to-rank into nate8

worleydl · 2022-05-31T12:29:13Z

Hey @nomoa I was wondering if you had any thoughts on this update.

The tldr; is ES has removed the createClassLoader permission from plugins. That seems to break precompiled derived features, maybe TermStatQuery as well but that seemed to work okay in my local testing. Associated PR: elastic/elasticsearch#67127

Possible workarounds:

Override ES security policy, or local JVM policy. (This works for getting a build but seems impractical for most users)
Refactor offending code to no longer require these permissions

risdenk · 2022-06-20T13:58:51Z

The exception being generated in the integration tests is:

java.util.concurrent.ExecutionException: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "createClassLoader")
	at __randomizedtesting.SeedInfo.seed([79CB2C19B27CDC29:7AAE450520CBAE74]:0)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at com.o19s.es.ltr.action.BaseIntegrationTest.addElement(BaseIntegrationTest.java:123)
	at com.o19s.es.ltr.action.BaseIntegrationTest.addElement(BaseIntegrationTest.java:99)
	at com.o19s.es.ltr.logging.LoggingIT.prepareModels(LoggingIT.java:74)
	at com.o19s.es.ltr.logging.LoggingIT.testFailures(LoggingIT.java:128)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "createClassLoader")
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)
	at java.base/java.security.AccessController.checkPermission(AccessController.java:1068)
	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416)
	at java.base/java.lang.SecurityManager.checkCreateClassLoader(SecurityManager.java:479)
	at java.base/java.lang.ClassLoader.checkCreateClassLoader(ClassLoader.java:371)
	at java.base/java.lang.ClassLoader.checkCreateClassLoader(ClassLoader.java:360)
	at java.base/java.lang.ClassLoader.<init>(ClassLoader.java:456)
	at org.apache.lucene.expressions.js.JavascriptCompiler$Loader.<init>(JavascriptCompiler.java:81)
	at org.apache.lucene.expressions.js.JavascriptCompiler.compileExpression(JavascriptCompiler.java:208)
	at org.apache.lucene.expressions.js.JavascriptCompiler.compile(JavascriptCompiler.java:158)
	at com.o19s.es.ltr.utils.Scripting$1.run(Scripting.java:48)
	at com.o19s.es.ltr.utils.Scripting$1.run(Scripting.java:43)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
	at com.o19s.es.ltr.utils.Scripting.compile(Scripting.java:43)
	at com.o19s.es.ltr.utils.Scripting.compile(Scripting.java:34)
	at com.o19s.es.ltr.feature.store.PrecompiledExpressionFeature.compile(PrecompiledExpressionFeature.java:61)
	at com.o19s.es.ltr.feature.store.StoredFeature.optimize(StoredFeature.java:178)
	at com.o19s.es.ltr.feature.store.StoredFeatureSet.optimize(StoredFeatureSet.java:108)
	at com.o19s.es.ltr.action.TransportFeatureStoreAction.precheck(TransportFeatureStoreAction.java:132)
	at com.o19s.es.ltr.action.TransportFeatureStoreAction.doExecute(TransportFeatureStoreAction.java:79)
	at com.o19s.es.ltr.action.TransportFeatureStoreAction.doExecute(TransportFeatureStoreAction.java:52)
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:79)
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:54)
	at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:170)
	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:105)
	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:83)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:380)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:366)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:34)
	... 42 more

So LTR -> calling the Lucene expressions module from the `Scripting.java' which fails due to the security policy.

Valenzione · 2022-08-10T14:38:18Z

Are there any updates on this? Very interested in this issue 🚀

nathancday · 2022-08-11T15:57:40Z

As of yesterday there is discussion in the linked elastic issue: elastic/elasticsearch#67127

worleydl · 2022-11-02T17:31:59Z

There is no version of the plugin for ES8 prior to ES 8.5 due to internal changes that were needed to preserve compatibility. We apologize for any inconvenience.

Please see #443 for resolution of the issue.

nathancday and others added 8 commits May 20, 2022 07:21

wip towards 8

db40e92

Various fixes

9f34679

More fixes

255b1ab

More fixes

8ac637f

Final test fix?

bcb1700

Merge branch 'main' into nate8

32b338e

Update workflows

ec216e4

Merge branch 'nate8' of https://github.com/o19s/elasticsearch-learnin…

d753e71

…g-to-rank into nate8

worleydl mentioned this pull request May 31, 2022

Es 8 0 0 #409

Closed

This was referenced Aug 11, 2022

elasticsearch-learning-to-rank plugin for ES 8.x #435

Closed

where to find elasticsearch-learning-to-rank plugin for ES 8.0 #419

Closed

grcevski mentioned this pull request Aug 12, 2022

Restrict createClassLoader to modules elastic/elasticsearch#67127

Merged

worleydl closed this Nov 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8.0 build #428

8.0 build #428

worleydl commented May 29, 2022 •

edited

Loading

worleydl commented May 31, 2022

risdenk commented Jun 20, 2022

Valenzione commented Aug 10, 2022

nathancday commented Aug 11, 2022

worleydl commented Nov 2, 2022

8.0 build #428

8.0 build #428

Conversation

worleydl commented May 29, 2022 • edited Loading

worleydl commented May 31, 2022

risdenk commented Jun 20, 2022

Valenzione commented Aug 10, 2022

nathancday commented Aug 11, 2022

worleydl commented Nov 2, 2022

worleydl commented May 29, 2022 •

edited

Loading