NoClassDefFoundError while executing groovy script

[darshanmehta10]

Elasticsearch version: 1.7.3

Plugins installed: [Kopf, Bigdesk]

JVM version: 1.8.0_101

OS version: GNU/Linux

Description of the problem including expected versus actual behavior:
ES update request throws NoClassDefFoundError for sun.security.util.SecurityConstants class after recovering from network outage. We have then tried restarting nodes but are still seeing the error.

There is no change in the scripts/JVM version, the error is thrown for every update request after node recovery/restart.

Steps to reproduce:

1. Send update request with script file param and json document
2. Update fails with error message : org.elasticsearch.script.groovy.GroovyScriptExecutionException: NoClassDefFoundError[Could not initialize class sun.security.util.SecurityConstants]

Provide logs (if relevant):
Failed to index document
org.elasticsearch.ElasticsearchIllegalArgumentException: failed to execute script
at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:202)
at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:176)
at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:170)
at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.run(TransportInstanceSingleOperationAction.java:187)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: NoClassDefFoundError[Could not initialize class sun.security.util.SecurityConstants]
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:274)
at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:198)
... 6 more

[jasontedor]

Sorry, but there is not nearly enough to reproduce here; can you start by sharing your script? I find it dubious that the script engine is attempting to access that class without provocation.

[darshanmehta10]

Hey @jasontedor,

Thanks for the quick reply, the script is just updates a couple of fields in source, e.g.:

if(userId){
ctx._source.user.id = userId
}

if(userName){
ctx._source.user.name = userName
}`

This script hasn't changed recently. Looks like something to do with security/permissions? SecurityConstants.java class does not seem to have anyting that prevents it from being loaded.

[jasontedor]

Yes, it's clearly a permissions issue but we need to come to an understanding why that class is being loaded here. Your script looks innocent, can you please provide a more detailed reproduction?

[darshanmehta10]

Here is the Java code that executes update request:

UpdateRequest updateRequest = new UpdateRequest();
updateRequest.script("update", ScriptType.FILE);
updateRequest.scriptParams(getUpdateScriptParams());
updateRequest.retryOnConflict(5);
updateRequest.upsert(//byte array);

UpdateResponse response = client.update(updateRequest).actionGet();

update.groovy is accessible to elasticsearch (to cross check, I tried changing the file name and got Script not found error). The script itself does not load Security packages/classes. Maybe it's Elasticsearch's groovy engine that tries to load them?

[darshanmehta10]

I am able to execure the script using _search endpoint (as explained here). However, when I try to execute it from Java app (using the code above), it fails with ClassNotFound error. Maybe I need to look into java policy files?

[nik9000]

Just to confirm: that client is a TransportClient, right? You aren't
running ES in the same JVM as your app?

On Oct 7, 2016 9:04 PM, "darshanmehta10" notifications@github.com wrote:

I am able to execure the script using _search endpoint (as explained here
https://www.elastic.co/blog/running-groovy-scripts-without-dynamic-scripting).
However, when I try to execute it from Java app (using the code above), it
fails with ClassNotFound error. Maybe I need to look into java policy files?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#20811 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANLosRSM4EgQIElQ0uVXSWRhmqpI4Erks5qxuwJgaJpZM4KRiT_
.

[darshanmehta10]

Yes, that's correct.

On 8 Oct 2016 2:22 a.m., "Nik Everett" notifications@github.com wrote:

Just to confirm: that client is a TransportClient, right? You aren't
running ES in the same JVM as your app?

On Oct 7, 2016 9:04 PM, "darshanmehta10" notifications@github.com wrote:

I am able to execure the script using _search endpoint (as explained here
<https://www.elastic.co/blog/running-groovy-scripts-
without-dynamic-scripting>).
However, when I try to execute it from Java app (using the code above),
it
fails with ClassNotFound error. Maybe I need to look into java policy
files?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#20811
issuecomment-252392429>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/
AANLosRSM4EgQIElQ0uVXSWRhmqpI4Erks5qxuwJgaJpZM4KRiT_>
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#20811 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AL5qjaKPwA9mKySNAAduSAoMgnrHDbI9ks5qxvBTgaJpZM4KRiT_
.

[jasontedor]

Maybe I need to look into java policy files?

We want to understand why this is happening first, the goal is not to just paper over the problem.

[darshanmehta10]

I have enabled the trace logging in the app and can see the following exception when the app starts up:

java.lang.ClassNotFoundException: groovy.lang.GroovyClassLoader at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_101] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_101] at org.springframework.boot.loader.LaunchedURLClassLoader.doLoadClass(LaunchedURLClassLoader.java:178) ~[elasticsearch-indexer.jar!/:1.1.1.SNAPSHOT] at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:142) ~[elasticsearch-indexer.jar!/:1.1.1.SNAPSHOT] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_101] at org.elasticsearch.script.ScriptModule.configure(ScriptModule.java:94) ~[elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:204) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:85) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:130) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:210) [elasticsearch-1.7.2.jar!/:na]

There is another CNF for LuceneExpression as well:

java.lang.ClassNotFoundException: org.apache.lucene.expressions.Expression at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_101] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_101] at org.springframework.boot.loader.LaunchedURLClassLoader.doLoadClass(LaunchedURLClassLoader.java:178) ~[elasticsearch-indexer.jar!/:1.1.1.SNAPSHOT] at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:142) ~[elasticsearch-indexer.jar!/:1.1.1.SNAPSHOT] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_101] at org.elasticsearch.script.ScriptModule.configure(ScriptModule.java:108) ~[elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:204) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:85) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:130) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:210) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159) [elasticsearch-1.7.2.jar!/:na] at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166) [elasticsearch-1.7.2.jar!/:na]

Not sure whether SecurityConstants error has anything to do with this, but I would add the required dependencies now and will see how it goes.

[darshanmehta10]

After adding the dependencies, CNF error for Expression is gone. However, the original error still persists.

[clintongormley]

It sounds like your installation is messed up. Presumably the node restarted at some stage and all the deps that were previously loaded are now now found?

[darshanmehta10]

Yes, the node restarted itself and we are seeing these errors since then (we haven't made any change in scripts). So, it looks like installation/permissions issue.
However, we are seeing the error from all the other nodes as well (we have 7 node cluster) which is a bit odd. Do you think we should try restarting the affected node (or all the nodes one by one, in the worst case)?

[darshanmehta10]

@clintongormley looks like there was a problem with installation. We have done a rolling restart of all the nodes and that exception is gone now. However, we are now seeing a different error:

org.elasticsearch.ElasticsearchIllegalArgumentException: failed to execute script at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:202) at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:176) at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:170) at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.run(TransportInstanceSingleOperationAction.java:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: NullPointerException[Cannot set property 'id' on null object] at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:274) at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:198)

Below is the json document:

{
    "id": "6uqgrnSyBfVOrEvtexwZig==",
    "createdDate": 1469699160000,
    "title": "Neues Harry Potter Buch",
    "user": {
        "id": "1e54a466-ed26-486f-8771-2480bacb5c80",
        "userName": "hamburgmagazin",
        "name": "hamburgmagazin"
    }
}

Even though it has id and username, we get the NPE while executing the script. Looks similar to this bug.

[clintongormley]

Given that we have an explanation for the original problem, I'm going to close this issue.

Even though it has id and username, we get the NPE while executing the script. Looks similar to this bug.

We never confirmed that #17845 was a bug or not. I was unable to replicate it and I assume the reporter was as well, as we never heard back from them. If you manage to come up with a recreation, then feel free to open a new ticket.