While Enabling Javaagent For Newrelic In Apache Solr v9.2 Pods Are Not Running

[pavanKumar2211]

We are deploying apache solr operator https://github.com/apache/solr-operator in AWS EKS 1.23 via helm
Currently we are using:
Apache Solr operator Image: v0.5.1
Apache Solrcloud Image: v8.8

Now we are upgrading to 9.2 which is compatible with operator 0.5.1

We are installing newrelic agent by downloading newrelic.jar file from https://download.newrelic.com/newrelic/java-agent/newrelic-agent/current/newrelic-java.zip

Under solrOptions to execute newrelic jar we are giving javaOpts="-javaagent:/path/to/newrelic.jar" for solr 8.8. Now we are upgrading to 9.2 which is compatible with operator 0.5.1. With the solrOptions in solr v8.8 the apache-solrcloud pods are running fine and solr is being traced over newrelic but using same configuration for solr v9.2 apache-solrcloud pods are failing to start.

Attaching complete error stacktrace

We are getting below error:

Starting Solr
Java 17 detected. Enabled workaround for SOLR-16463
[0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
WARNING: A command line option has enabled the Security Manager
WARNING: The Security Manager is deprecated and will be removed in a future release
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
2023-05-26T10:06:38,650+0000 [??? 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/opt/solr-9.2.1/fnp-libs/./newrelic.yml"
2023-05-26T10:06:38,731+0000 [??? 1] com.newrelic INFO: Using default collector host: collector.newrelic.com
2023-05-26T10:06:38,731+0000 [??? 1] com.newrelic INFO: Using default metric ingest URI: https://metric-api.newrelic.com/metric/v1
2023-05-26T10:06:38,732+0000 [??? 1] com.newrelic INFO: Using default event ingest URI: https://insights-collector.newrelic.com/v1/accounts/events
2023-05-26T10:06:38,762+0000 [??? 1] com.newrelic WARN: Error parsing labels - empty name or value
2023-05-26T10:06:38,762+0000 [??? 1] com.newrelic WARN: Labels will not be sent to New Relic
2023-05-26T10:06:38,816+0000 [??? 1] com.newrelic INFO: New Relic Agent: Writing to log file: /opt/solr-9.2.1/fnp-libs/logs/newrelic_agent.log
Unable to start New Relic Agent. Please remove -javaagent from your startup arguments and contact New Relic support.
java.security.AccessControlException: access denied ("java.io.FilePermission" "/proc/sys/kernel/random/boot_id" "read")
at java.base/java.security.AccessControlContext.checkPermission(Unknown Source)
at java.base/java.security.AccessController.checkPermission(Unknown Source)
at java.base/java.lang.SecurityManager.checkPermission(Unknown Source)
at java.base/java.lang.SecurityManager.checkRead(Unknown Source)
at java.base/java.io.FileInputStream.(Unknown Source)
at com.newrelic.agent.deps.com.google.common.io.Files$FileByteSource.openStream(Files.java:132)
at com.newrelic.agent.deps.com.google.common.io.Files$FileByteSource.read(Files.java:156)
at com.newrelic.agent.deps.com.google.common.io.ByteSource$AsCharSource.read(ByteSource.java:486)
at com.newrelic.agent.utilization.DataFetcher.getBootId(DataFetcher.java:121)
at com.newrelic.agent.utilization.UtilizationService.(UtilizationService.java:94)
at com.newrelic.agent.service.ServiceManagerImpl.doStart(ServiceManagerImpl.java:225)
at com.newrelic.agent.service.AbstractService.start(AbstractService.java:63)
at com.newrelic.agent.Agent.continuePremain(Agent.java:166)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at com.newrelic.bootstrap.BootstrapAgent.startAgent(BootstrapAgent.java:172)
at com.newrelic.bootstrap.BootstrapAgent.premain(BootstrapAgent.java:114)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown Source)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown Source)
Error bootstrapping New Relic agent: java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at com.newrelic.bootstrap.BootstrapAgent.startAgent(BootstrapAgent.java:172)
at com.newrelic.bootstrap.BootstrapAgent.premain(BootstrapAgent.java:114)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown Source)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown Source)
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "exitVM.1")
at java.base/java.security.AccessControlContext.checkPermission(Unknown Source)
at java.base/java.security.AccessController.checkPermission(Unknown Source)
at java.base/java.lang.SecurityManager.checkPermission(Unknown Source)
at java.base/java.lang.SecurityManager.checkExit(Unknown Source)
at java.base/java.lang.Runtime.exit(Unknown Source)
at java.base/java.lang.System.exit(Unknown Source)
at com.newrelic.agent.Agent.continuePremain(Agent.java:222)
... 12 more
2023-05-26 10:06:42.659 INFO (main) [ ] o.e.j.s.Server jetty-10.0.13; built: 2022-12-07T20:13:20.134Z; git: 1c2636ea05c0ca8de1ffd6ca7f3a98ac084c766d; jvm 17.0.6+10
2023-05-26 10:06:42.699 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///opt/solr-9.2.1/server/contexts/]
2023-05-26 10:06:43.152 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
2023-05-26 10:06:43.168 WARN (main) [ ] o.e.j.w.WebAppContext Failed startup of context o.e.j.w.WebAppContext@8432469{solr-jetty-context.xml,/solr,file:///opt/solr-9.2.1/server/solr-webapp/webapp/,UNAVAILABLE}{/opt/solr-9.2.1/server/solr-webapp/webapp} => java.lang.VerifyError: Bad type on operand stack
Exception Details:
java.lang.VerifyError: Bad type on operand stack
Exception Details:
Location:
org/eclipse/jetty/servlet/listener/ELContextCleaner.contextDestroyed(Ljavax/servlet/ServletContextEvent;)V @157: invokeinterface
Reason:
Type 'java/lang/Object' (current frame, stack[2]) is not assignable to 'java/lang/Throwable'
Current Frame:
bci: @157
flags: { }
locals: { 'org/eclipse/jetty/servlet/listener/ELContextCleaner', 'javax/servlet/ServletContextEvent', 'com/newrelic/agent/bridge/ExitTracer', integer, top, 'javax/servlet/ServletContextEvent', 'org/eclipse/jetty/servlet/listener/ELContextCleaner', 'java/lang/Object' }
stack: { 'org/slf4j/Logger', 'java/lang/String', 'java/lang/Object' }
Bytecode:
0000000: 014d b200 272a 1007 0110 0eb9 003b 0500
0000010: 4da7 0004 5703 3eb8 003f b900 4501 00b2
0000020: 004b 0312 4d05 bd00 4f59 032a b600 53b6
0000030: 0059 5359 0412 7b53 b900 6005 0057 a700
0000040: 163a 04b2 0027 1904 1228 b900 2e03 00a7
0000050: 0003 043e 2a2b 3a05 3a06 127d b800 833a
0000060: 0719 0619 07b6 0087 3a08 1908 04b6 008d
0000070: 1906 1908 b600 91b2 0093 b900 9901 0099
0000080: 000d b200 9312 9bb9 009f 0200 a700 253a
0000090: 07a7 0020 3a07 b200 9312 a119 07b9 00a5
00000a0: 0300 a700 0f3a 07b2 0093 12a7 b900 9f02
00000b0: 00a7 0003 031d 9f00 122c c600 0d2c 1100
00000c0: b101 b900 6603 00b1 2cc6 000d 2c11 00b1
00000d0: 01b9 0066 0300 b13a 04b2 0027 1904 1228
00000e0: b900 2e03 00a7 0003 2cc6 000d 2c11 00b1
00000f0: 01b9 0066 0300 b12c c600 0b59 2c5f b900
0000100: 6902 00bf
Exception Handler Table:
bci [2, 21] => handler: 20
bci [90, 140] => handler: 143
bci [90, 140] => handler: 148
bci [90, 140] => handler: 148
bci [90, 140] => handler: 148
bci [90, 140] => handler: 165
bci [23, 65] => handler: 65
bci [200, 215] => handler: 215
bci [2, 247] => handler: 247
Stackmap Table:
full_frame(@20,{Object[#3],Object[#111],Object[#98]},{Object[#31]})
same_frame(@21)
full_frame(@65,{Object[#3],Object[#111],Object[#98],Integer},{Object[#31]})
append_frame(@82,Object[#31])
chop_frame(@84,1)
full_frame(@140,{Object[#3],Object[#111],Object[#98],Integer,Top,Object[#111],Object[#3],Object[#85],Object[#137]},{})
full_frame(@143,{Object[#3],Object[#111],Object[#98],Integer,Top,Object[#111],Object[#3]},{Object[#114]})
same_locals_1_stack_item_frame(@148,Object[#5])
same_locals_1_stack_item_frame(@165,Object[#122])
append_frame(@177,Object[#5])
same_frame(@180)
same_frame(@199)
same_frame(@200)
same_frame(@214)
same_locals_1_stack_item_frame(@215,Object[#31])
full_frame(@232,{Object[#3],Object[#111],Object[#98],Integer,Object[#31],Object[#111],Object[#3],Object[#5]},{})
same_frame(@246)
full_frame(@247,{Object[#3],Object[#111],Object[#98]},{Object[#31]})
same_locals_1_stack_item_frame(@259,Object[#31])

at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:?]
at java.lang.Class.privateGetDeclaredConstructors(Unknown Source) ~[?:?]
at java.lang.Class.getConstructor0(Unknown Source) ~[?:?]
at java.lang.Class.getDeclaredConstructor(Unknown Source) ~[?:?]
at org.eclipse.jetty.server.handler.ContextHandler$StaticContext.createInstance(ContextHandler.java:2896) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ServletContextHandler$Context.createInstance(ServletContextHandler.java:1292) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ServletContextHandler$Context.createInstance(ServletContextHandler.java:1301) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.BaseHolder.createInstance(BaseHolder.java:204) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ListenerHolder.createInstance(ListenerHolder.java:100) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ListenerHolder.doStart(ListenerHolder.java:89) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:382) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1304) ~[jetty-webapp-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:900) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:532) ~[jetty-webapp-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:221) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.Server.start(Server.java:470) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.Server.doStart(Server.java:415) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$4(XmlConfiguration.java:1880) ~[jetty-xml-10.0.13.jar:10.0.13]
at java.security.AccessController.doPrivileged(Unknown Source) ~[?:?]
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1818) ~[jetty-xml-10.0.13.jar:10.0.13]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.eclipse.jetty.start.Main.invokeMain(Main.java:229) ~[start.jar:10.0.13]
at org.eclipse.jetty.start.Main.start(Main.java:527) ~[start.jar:10.0.13]
at org.eclipse.jetty.start.Main.main(Main.java:76) ~[start.jar:10.0.13]

2023-05-26 10:06:43.194 INFO (main) [ ] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2023_05_26.request.log
2023-05-26 10:06:43.215 INFO (main) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2017f6e6{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2023-05-26 10:06:43.219 INFO (main) [ ] o.e.j.s.Server Started Server@3db6dd52{STARTING}[10.0.13,sto=0] @5552ms
2023-05-26 10:06:56.685 WARN (qtp718057154-60) [ ] o.e.j.i.ManagedSelector Could not accept java.nio.channels.SocketChannel[closed]: java.lang.VerifyError: Bad type on operand stack

[HoustonPutman]

So the issue is that Solr 9.x has the Java Security Manager enabled by default, while Solr 8.x did not.

It looks like the Newrelic jar you are using requires permissions that do not come out-of-the-box with Solr, which is unsurprising. You can provide a custom envVar SOLR_SECURITY_MANAGER_ENABLED=false to disable the Java security manager.

Unfortunately, I don't think there is a very good way to upload a custom security manager via the operator, unless you want to make your own Solr image that pulls the official image, and overrides the security manager config.

[pavanKumar2211]

So the issue is that Solr 9.x has the Java Security Manager enabled by default, while Solr 8.x did not.

It looks like the Newrelic jar you are using requires permissions that do not come out-of-the-box with Solr, which is unsurprising. You can provide a custom envVar SOLR_SECURITY_MANAGER_ENABLED=false to disable the Java security manager.

Unfortunately, I don't think there is a very good way to upload a custom security manager via the operator, unless you want to make your own Solr image that pulls the official image, and overrides the security manager config.

Thanks a lot! I added the envVar in Dockerfile provided by solr and pushed into my repository and used it. It worked.

[HoustonPutman]

You can also provide custom envVars through SolrCloud.spec.customSolrKubeOptions.podOptions.envVars. That's probably an easier solution than creating a custom image 🙂