Newrelic agent with 2.0.0

[krisb78]

Posting this in case its related to #13785

I'm trying to run ES 2.0.0 with a newrelic agent, but I'm getting

Exception in thread "main" java.lang.IllegalStateException: failed to load bundle [] due to jar hell
Likely root cause: java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/local/newrelic/newrelic.jar" "read")

See complete log below:

root@cubitsearch-client-1:~# sudo -u elasticsearch /usr/bin/java -Xms3g -Xmx3g -Xmn200m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Dnewrelic.config.file=/usr/local/newrelic/elasticsearch.yml -javaagent:/usr/local/newrelic/newrelic.jar -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.0.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch -Des.max-open-files=true
Nov 11, 2015 12:16:17 +0000 [1946 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/usr/local/newrelic/elasticsearch.yml"
Nov 11, 2015 12:16:17 +0000 [1946 1] com.newrelic WARN: New Relic Agent: Unable to write log file: /var/log/newrelic/newrelic_agent.log. Please check permissions on the file and directory.
log4j:WARN No appenders could be found for logger (com.newrelic.agent.deps.org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[2015-11-11 12:16:54,791][INFO ][bootstrap                ] max_open_files [4096]
[2015-11-11 12:16:56,233][INFO ][node                     ] [cubitsearch-client-1] version[2.0.0], pid[1946], build[de54438/2015-10-22T08:09:48Z]
[2015-11-11 12:16:56,233][INFO ][node                     ] [cubitsearch-client-1] initializing ...
Exception in thread "main" java.lang.IllegalStateException: failed to load bundle [] due to jar hell
Likely root cause: java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/local/newrelic/newrelic.jar" "read")
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
    at java.security.AccessController.checkPermission(AccessController.java:559)
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
    at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
    at java.util.zip.ZipFile.<init>(ZipFile.java:206)
    at java.util.zip.ZipFile.<init>(ZipFile.java:145)
    at java.util.jar.JarFile.<init>(JarFile.java:154)
    at java.util.jar.JarFile.<init>(JarFile.java:91)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:173)
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:340)
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:113)
    at org.elasticsearch.node.Node.<init>(Node.java:144)
    at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.

I tried setting a custom java policy for my elasticsearch user by creating /usr/share/elasticsearch/.java.policy with the following content:

grant codeBase "file:/usr/local/newrelic/-" {
       permission java.security.AllPermission;
       permission java.net.SocketPermission "*.newrelic.com", "connect,accept,resolve";
};

, but to no effect.

I also tried to add the above to the global java policy, but it didn't work either.

I managed to get the agent to work by disabling the security manager entirely (-Dsecurity.manager.enabled=false), but I don't really want to do this in production.

[rmuir]

We can fix it, here is the deal:

1. existing 1.x startup scripts had a bug that would add CWD to CLASSPATH (elasticsearch.in.sh unintentionally adds CWD to classpath #12000). Java classloaders are lenient, so this went undetected forever. In many cases such as running as a service, CWD would be /, which is terrible: it would mean total read access to all files on the system. So we initially (and still do) jumped through hoops to avoid granting access to what is in CLASSPATH etc, since it was too arbitrary and crazy.
2. as far as ubuntu's agent (Elastic search 2.0.0-beta2 zip package does not start on ubuntu #13785), that is unrelated to your situation, in that case its an unwanted agent and its being added in a sneaky way that the user is not aware of. We detect that and kick it out, if you want an agent, it should be that you asked for it explicit (like in your case, where you are asking to run newrelic because you pass -javaagent:/usr/local/newrelic/newrelic.jar)
3. ES_CLASSPATH was exposed directly to users, but this is problematic, because most likely it will just result in pain, its not a good or tested way to "plugin code", no isolation etc, we have the plugin mechanism for that.

we fixed and added safety around these situations too in #13880.

Anyway, I think we can now safely add permissions to CLASSPATH, to ensure agents work, since now we defend against the silliness, and that will solve it for your agent or any other agent where the user has explicitly configured it.

[rmuir]

By the way, this is the best way to workaround it in the meantime: https://docs.oracle.com/javase/tutorial/ext/basics/install.html

[rmuir]

I tried setting a custom java policy for my elasticsearch user by creating /usr/share/elasticsearch/.java.policy with the following content:

Yeah, we don't read from any of the standard locations and add to our policy. Its worth looking at too, since what you did probably should have worked, but i want to investigate first, just concerned about what kind of stuff might be in these standard locations on various distros etc.

[krisb78]

Thanks a lot, I'll try the workaround.

[krisb78]

FTR, I also tried to pass in the policy explicitly by adding -Djava.security.policy=/usr/share/elasticsearch/.java.policy to ES_JAVA_OPTS, but it didn't work either.

[rmuir]

yeah, I see this is the mechanism newrelic support recommends to their users: https://discuss.newrelic.com/t/flux-scheduler-not-able-to-load-newrelic-jar/2114

so its less about us changing our classpath logic I think, changing that would only mean a more confusing exception follows (like newrelic not having access to connect somewhere, write to its log file, something like that).

its more about allowing you to do this in the system files like they recommend (without having to resort to installing it into an extension directory). and if you don't configure access for your agent then it predictably fails on that agent jar.

[rmuir]

just concerned about what kind of stuff might be in these standard locations on various distros etc.

its kind of a downer, on java 7 < update 51:

// default permissions granted to all domains

grant {
        // Allows any thread to stop itself using the java.lang.Thread.stop()
        // method that takes no argument.
        // Note that this permission is granted by default only to remain
        // backwards compatible.
        // It is strongly recommended that you either remove this permission
        // from this policy file or further restrict it to code sources
        // that you specify, because Thread.stop() is potentially unsafe.
        // See the API specification of java.lang.Thread.stop() for more
        // information.
        permission java.lang.RuntimePermission "stopThread";

        // allows anyone to listen on un-privileged ports
        permission java.net.SocketPermission "localhost:1024-", "listen";

After update 51 the SocketPermission improves into two (they broke back compat here and changed how listen ports are checked, yet kept stopThread for back compat?!?!?!):

        // allows anyone to listen on dynamic ports
        permission java.net.SocketPermission "localhost:0", "listen";

        // permission for standard RMI registry port
        permission java.net.SocketPermission "localhost:1099", "listen";

On java 8 and 9 the RMI is cleaned up and removed, but ephemeral port and stopThread remains. Looks like in 9 they work on cleaning up their own house with permissions to jdk internals though, which is a positive.

Anyway, sucking in the system/user configuration naively has some downsides and undoes some work, but its probably what we should do. Just makes life difficult as usual, I'll work up something.

[krisb78]

Awesome, thanks @rmuir !