Improve plugin classloader visiblity

[kroepke]

Continuing the discussion from #2280 (comment)

Since 2.0.2 Graylog uses a single classloader to load all plugins. This is necessary so that plugins can extend/use other plugins.

The downside of this setup is that conflicting dependency versions now clash, even if a given plugin does not need access to other plugins.
One possible solution could be to let plugins declare their classloader isolation preference, with defaulting to "fully isolated".
Plugins which do need access to other plugins could then declare that preference and thus extend them.

This issue exists so we can discuss possible approaches and make a decision how to proceed (or to come up with workarounds for affected plugin authors)

[magicdude4eva]

Thank you for logging this issue. Another suggestion / possible solution might be to introduce a "common" library folder for plugins which are loaded by the parent classloader.

This would then prevent that plugins could cause compatibility issues when shipping outdated / incompatible libraries.

[kroepke]

@magicdude4eva Could you see if this build fixes the issues you have been seeing? A recent tarball from http://packages.graylog2.org/_/snapshots?q=2.1.0 should be enough to test.

Please recompile against that version, and then try to load your plugin. Only plugins that requested a shared classloader will be able to see each other, all other plugins should be isolated and only see the server and its dependencies.

I think that will solve your pain :)

[magicdude4eva]

Sounds great. I will not have time to recompile/test within the next 2 weeks (holidays) - I hope that when I am back there will be an alpha/beta release where I can just upgrade Graylog and verify?

[kroepke]

@magicdude4eva I've just tested your plugin on current master with the current Graylog server master and everything works as expected.
I created a map widget and configured a jira alert callback, so this should be fine. Your plugin should work fine again with Graylog 2.1.

[magicdude4eva]

Great news @kroepke - thanks for taking the time and effort to test it on master and providing feedback here.

[123dev]

Sorry to rain on the parade guys, Using Graylog 2.1.1 with graylog-plugin-jira-1.0.8.jar and the problem is still occurring.
Kroepke reports success, wonder how many times it was tested, the problem has always been sporadic with us. (In a clustered Graylog environment).

Do we need a recompiled graylog-plugin-jira jar?
Thanks

[joschi]

@123dev Yes. Please file a bug report in the repository for the JIRA plugin.

[magicdude4eva]

I don't think this is JIRA plugin issue. I am running latest version of Graylog with my current version of JIRA plugin and it works perfectly fine. @123dev log an issue in the JIRA plugin repository with full stack trace/log of Graylog.

[123dev]

Thanks Gerd and Jochen,
I'll do that next time it happens.
It happened yesterday, and the graylog log file dated yesterday seems to be of an archive format that I'm not familiar with, so can't extract the relevant details.

-rwxr--r-- 1 root root 7030 Oct 25 15:45 @40000000580f7e0d22991fd4.s*

[joschi]

@magicdude4eva Where exactly did you get this file from?
How did you install Graylog (manually, OS packages, OVA etc.)?

[magicdude4eva]

@joschi the initial issue was raised by me as the change to the classloader visibility in Graylog 2.0 introduced conflicting dependencies (i.e. another plugin bootstrapped an older version of HTTPClient causing my plugin to fail).

The classloader issue was fixed in 2.1.0. I am currently running Graylog 2.1.1+01d50e5 (installed via CentOS repo) and do not have the issue.

@123dev please have a look at #2280 (comment) - if your error is the same, then it is a classloader issue. In this case confirm what version of Graylog server you are using. I am also not sure why your log-file looks like this. Shouldn't it be server.log?

A workaround is to remove the conflicting plugin (in my case under Graylog 2.0.0 it was the map-widget plugin) if you are not able to upgrade to a more recent version of Graylog server.

There is unfortunately no way that I can patch/fix my plugin, as I require that specific version of HTTPClient, but as @kroepke said, this has been fixed and it also works in my installation. I do not run a Graylog server cluster, but I doubt that clustering would affect the classloader for plugins.

[123dev]

@joschi
We're using AWS Image (the image was first fired up when graylog was at 1.x, and then we've just upgraded graylog, and never the image, so the latest image might be different)

the files are under /var/log/graylog/server

here's the full list

-rw-r--r-- 1 root root  716455 Sep 27 15:24 @4000000057ea8f4925a0aecc.u
-rwxr--r-- 1 root root   22118 Sep 28 15:24 @4000000057ebe0c929a4166c.s*
-rwxr--r-- 1 root root    1715 Sep 29 15:24 @4000000057ed3249308edad4.s*
-rwxr--r-- 1 root root    4109 Sep 30 15:24 @4000000057ee83c93758b95c.s*
-rwxr--r-- 1 root root    5334 Oct  1 15:24 @4000000057efd54a02c772dc.s*
-rwxr--r-- 1 root root   44307 Oct  2 15:24 @4000000057f126ca09b15c84.s*
-rwxr--r-- 1 root root    1719 Oct  3 15:24 @4000000057f2784a10a56bd4.s*
-rwxr--r-- 1 root root    1786 Oct  4 15:24 @4000000057f3c9ca17c263cc.s*
-rwxr--r-- 1 root root    9152 Oct  5 15:24 @4000000057f51b4a1ee9622c.s*
-rwxr--r-- 1 root root    2384 Oct  6 15:24 @4000000057f66cca25d3f3cc.s*
-rwxr--r-- 1 root root    4808 Oct  7 15:24 @4000000057f7be4a2c2692fc.s*
-rwxr--r-- 1 root root    5307 Oct  8 15:24 @4000000057f90fca332d5b44.s*
-rwxr--r-- 1 root root    1586 Oct  9 15:24 @4000000057fa614a3a17be04.s*
-rwxr--r-- 1 root root    6418 Oct 10 15:24 @4000000057fbb2cb0566975c.s*
-rwxr--r-- 1 root root    3154 Oct 11 15:24 @4000000057fd044b0b0991b4.s*
-rwxr--r-- 1 root root    5221 Oct 12 15:24 @4000000057fe55cb12097074.s*
-rwxr--r-- 1 root root    8844 Oct 13 15:24 @4000000057ffa74b18f38514.s*
-rwxr--r-- 1 root root    4806 Oct 14 15:24 @400000005800f8cb1fef490c.s*
-rwxr--r-- 1 root root 1753482 Oct 15 19:17 @40000000580280bc1598f41c.s*
-rwxr--r-- 1 root root    7218 Oct 16 19:17 @400000005803d23c1ca3e2e4.s*
-rwxr--r-- 1 root root   15777 Oct 17 19:17 @40000000580523bc2327e2b4.s*
-rwxr--r-- 1 root root   34112 Oct 18 19:17 @400000005806753c2a3adffc.s*
-rwxr--r-- 1 root root   19077 Oct 19 19:17 @400000005807c6bc3161edfc.s*
-rwxr--r-- 1 root root    9413 Oct 20 19:17 @400000005809183c388903cc.s*
-rw-r--r-- 1 root root  571956 Oct 21 15:44 @40000000580a380d19fc3ab4.u
-rwxr--r-- 1 root root  786127 Oct 22 15:45 @40000000580b898d18c26eac.s*
-rwxr--r-- 1 root root    7098 Oct 23 15:45 @40000000580cdb0d1c16067c.s*
-rwxr--r-- 1 root root    6971 Oct 24 15:45 @40000000580e2c8d1f58509c.s*
-rwxr--r-- 1 root root    7030 Oct 25 15:45 @40000000580f7e0d22991fd4.s*
-rwxr--r-- 1 root root    5861 Oct 26 15:45 @400000005810cf8d25e91dc4.s*
-rw-r--r-- 1 root root      30 Apr 24  2015 config
-rw-r--r-- 1 root root   95559 Oct 27 14:39 current
-rw------- 1 root root       0 Apr 24  2015 lock
-rw-r--r-- 1 root root       0 Oct 26 15:45 state

@magicdude4eva
Unfortunately we use the map plugin, hence can't disable it.

I'm using the latest Graylog version Graylog 2.1.1+01d50e5
That's why I'm confused as why this is still an issue.
The latest jira plugin jar is dated May,
https://github.com/magicdude4eva/graylog-jira-alarmcallback/releases/tag/1.0.8
Was the compiled with the latest Graylog, I'm not sure if 2.1 was released in May.

Thanks

[joschi]

@123dev These log files are written by multilog (part of daemontools), see https://cr.yp.to/daemontools/multilog.html for details.

What problem are you having specifically with the JIRA alarm callback plugin? Which error messages do you get?

[magicdude4eva]

@123dev Could you please do a ls on the plugins directory as well as show us a stack-trace on the JIRA plugin. Perhaps a gist of a clean log-file after restart up to the point where the error occurs would be the best option.

[123dev]

@joschi
Thanks for the identifying the log files, I'm surprised that I have multilog generated files, daemontools is not even installed.

# multilog
The program 'multilog' is currently not installed. You can install it by typing:
apt-get install daemontools

Besides I couldn't find any viewer that would display one of those binary files so that I can extract the log for magicdude4eva .

The problem I'm encountering is that the plugin gets triggered, but the message is not attached.
magicdude4eva/graylog-jira-alarmcallback#11

#2382

This doesn't always happen, but does happen every now and then.
According this ticket, the issue should be resolved, but I'm still encountering it with the latest Graylog and Jira plugin.

Did I comment in the wrong thread?
Can you move my comments to:
#2382

@magicdude4eva
ls -l

-rw-r--r-- 1 root root  2111402 Mar 15  2016 graylog2-plugin-input-httpmonitor-1.0.3.jar
-rw-r--r-- 1 root root 12670356 Aug  2 04:14 graylog-plugin-aws-1.0.0.jar
-rw-r--r-- 1 root root    23986 Sep 14 15:26 graylog-plugin-beats-1.1.1.jar
-rw-r--r-- 1 root root  2795675 Sep 14 15:26 graylog-plugin-collector-1.1.1.jar
-rw-r--r-- 1 root root    10515 Oct 10  2015 graylog-plugin-dnsresolver-1.1.2.jar
-rw-r--r-- 1 root root  2377123 Sep 14 15:26 graylog-plugin-enterprise-integration-1.1.1.jar
-rw-r--r-- 1 root root    15174 Jul  6 09:31 graylog-plugin-hipchat-1.3.0.jar
-rw-r--r-- 1 root root  2365436 Jun 29 13:43 graylog-plugin-jira-1.0.8.jar
-rw-r--r-- 1 root root  5749764 Sep 14 15:26 graylog-plugin-map-widget-1.1.1.jar
-rw-r--r-- 1 root root  3594006 Sep 10  2015 graylog-plugin-mqtt-1.1.1.jar
-rw-r--r-- 1 root root  5092685 Sep 14 15:26 graylog-plugin-pipeline-processor-1.1.1.jar
-rw-r--r-- 1 root root    24028 Jun  1 14:26 graylog-plugin-slack-2.2.1.jar
-rw-r--r-- 1 root root  2439075 Sep  9  2015 graylog-plugin-twiliosms-1.0.0.jar
-rw-r--r-- 1 root root   479800 Sep 14 15:26 usage-statistics-2.1.1.jar

How do I get stack trace of Jira plugin?

The problem hasn't happened again since I brought this up two days ago.

Thanks guys.