New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve plugin classloader visiblity #2436

Closed
kroepke opened this Issue Jun 29, 2016 · 15 comments

Comments

Projects
None yet
5 participants
@kroepke
Member

kroepke commented Jun 29, 2016

Continuing the discussion from #2280 (comment)

Since 2.0.2 Graylog uses a single classloader to load all plugins. This is necessary so that plugins can extend/use other plugins.

The downside of this setup is that conflicting dependency versions now clash, even if a given plugin does not need access to other plugins.
One possible solution could be to let plugins declare their classloader isolation preference, with defaulting to "fully isolated".
Plugins which do need access to other plugins could then declare that preference and thus extend them.

This issue exists so we can discuss possible approaches and make a decision how to proceed (or to come up with workarounds for affected plugin authors)

@kroepke kroepke added this to the 2.x milestone Jun 29, 2016

@magicdude4eva

This comment has been minimized.

magicdude4eva commented Jun 29, 2016

Thank you for logging this issue. Another suggestion / possible solution might be to introduce a "common" library folder for plugins which are loaded by the parent classloader.

This would then prevent that plugins could cause compatibility issues when shipping outdated / incompatible libraries.

@bernd bernd modified the milestones: 2.1.0, 2.x Jul 20, 2016

@bernd bernd self-assigned this Jul 20, 2016

bernd added a commit that referenced this issue Jul 20, 2016

Make it possible for plugins to request a shared class loader
In #2280 we changed the plugin loader so that all plugins share one
class loader and they can see each other. (needed for plugin
inder-dependencies)

This is problematic when different plugins have conflicting
dependencies. (see #2436)

With this change, every plugin gets its own class loader by default so
we can avoid dependency clashes. Plugins which depend on other plugins
can request a shared class loader via a graylog-plugin.properties file.

Fixes #2436

bernd added a commit that referenced this issue Jul 20, 2016

Make it possible for plugins to request a shared class loader
In #2280 we changed the plugin loader so that all plugins share one
class loader and they can see each other. (needed for plugin
inter-dependencies)

This is problematic when different plugins have conflicting
dependencies. (see #2436)

With this change, every plugin gets its own class loader by default so
we can avoid dependency clashes. Plugins which depend on other plugins
can request a shared class loader via a graylog-plugin.properties file.

Fixes #2436

bernd added a commit that referenced this issue Jul 25, 2016

Make it possible for plugins to request a shared class loader
In #2280 we changed the plugin loader so that all plugins share one
class loader and they can see each other. (needed for plugin
inter-dependencies)

This is problematic when different plugins have conflicting
dependencies. (see #2436)

With this change, every plugin gets its own class loader by default so
we can avoid dependency clashes. Plugins which depend on other plugins
can request a shared class loader via a graylog-plugin.properties file.

Fixes #2436

kroepke added a commit that referenced this issue Jul 28, 2016

Make it possible for plugins to request a shared class loader (#2508)
* Make it possible for plugins to request a shared class loader

In #2280 we changed the plugin loader so that all plugins share one
class loader and they can see each other. (needed for plugin
inter-dependencies)

This is problematic when different plugins have conflicting
dependencies. (see #2436)

With this change, every plugin gets its own class loader by default so
we can avoid dependency clashes. Plugins which depend on other plugins
can request a shared class loader via a graylog-plugin.properties file.

Fixes #2436

* Adjust log messages in PluginLoader

* Move properties file into a subdirectory to avoid clashes

The path to the properties file is stored in the
"Graylog-Plugin-Properties-Path" attribute of the JAR manifest.

* Only create shared class loader if any plugin requests to be shared
@kroepke

This comment has been minimized.

Member

kroepke commented Jul 28, 2016

@magicdude4eva Could you see if this build fixes the issues you have been seeing? A recent tarball from http://packages.graylog2.org/_/snapshots?q=2.1.0 should be enough to test.

Please recompile against that version, and then try to load your plugin. Only plugins that requested a shared classloader will be able to see each other, all other plugins should be isolated and only see the server and its dependencies.

I think that will solve your pain :)

@magicdude4eva

This comment has been minimized.

magicdude4eva commented Jul 28, 2016

Sounds great. I will not have time to recompile/test within the next 2 weeks (holidays) - I hope that when I am back there will be an alpha/beta release where I can just upgrade Graylog and verify?

@kroepke

This comment has been minimized.

Member

kroepke commented Jul 28, 2016

@magicdude4eva I've just tested your plugin on current master with the current Graylog server master and everything works as expected.
I created a map widget and configured a jira alert callback, so this should be fine. Your plugin should work fine again with Graylog 2.1.

@magicdude4eva

This comment has been minimized.

magicdude4eva commented Jul 28, 2016

Great news @kroepke - thanks for taking the time and effort to test it on master and providing feedback here.

@123dev

This comment has been minimized.

123dev commented Oct 26, 2016

Sorry to rain on the parade guys, Using Graylog 2.1.1 with graylog-plugin-jira-1.0.8.jar and the problem is still occurring.
Kroepke reports success, wonder how many times it was tested, the problem has always been sporadic with us. (In a clustered Graylog environment).

Do we need a recompiled graylog-plugin-jira jar?
Thanks

@joschi

This comment has been minimized.

Contributor

joschi commented Oct 26, 2016

@123dev Yes. Please file a bug report in the repository for the JIRA plugin.

@magicdude4eva

This comment has been minimized.

magicdude4eva commented Oct 26, 2016

I don't think this is JIRA plugin issue. I am running latest version of Graylog with my current version of JIRA plugin and it works perfectly fine. @123dev log an issue in the JIRA plugin repository with full stack trace/log of Graylog.

@123dev

This comment has been minimized.

123dev commented Oct 26, 2016

Thanks Gerd and Jochen,
I'll do that next time it happens.
It happened yesterday, and the graylog log file dated yesterday seems to be of an archive format that I'm not familiar with, so can't extract the relevant details.

-rwxr--r-- 1 root root 7030 Oct 25 15:45 @40000000580f7e0d22991fd4.s*

@joschi

This comment has been minimized.

Contributor

joschi commented Oct 27, 2016

@magicdude4eva Where exactly did you get this file from?
How did you install Graylog (manually, OS packages, OVA etc.)?

@magicdude4eva

This comment has been minimized.

magicdude4eva commented Oct 27, 2016

@joschi the initial issue was raised by me as the change to the classloader visibility in Graylog 2.0 introduced conflicting dependencies (i.e. another plugin bootstrapped an older version of HTTPClient causing my plugin to fail).

The classloader issue was fixed in 2.1.0. I am currently running Graylog 2.1.1+01d50e5 (installed via CentOS repo) and do not have the issue.

@123dev please have a look at #2280 (comment) - if your error is the same, then it is a classloader issue. In this case confirm what version of Graylog server you are using. I am also not sure why your log-file looks like this. Shouldn't it be server.log?

A workaround is to remove the conflicting plugin (in my case under Graylog 2.0.0 it was the map-widget plugin) if you are not able to upgrade to a more recent version of Graylog server.

There is unfortunately no way that I can patch/fix my plugin, as I require that specific version of HTTPClient, but as @kroepke said, this has been fixed and it also works in my installation. I do not run a Graylog server cluster, but I doubt that clustering would affect the classloader for plugins.

@123dev

This comment has been minimized.

123dev commented Oct 27, 2016

@joschi
We're using AWS Image (the image was first fired up when graylog was at 1.x, and then we've just upgraded graylog, and never the image, so the latest image might be different)

the files are under /var/log/graylog/server

here's the full list

-rw-r--r-- 1 root root  716455 Sep 27 15:24 @4000000057ea8f4925a0aecc.u
-rwxr--r-- 1 root root   22118 Sep 28 15:24 @4000000057ebe0c929a4166c.s*
-rwxr--r-- 1 root root    1715 Sep 29 15:24 @4000000057ed3249308edad4.s*
-rwxr--r-- 1 root root    4109 Sep 30 15:24 @4000000057ee83c93758b95c.s*
-rwxr--r-- 1 root root    5334 Oct  1 15:24 @4000000057efd54a02c772dc.s*
-rwxr--r-- 1 root root   44307 Oct  2 15:24 @4000000057f126ca09b15c84.s*
-rwxr--r-- 1 root root    1719 Oct  3 15:24 @4000000057f2784a10a56bd4.s*
-rwxr--r-- 1 root root    1786 Oct  4 15:24 @4000000057f3c9ca17c263cc.s*
-rwxr--r-- 1 root root    9152 Oct  5 15:24 @4000000057f51b4a1ee9622c.s*
-rwxr--r-- 1 root root    2384 Oct  6 15:24 @4000000057f66cca25d3f3cc.s*
-rwxr--r-- 1 root root    4808 Oct  7 15:24 @4000000057f7be4a2c2692fc.s*
-rwxr--r-- 1 root root    5307 Oct  8 15:24 @4000000057f90fca332d5b44.s*
-rwxr--r-- 1 root root    1586 Oct  9 15:24 @4000000057fa614a3a17be04.s*
-rwxr--r-- 1 root root    6418 Oct 10 15:24 @4000000057fbb2cb0566975c.s*
-rwxr--r-- 1 root root    3154 Oct 11 15:24 @4000000057fd044b0b0991b4.s*
-rwxr--r-- 1 root root    5221 Oct 12 15:24 @4000000057fe55cb12097074.s*
-rwxr--r-- 1 root root    8844 Oct 13 15:24 @4000000057ffa74b18f38514.s*
-rwxr--r-- 1 root root    4806 Oct 14 15:24 @400000005800f8cb1fef490c.s*
-rwxr--r-- 1 root root 1753482 Oct 15 19:17 @40000000580280bc1598f41c.s*
-rwxr--r-- 1 root root    7218 Oct 16 19:17 @400000005803d23c1ca3e2e4.s*
-rwxr--r-- 1 root root   15777 Oct 17 19:17 @40000000580523bc2327e2b4.s*
-rwxr--r-- 1 root root   34112 Oct 18 19:17 @400000005806753c2a3adffc.s*
-rwxr--r-- 1 root root   19077 Oct 19 19:17 @400000005807c6bc3161edfc.s*
-rwxr--r-- 1 root root    9413 Oct 20 19:17 @400000005809183c388903cc.s*
-rw-r--r-- 1 root root  571956 Oct 21 15:44 @40000000580a380d19fc3ab4.u
-rwxr--r-- 1 root root  786127 Oct 22 15:45 @40000000580b898d18c26eac.s*
-rwxr--r-- 1 root root    7098 Oct 23 15:45 @40000000580cdb0d1c16067c.s*
-rwxr--r-- 1 root root    6971 Oct 24 15:45 @40000000580e2c8d1f58509c.s*
-rwxr--r-- 1 root root    7030 Oct 25 15:45 @40000000580f7e0d22991fd4.s*
-rwxr--r-- 1 root root    5861 Oct 26 15:45 @400000005810cf8d25e91dc4.s*
-rw-r--r-- 1 root root      30 Apr 24  2015 config
-rw-r--r-- 1 root root   95559 Oct 27 14:39 current
-rw------- 1 root root       0 Apr 24  2015 lock
-rw-r--r-- 1 root root       0 Oct 26 15:45 state

@magicdude4eva
Unfortunately we use the map plugin, hence can't disable it.

I'm using the latest Graylog version Graylog 2.1.1+01d50e5
That's why I'm confused as why this is still an issue.
The latest jira plugin jar is dated May,
https://github.com/magicdude4eva/graylog-jira-alarmcallback/releases/tag/1.0.8
Was the compiled with the latest Graylog, I'm not sure if 2.1 was released in May.

Thanks

@joschi

This comment has been minimized.

Contributor

joschi commented Oct 27, 2016

@123dev These log files are written by multilog (part of daemontools), see https://cr.yp.to/daemontools/multilog.html for details.

What problem are you having specifically with the JIRA alarm callback plugin? Which error messages do you get?

@magicdude4eva

This comment has been minimized.

magicdude4eva commented Oct 28, 2016

@123dev Could you please do a ls on the plugins directory as well as show us a stack-trace on the JIRA plugin. Perhaps a gist of a clean log-file after restart up to the point where the error occurs would be the best option.

@123dev

This comment has been minimized.

123dev commented Oct 28, 2016

@joschi
Thanks for the identifying the log files, I'm surprised that I have multilog generated files, daemontools is not even installed.

# multilog
The program 'multilog' is currently not installed. You can install it by typing:
apt-get install daemontools

Besides I couldn't find any viewer that would display one of those binary files so that I can extract the log for magicdude4eva .

The problem I'm encountering is that the plugin gets triggered, but the message is not attached.
magicdude4eva/graylog-jira-alarmcallback#11

#2382

This doesn't always happen, but does happen every now and then.
According this ticket, the issue should be resolved, but I'm still encountering it with the latest Graylog and Jira plugin.

Did I comment in the wrong thread?
Can you move my comments to:
#2382

@magicdude4eva
ls -l

-rw-r--r-- 1 root root  2111402 Mar 15  2016 graylog2-plugin-input-httpmonitor-1.0.3.jar
-rw-r--r-- 1 root root 12670356 Aug  2 04:14 graylog-plugin-aws-1.0.0.jar
-rw-r--r-- 1 root root    23986 Sep 14 15:26 graylog-plugin-beats-1.1.1.jar
-rw-r--r-- 1 root root  2795675 Sep 14 15:26 graylog-plugin-collector-1.1.1.jar
-rw-r--r-- 1 root root    10515 Oct 10  2015 graylog-plugin-dnsresolver-1.1.2.jar
-rw-r--r-- 1 root root  2377123 Sep 14 15:26 graylog-plugin-enterprise-integration-1.1.1.jar
-rw-r--r-- 1 root root    15174 Jul  6 09:31 graylog-plugin-hipchat-1.3.0.jar
-rw-r--r-- 1 root root  2365436 Jun 29 13:43 graylog-plugin-jira-1.0.8.jar
-rw-r--r-- 1 root root  5749764 Sep 14 15:26 graylog-plugin-map-widget-1.1.1.jar
-rw-r--r-- 1 root root  3594006 Sep 10  2015 graylog-plugin-mqtt-1.1.1.jar
-rw-r--r-- 1 root root  5092685 Sep 14 15:26 graylog-plugin-pipeline-processor-1.1.1.jar
-rw-r--r-- 1 root root    24028 Jun  1 14:26 graylog-plugin-slack-2.2.1.jar
-rw-r--r-- 1 root root  2439075 Sep  9  2015 graylog-plugin-twiliosms-1.0.0.jar
-rw-r--r-- 1 root root   479800 Sep 14 15:26 usage-statistics-2.1.1.jar

How do I get stack trace of Jira plugin?

The problem hasn't happened again since I brought this up two days ago.

Thanks guys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment