Zorka apparently conflicts with WAR class loader policy

[B5r1oJ0A9G]

Hi!

We're giving Zorka agent a try regarding monitoring of application servers via Zabbix.

Now some applications stopped working after deploying Zorka agent. It turned out that the reason for that is somehow connected to having 'Single class loader for application' selected in 'WAR class loader policy'.

Since the AS in question can't be configured to use a 'Class loader for each WAR file in application' (dunno why, it's a statement from the developers), I wonder if this is something that can be "fixed' or worked-around in Zorka agent.

This is a log message I got from an application administrator encountering this issue:

WebContainer  E com.ibm.ws.webcontainer.internal.WebContainer handleRequest SRVE0255E: A WebGroup/Virtual Host to handle ***.jsp has not been defined.

Another possibly important information is that the application itself completely stopped logging when Zorka was deployed.

I recall a similar issues when using Zorka agent on some of our JBoss applications servers. Can't give any details yet since the respective admin is currently on vacation.

Environment:

• Zorka agent 1.0.6
• Websphere Application Server 8.5.5.2
• IBM JDK 1.7 (com.ibm.websphere.IBMJAVA.v70_7.0.6001.20140324_2202)
• Zabbix 2.2

[jitlogic]

Hello,

Can you send me zorka.properties file (without password of course) ?

You can also try disabling all subsystem of agent: spy, tracer, mbean registration etc. and see if "dormant" agent still breaks WAS (I'll try it myself as from your description this bug should be repeatable).

Regards,
rle

[B5r1oJ0A9G]

Hi!

Here the relevant entries in zorka.properties:

# grep -v '^$\|^#' zorka.properties 
scripts = jvm.bsh, zabbix.bsh, ibm/was.bsh
was.user = zabbix
was.pass = secret
zabbix.server.addr = 127.0.0.1,zabbix.example.com
zabbix.listen.port = 10055
tracer = no

As soon as we have identified which options have to be set in zorka.properties to disable all subsystems we'll give it another try. After this I'll report again.

Thanks in advance!

[jitlogic]

Options to disable everything AFAIR:

spy = no
zabbix = no
nagios = no

And comment out scripts line (or leave it empty AFAIR, so it will not load any BSH scripts).

Then check agent log if there are errors (or more logs about something starting up etc.).

I hope it is all, maybe I'll check it out later and describe somewhere in documentation.

Regards,
rle

[B5r1oJ0A9G]

So, just got the result from the admin.
This issue exists only when ibm/was.bsh is configured.

[jitlogic]

Ok, this tells me a lot. So now one thing to check is was.bsh script - if you have time for it.

Please extract was.bsh from zorka.jar: com/jitlogic/zorka/scripts/ibm/was.bsh.

Create directory scripts in agent home directory, create directory ibm inside scripts and copy 'was.bsh' file to 'ibm' directory.

Edit was.bsh file and find two sections:

  spy.add(spy.instance("WAS_MBS_REGISTER")
    .onReturn(spy.fetchArg("THIS", 0), jvm.autoregister_plugin(), pmi_register())
    .include(spy.byMethod("com.ibm.ws.management.AdminServiceImpl", "<init>")));

and

  spy.add(spy.instance("WAS_CATCH_CLASS_LOADER")
    .onReturn(spy.fetchArg("THIS", 0), class_loader_catcher())
    .include(spy.byMethod("com.ibm.ws.classloader.ProtectionClassLoader", "<init>")));

Try comment each one of those sections (separately) and tell me which one (if any) causes problem.

I'll try configuring may own environment and replicating this problem but it might take some time due to significant backlog of various tasks I have to do next week.

[B5r1oJ0A9G]

Sorry for the delay!

We made the requested test today and even commenting both sections didn't help.
Please don't hesitate to ask for any test that might support finding the cause.

[jitlogic]

I've fixed a nasty bug regarding JDK7/8 and stack maps calculation some time ago. It was causing spurious class loads from time to time that were bypassing instrumentation (and possibly violating custom class loader policies). It was appearing in various contexts especially with JDK8, I suppose it was the cause of this issue. I'm closing this, if you still have problems please reopen.