JVM Crash using Maven Multi-Module + Jetty

[ricardojlrufino]

I'm having problems with a maven environment with multiple modules.
When I change anything in the main application it works normally, but when I change any line in a module, the JVM stops.

EDIT: I believe the problem is related by the fact that the method is being called in scriplet from within the JSP.

I am using:

-noverify -XX:+UnlockDiagnosticVMOptions -XX:+EnableInvokeDynamic -javaagent:/media/hotswap-agent-1.1.0-SNAPSHOT.jar -XXaltjvm=dcevm

OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

Console error

HOTSWAP AGENT: 19:10:49.139 RELOAD (org.hotswap.agent.config.PluginManager) - Reloading classes [br.com.criativasoft.modulo.boletim.metadados.BoletimJspTableCellRender] (autoHotswap)
HOTSWAP AGENT: 19:10:49.139 RELOAD (org.hotswap.agent.config.PluginManager) - Reloading classes [br.com.criativasoft.modulo.boletim.metadados.BoletimJspTableCellRender] (autoHotswap)
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8cf8f1f550, pid=2014, tid=0x00007f8ce2fb3700

The package br.com.criativasoft.modulo.boletim, is another WAR module.

Will the problem be that the class is being loaded twice?

I've attached the JVM error log

hs_err_pid2014.log.txt

[skybber]

Problem is in dcevm look at this lines from hs_err_pid:

Stack: [0x00007f8ce2eb3000,0x00007f8ce2fb4000], sp=0x00007f8ce2fb23e0, free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x4cf550] ConstantPool::klass_ref_at_noresolve(int)+0x60
V [libjvm.so+0x850235] MethodComparator::args_same(Bytecodes::Code, Bytecodes::Code)+0x4f5
V [libjvm.so+0x852c2f] MethodComparator::methods_EMCP(Method*, Method*)+0x9ef

[skybber]

I've inspected the problematic place in DCEVM, what kind of method change did you do before redefinition?

[ricardojlrufino]

The changes are in the method body

I believe the problem is related by the fact that the method is being called in scriplet from within the JSP.

I did the tests in other classes of the modules and it is working.

Another (related) error happens when i change a TagLib

HOTSWAP AGENT: 12:51:03.775 RELOAD (org.hotswap.agent.config.PluginManager) - Reloading classes [br.com.linkzsistemas.framework.taglib.html.FormItem] (autoHotswap)
HOTSWAP AGENT: 12:51:03.783 RELOAD (org.hotswap.agent.config.PluginManager) - Reloading classes [br.com.linkzsistemas.framework.taglib.html.FormItem] (autoHotswap)

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f2f1b5c954c, pid=14086, tid=0x00007f2e85e8c700
#
# JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
# Java VM: Dynamic Code Evolution 64-Bit Server VM (25.71-b01-dcevmlight-8 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x75354c]  VM_EnhancedRedefineClasses::load_new_class_versions(Thread*)+0x2cc
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /media/ricardo/Dados/Workspace/Criativa/Edu3/hs_err_pid14086.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/dcevm/dcevm/issues

hs_err_pid14086.log.txt

Would the error be related to the jsps compiler?

[skybber]

The problem is definitively in dcevm. It seems the jsp compiler produces some specific code that breaks DCEVM. I can check the problem but I need reproducible example project.

[ricardojlrufino]

Sample projetct:
https://github.com/ricardojlrufino/Issue-HotswapAgent-186

1. install test-dep
2. set params
   export MAVEN_OPTS="-noverify -XX:+UnlockDiagnosticVMOptions -XX:+EnableInvokeDynamic -javaagent:/path/hotswap-agent-1.1.0-SNAPSHOT.jar -XXaltjvm=dcevm"
3. run test-main
   mvn jetty:run
4. change test-dep and compile
   nano test-dep/src/main/java/FromScriptlet.java
   mvn compile

[skybber]

Thanks! btw, why do you need -XX:+EnableInvokeDynamic ?

[ricardojlrufino]

I found recommendation in a blog

[skybber]

Could you check it without it ?

[ricardojlrufino]

I tested it, it does not matter

[skybber]

So, very interesting issue! The cause of the problem is missing lock on jvmti redefine method in dcevm. Dcevm redefinition works, if it is used from IDE, since it is not called parallely. But in your case the redefinition is called from HA's Hotswapper module. HotswapAgent creates one Hotswapper instance for each ClassLoader that is found. In your case there are 2 classloaders, both watching the same extraClasspath. Then, if some .class is changed on the extraClassPath, the HA calls dcevm jvmti redefinition parallelly and dcevm crashes. The most straightforward workaround is to disable JasperClassloader (as I think it is useless to register it) by putting following line to hotswap-agent.properties:

excludedClassLoaderPatterns=org.apache.jasper.servlet.JasperLoader

but the proper solution is the lock in dcevm jvmti redefinition!

Thanks for nice and helpful example project !

[ricardojlrufino]

Cool, Its working ...

Could this be excluded by default in the Jetty plugin?

Thanks

[skybber]

Jasper is not used by Jetty only, but by many others containers as well. I'm not sure if it would be necesary in some very rare cases. If not it can be excluded by default.