VisualVM freezes if JVM stops while connecting

[jukzi]

I clicked on a local JVM. Typically it takes some seconds to connect, which is fine. But when the target JVM terminates during that time VisualVm freezes and all previous recorded measurements are lost :-(

I took a jstack from VisualVM while it is frozen:

DataSourceWindowManager Processor" #38 daemon prio=1 os_prio=-2 tid=0x0000000030567800 nid=0x72bc runnable [0x00000000340ae000]
   java.lang.Thread.State: RUNNABLE
	at sun.tools.attach.WindowsVirtualMachine.connectPipe(Native Method)
	at sun.tools.attach.WindowsVirtualMachine.execute(WindowsVirtualMachine.java:100)
	at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:261)
	at sun.tools.attach.HotSpotVirtualMachine.executeJCmd(HotSpotVirtualMachine.java:243)
	at sun.tools.attach.HotSpotVirtualMachine.startLocalManagementAgent(HotSpotVirtualMachine.java:204)
	at org.graalvm.visualvm.jmx.impl.JmxModelImpl$LocalVirtualMachine.loadManagementAgentViaJcmd(JmxModelImpl.java:518)
	at org.graalvm.visualvm.jmx.impl.JmxModelImpl$LocalVirtualMachine.loadManagementAgent(JmxModelImpl.java:478)
	- locked <0x00000005c3b0e3a8> (a org.graalvm.visualvm.jmx.impl.JmxModelImpl$LocalVirtualMachine)
	at org.graalvm.visualvm.jmx.impl.JmxModelImpl$LocalVirtualMachine.startManagementAgent(JmxModelImpl.java:454)
	- locked <0x00000005c3b0e3a8> (a org.graalvm.visualvm.jmx.impl.JmxModelImpl$LocalVirtualMachine)
	at org.graalvm.visualvm.jmx.impl.ProxyClient.tryConnect(ProxyClient.java:265)
	at org.graalvm.visualvm.jmx.impl.ProxyClient.connectImpl(ProxyClient.java:233)
	at org.graalvm.visualvm.jmx.impl.ProxyClient.connect(ProxyClient.java:175)
	at org.graalvm.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:180)
	at org.graalvm.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:67)
	at org.graalvm.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42)
	at org.graalvm.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:105)
	- locked <0x00000005c3a0ad70> (a org.graalvm.visualvm.jvmstat.application.JvmstatApplication)
	at org.graalvm.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
	at org.graalvm.visualvm.application.views.threads.ApplicationThreadsViewProvider.resolveThreads(ApplicationThreadsViewProvider.java:84)
	at org.graalvm.visualvm.application.views.threads.ApplicationThreadsViewProvider.supportsViewFor(ApplicationThreadsViewProvider.java:50)
	at org.graalvm.visualvm.application.views.threads.ApplicationThreadsViewProvider.supportsViewFor(ApplicationThreadsViewProvider.java:45)
	at org.graalvm.visualvm.core.ui.DataSourceViewsManager.getViews(DataSourceViewsManager.java:136)
	at org.graalvm.visualvm.core.ui.DataSourceWindowManager.openWindowAndAddView(DataSourceWindowManager.java:262)
	at org.graalvm.visualvm.core.ui.DataSourceWindowManager.access$000(DataSourceWindowManager.java:52)
	at org.graalvm.visualvm.core.ui.DataSourceWindowManager$2.run(DataSourceWindowManager.java:121)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1419)
	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:45)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:287)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2034)

visualVMFreeze.log

[thurka]

Do you have steps to reproduce it? According to thread dump, it is blocked in native code deep in JDK. It is not immediatelly clear to me, how to prevent it.

[jukzi]

Well its reproducable with the shown VerifyTests of JDT, which spawn new JDK like every second. But i have not minimal reproducer yet. A "solution"/workaround would be to have the connect in a different thread that does not block the ui. At worst such thread would just not die.

[thurka]

A "solution"/workaround would be to have the connect in a different thread that does not block the ui.

Hmm, "DataSourceWindowManager Processor" is not AWT thread.

[jukzi]

Hmm, i tried to create a reproducer by just restarting shortliving JVMs, but i can NOT reproduce with such simple program :-( :

package my;

import java.lang.management.ManagementFactory;
import java.util.ArrayList;

public class Main {

	private static final int SLEEP = 10;

	public static void main(String[] args) throws Exception {
		if (args != null && args.length != 0) {
			Thread.sleep(SLEEP);
		} else {
			ArrayList<String> cmd = new ArrayList<>();
			cmd.add(ProcessHandle.current().info().command().get());
			for (String jvmArg : ManagementFactory.getRuntimeMXBean().getInputArguments()) {
				cmd.add(jvmArg);
			}
			cmd.add("-cp");
			cmd.add(ManagementFactory.getRuntimeMXBean().getClassPath());
			cmd.add(Main.class.getName());
			cmd.add("forked");
			for (int i = 0; i < 1000; i++) {
				System.out.println(cmd);
				Thread.sleep(SLEEP);
				Runtime.getRuntime().exec(cmd.toArray(String[]::new));
			}
		}
	}

}

[jukzi]

Hmm, "DataSourceWindowManager Processor" is not AWT thread.

The AWT thread is waiting for a lock hold by the starting thread:

"AWT-EventQueue-0" #22 prio=6 os_prio=0 tid=0x0000000029b59800 nid=0x53e4 waiting for monitor entry [0x000000002c99d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.graalvm.visualvm.core.datasource.DataSource.getChangeSupport(DataSource.java:304)
	- waiting to lock <0x00000005c3a0ad70> (a org.graalvm.visualvm.jvmstat.application.JvmstatApplication)
	at org.graalvm.visualvm.core.datasource.DataSource.removePropertyChangeListener(DataSource.java:215)
	at org.graalvm.visualvm.profiler.ProfileApplicationAction.updateState(ProfileApplicationAction.java:75)
	at org.graalvm.visualvm.core.ui.actions.DataSourceAction$1.selectionChanged(DataSourceAction.java:87)
	at org.graalvm.visualvm.core.explorer.ExplorerSupport$ExplorerTreeSelectionListener.valueChanged(ExplorerSupport.java:272)

[thurka]

The AWT thread is waiting for a lock hold by the starting thread

This shows how important is to post whole thread dump, not just the part you think is important. So, please post the whole thread dump. Thanks.

[jukzi]

i did that, see "visualVMFreeze.log" in the issue

[thurka]

OK, this link without description is almost invisble. :-)

[thurka]

Any update on the steps how to reproduce it?

[jukzi]

My usecase is reproducible with setting up eclipse IDE workspace as described in
https://github.com/eclipse-platform/.github/blob/main/CONTRIBUTING.md#creating-an-eclipse-development-environment
and then running org.eclipse.jdt.core.tests.compiler.regression.MethodVerifyTest as JUnit Plugin Test.
It starts multiple VMs, and if i try to connect to each of them as soon as they start eventually one of them fails to connect.

minimal reproducer:

package my;

import java.lang.management.ManagementFactory;
import java.util.ArrayList;

public class Main {

	private static final int SLEEP = 1000;

	public static void main(String[] args) throws Exception {
		if (args != null && args.length != 0) {
			long end = System.nanoTime()+SLEEP*1_000_000;
			while (System.nanoTime() < end) {
				System.out.print("");
			}
		} else {
			ArrayList<String> cmd = new ArrayList<>();
			cmd.add(ProcessHandle.current().info().command().get());
			for (String jvmArg : ManagementFactory.getRuntimeMXBean().getInputArguments()) {
				cmd.add(jvmArg);
			}
			cmd.add("-cp");
			cmd.add(ManagementFactory.getRuntimeMXBean().getClassPath());
			cmd.add(Main.class.getName());
			cmd.add("forked");
			for (int i = 0; i < 100; i++) {
				System.out.println(cmd);
				Runtime.getRuntime().exec(cmd.toArray(String[]::new)).waitFor();
			}
		}
	}

}

this starts multiple VMs one after another. trying to connect with VisualVm to each of them as soon as they start eventually one of them fails to connect. (tested on Windows)

[thurka]

Thanks a lot. I will try to reproduce it.

[thurka]

FYI: I am able to reproduce it.

[jukzi]

good to know