Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(jmx): enable connection to GraalVM native images #222

Merged

Conversation

mwangggg
Copy link
Contributor

@mwangggg mwangggg commented May 25, 2023

fixes: #217
Depends on #223

@andrewazores andrewazores changed the title fix(JMV Connection): allow cryostat connection to native images fix(jmx): enable connection to GraalVM native images May 25, 2023
@mwangggg
Copy link
Contributor Author

error trace after attempting to view a native image in recording view:

May 25, 2023 7:20:43 PM io.cryostat.core.log.Logger error
SEVERE: HTTP 500: Cannot invoke "org.openjdk.jmc.rjmx.IConnectionHandle.getServiceOrDummy(java.lang.Class)" because "connectionHandle" is null
io.vertx.ext.web.handler.HttpException: Internal Server Error
Caused by: java.lang.NullPointerException: Cannot invoke "org.openjdk.jmc.rjmx.IConnectionHandle.getServiceOrDummy(java.lang.Class)" because "connectionHandle" is null
	at org.openjdk.jmc.rjmx.ConnectionToolkit.getVMName(ConnectionToolkit.java:417)
	at org.openjdk.jmc.rjmx.ConnectionToolkit.isHotSpot(ConnectionToolkit.java:364)
	at org.openjdk.jmc.rjmx.services.jfr.internal.FlightRecorderServiceV2.isFlightRecorderCommercial(FlightRecorderServiceV2.java:111)
	at org.openjdk.jmc.rjmx.services.jfr.internal.FlightRecorderServiceV2.isFlightRecorderDisabled(FlightRecorderServiceV2.java:116)
	at org.openjdk.jmc.rjmx.services.jfr.internal.FlightRecorderServiceV2.<init>(FlightRecorderServiceV2.java:129)
	at org.openjdk.jmc.rjmx.services.jfr.internal.FlightRecorderServiceFactory.getServiceInstance(FlightRecorderServiceFactory.java:47)
	at io.cryostat.core.net.JFRJMXConnection.getService(JFRJMXConnection.java:140)
	at io.cryostat.net.web.http.api.v1.TargetRecordingsGetHandler.lambda$handleAuthenticated$0(TargetRecordingsGetHandler.java:128)
	at io.cryostat.net.TargetConnectionManager.executeConnectedTask(TargetConnectionManager.java:168)
	at io.cryostat.net.web.http.api.v1.TargetRecordingsGetHandler.handleAuthenticated(TargetRecordingsGetHandler.java:124)
	at io.cryostat.net.web.http.AbstractAuthenticatedRequestHandler.handle(AbstractAuthenticatedRequestHandler.java:102)
	at io.cryostat.net.web.http.AbstractAuthenticatedRequestHandler.handle(AbstractAuthenticatedRequestHandler.java:72)
	at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
	at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
	at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135)
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)

@andrewazores
Copy link
Member

https://github.com/openjdk/jmc/blob/7097ed26ab4d04a691544aaf02f0ca12dc2db947/application/org.openjdk.jmc.rjmx/src/main/java/org/openjdk/jmc/rjmx/ConnectionToolkit.java#L417

IFlightRecorderService service = serviceFactory.getServiceInstance(handle);

Looks simple enough: connect() has probably not been called yet at this point, though I don't know why. The Cryostat server should probably be doing that, but also in the JFRJMXConnection class, there is a this.handle field as well as a getHandle() accessor, and the accessor method will initialize the connection before returning the reference if required. So anywhere within JFRJMXConnection that directly reads the field should probably call getHandle() instead, and I think that will fix this next bug.

@andrewazores
Copy link
Member

I guess that's not actually it. Tracing the calls:

This is in our -core code:

    public synchronized IFlightRecorderService getService()
            throws ConnectionException, IOException, ServiceNotAvailableException {
        if (!isConnected()) {
            connect();
        }
        IFlightRecorderService service = serviceFactory.getServiceInstance(getHandle());
        if (service == null || !isConnected()) {
            throw new ConnectionException(
                    String.format(
                            "Could not connect to remote target %s",
                            this.connectionDescriptor.createJMXServiceURL().toString()));
        }
        return service;
    }

Here with the serviceFactory we are dropping into JMC code. At this point, getHandle() is returning a non-null reference.

	@Override
	public IFlightRecorderService getServiceInstance(IConnectionHandle handle)
			throws ConnectionException, ServiceNotAvailableException {
		if (FlightRecorderServiceV2.isAvailable(handle)) {
			return new FlightRecorderServiceV2(handle);
		}
		return new FlightRecorderServiceV1(handle);
	}

We go into the V2 service constructor and pass in the same non-null reference.

	public FlightRecorderServiceV2(IConnectionHandle handle) throws ConnectionException, ServiceNotAvailableException {
		cfs = handle.getServiceOrThrow(ICommercialFeaturesService.class);
		if (!isDynamicFlightRecorderSupported(handle) && isFlightRecorderDisabled(handle)) {
			throw new ServiceNotAvailableException(""); //$NON-NLS-1$
		}
		if (JVMSupportToolkit.isFlightRecorderDisabled(handle, true)) {
			throw new ServiceNotAvailableException(""); //$NON-NLS-1$
		}
		connection = handle;
		helper = new FlightRecorderCommunicationHelperV2(handle.getServiceOrThrow(MBeanServerConnection.class));
		mbhs = handle.getServiceOrThrow(IMBeanHelperService.class);
		serverId = handle.getServerDescriptor().getGUID();
	}

Here the connection field is not assigned until after several precondition checks are performed. We see these in the stack trace @mwangggg posted.

	private boolean isFlightRecorderDisabled(IConnectionHandle handle) {
		if (cfs != null && isFlightRecorderCommercial()) {
			return !cfs.isCommercialFeaturesEnabled() || JVMSupportToolkit.isFlightRecorderDisabled(handle, false);
		} else {
			return JVMSupportToolkit.isFlightRecorderDisabled(handle, false);
		}
	}

Again, passing in the non-null reference as handle. Then we get to a call to isFlightRecorderCommercial() with no arguments.

	private boolean isFlightRecorderCommercial() {
		return ConnectionToolkit.isHotSpot(connection)
				&& !ConnectionToolkit.isJavaVersionAboveOrEqual(connection, JavaVersionSupport.JFR_NOT_COMMERCIAL);
	}

This is referencing the connection field, which has not yet been initialized. It should be the same reference as the handle we are passing through the stack, but that has not been done yet by the time this is called.

@andrewazores
Copy link
Member

We can patch this since it is occurring in our embedded JMC sources. Worth noting that our embedded JMC sources have diverged from what I see in JMC:

https://github.com/openjdk/jmc/blob/master/application/org.openjdk.jmc.rjmx.services.jfr/src/main/java/org/openjdk/jmc/rjmx/services/jfr/internal/FlightRecorderServiceV2.java

vs

https://github.com/cryostatio/cryostat-core/blob/main/src/main/java/org/openjdk/jmc/rjmx/services/jfr/internal/FlightRecorderServiceV2.java

It looks like our FlightRecorderServiceV2 is still a carry-forward from JMC 7.1.1, whereas this bug has probably already been fixed in JMC 8+.

@andrewazores
Copy link
Member

So, it looks like this ends up being a partial duplicate of #154 / #155 / #95 / #201 . At least some of our embedded JMC sources did not get upgraded from JMC7 to JMC8 equivalents.

@Josh-Matsuoka did you come across these classes in the course of #201 ? We should re-update again in this development cycle (sooner rather than later) to make sure all of our forked/embedded JMC sources are from the same version, at least.

@andrewazores
Copy link
Member

Or perhaps these JMX-related classes are ones that we're expecting the JMC Core artifact to extract and distribute so we can consume that as a proper dependency instead of embedded forked sources. I think we still want to upgrade the embedded sources in the meantime regardless.

@andrewazores
Copy link
Member

diff --git a/src/main/java/org/openjdk/jmc/rjmx/services/ICommercialFeaturesService.java b/src/main/java/org/openjdk/jmc/rjmx/services/ICommercialFeaturesService.java
index e08a443..6a950cc 100644
--- a/src/main/java/org/openjdk/jmc/rjmx/services/ICommercialFeaturesService.java
+++ b/src/main/java/org/openjdk/jmc/rjmx/services/ICommercialFeaturesService.java
@@ -1,6 +1,6 @@
 /*
  * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
- * 
+ *
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * The contents of this file are subject to the terms of either the Universal Permissive License
@@ -10,17 +10,17 @@
  *
  * Redistribution and use in source and binary forms, with or without modification, are permitted
  * provided that the following conditions are met:
- * 
+ *
  * 1. Redistributions of source code must retain the above copyright notice, this list of conditions
  * and the following disclaimer.
- * 
+ *
  * 2. Redistributions in binary form must reproduce the above copyright notice, this list of
  * conditions and the following disclaimer in the documentation and/or other materials provided with
  * the distribution.
- * 
+ *
  * 3. Neither the name of the copyright holder nor the names of its contributors may be used to
  * endorse or promote products derived from this software without specific prior written permission.
- * 
+ *
  * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
  * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
  * FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
@@ -52,4 +52,6 @@ public interface ICommercialFeaturesService {
 	 */
 	void enableCommercialFeatures() throws Exception;
 
+    boolean hasCommercialFeatures();
+
 }
diff --git a/src/main/java/org/openjdk/jmc/rjmx/services/internal/HotSpot23CommercialFeaturesService.java b/src/main/java/org/openjdk/jmc/rjmx/services/internal/HotSpot23CommercialFeaturesService.java
index 94549e3..dccc88f 100644
--- a/src/main/java/org/openjdk/jmc/rjmx/services/internal/HotSpot23CommercialFeaturesService.java
+++ b/src/main/java/org/openjdk/jmc/rjmx/services/internal/HotSpot23CommercialFeaturesService.java
@@ -1,6 +1,6 @@
 /*
  * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
- * 
+ *
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * The contents of this file are subject to the terms of either the Universal Permissive License
@@ -10,17 +10,17 @@
  *
  * Redistribution and use in source and binary forms, with or without modification, are permitted
  * provided that the following conditions are met:
- * 
+ *
  * 1. Redistributions of source code must retain the above copyright notice, this list of conditions
  * and the following disclaimer.
- * 
+ *
  * 2. Redistributions in binary form must reproduce the above copyright notice, this list of
  * conditions and the following disclaimer in the documentation and/or other materials provided with
  * the distribution.
- * 
+ *
  * 3. Neither the name of the copyright holder nor the names of its contributors may be used to
  * endorse or promote products derived from this software without specific prior written permission.
- * 
+ *
  * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
  * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
  * FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
@@ -96,4 +96,9 @@ public class HotSpot23CommercialFeaturesService implements ICommercialFeaturesSe
 		server.getMBeanInfo(candidateObjectName);
 		return candidateObjectName;
 	}
+
+	@Override
+	public boolean hasCommercialFeatures() {
+		return false;
+	}
 }
diff --git a/src/main/java/org/openjdk/jmc/rjmx/services/internal/Jdk11CommercialFeaturesService.java b/src/main/java/org/openjdk/jmc/rjmx/services/internal/Jdk11CommercialFeaturesService.java
index 8453526..1ec3d21 100644
--- a/src/main/java/org/openjdk/jmc/rjmx/services/internal/Jdk11CommercialFeaturesService.java
+++ b/src/main/java/org/openjdk/jmc/rjmx/services/internal/Jdk11CommercialFeaturesService.java
@@ -1,6 +1,6 @@
 /*
  * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
- * 
+ *
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * The contents of this file are subject to the terms of either the Universal Permissive License
@@ -10,17 +10,17 @@
  *
  * Redistribution and use in source and binary forms, with or without modification, are permitted
  * provided that the following conditions are met:
- * 
+ *
  * 1. Redistributions of source code must retain the above copyright notice, this list of conditions
  * and the following disclaimer.
- * 
+ *
  * 2. Redistributions in binary form must reproduce the above copyright notice, this list of
  * conditions and the following disclaimer in the documentation and/or other materials provided with
  * the distribution.
- * 
+ *
  * 3. Neither the name of the copyright holder nor the names of its contributors may be used to
  * endorse or promote products derived from this software without specific prior written permission.
- * 
+ *
  * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
  * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
  * FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
@@ -35,7 +35,7 @@ package org.openjdk.jmc.rjmx.services.internal;
 import org.openjdk.jmc.rjmx.services.ICommercialFeaturesService;
 
 public class Jdk11CommercialFeaturesService implements ICommercialFeaturesService {
-	
+
 	@Override
 	public boolean isCommercialFeaturesEnabled() {
 		return true;
@@ -45,4 +45,9 @@ public class Jdk11CommercialFeaturesService implements ICommercialFeaturesServic
 	public void enableCommercialFeatures() throws Exception {
 		// Noop
 	}
+
+	@Override
+	public boolean hasCommercialFeatures() {
+		return false;
+	}
 }
diff --git a/src/main/java/org/openjdk/jmc/rjmx/services/jfr/internal/FlightRecorderServiceV2.java b/src/main/java/org/openjdk/jmc/rjmx/services/jfr/internal/FlightRecorderServiceV2.java
index dd1e423..bcd09f3 100644
--- a/src/main/java/org/openjdk/jmc/rjmx/services/jfr/internal/FlightRecorderServiceV2.java
+++ b/src/main/java/org/openjdk/jmc/rjmx/services/jfr/internal/FlightRecorderServiceV2.java
@@ -1,6 +1,6 @@
 /*
- * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
- * 
+ * Copyright (c) 2018, 2019, Oracle and/or its affiliates. All rights reserved.
+ *
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * The contents of this file are subject to the terms of either the Universal Permissive License
@@ -10,17 +10,17 @@
  *
  * Redistribution and use in source and binary forms, with or without modification, are permitted
  * provided that the following conditions are met:
- * 
+ *
  * 1. Redistributions of source code must retain the above copyright notice, this list of conditions
  * and the following disclaimer.
- * 
+ *
  * 2. Redistributions in binary form must reproduce the above copyright notice, this list of
  * conditions and the following disclaimer in the documentation and/or other materials provided with
  * the distribution.
- * 
+ *
  * 3. Neither the name of the copyright holder nor the names of its contributors may be used to
  * endorse or promote products derived from this software without specific prior written permission.
- * 
+ *
  * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
  * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
  * FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
@@ -103,17 +103,14 @@ public class FlightRecorderServiceV2 implements IFlightRecorderService {
 	}
 
 	private boolean isDynamicFlightRecorderSupported(IConnectionHandle handle) {
-		return ConnectionToolkit.isHotSpot(handle)
-				&& ConnectionToolkit.isJavaVersionAboveOrEqual(handle, JavaVersionSupport.DYNAMIC_JFR_SUPPORTED);
-	}
-
-	private boolean isFlightRecorderCommercial() {
-		return ConnectionToolkit.isHotSpot(connection)
-				&& !ConnectionToolkit.isJavaVersionAboveOrEqual(connection, JavaVersionSupport.JFR_NOT_COMMERCIAL);
+		// All OpenJDK versions of JFR support dynamic enablement of JFR, so if there are no commercial features in play
+		// all is A-OK.
+		return !cfs.hasCommercialFeatures() || (ConnectionToolkit.isHotSpot(handle)
+				&& ConnectionToolkit.isJavaVersionAboveOrEqual(handle, JavaVersionSupport.DYNAMIC_JFR_SUPPORTED));
 	}
 
 	private boolean isFlightRecorderDisabled(IConnectionHandle handle) {
-		if (cfs != null && isFlightRecorderCommercial()) {
+		if (cfs != null && cfs.hasCommercialFeatures()) {
 			return !cfs.isCommercialFeaturesEnabled() || JVMSupportToolkit.isFlightRecorderDisabled(handle, false);
 		} else {
 			return JVMSupportToolkit.isFlightRecorderDisabled(handle, false);
@@ -126,6 +123,7 @@ public class FlightRecorderServiceV2 implements IFlightRecorderService {
 
 	public FlightRecorderServiceV2(IConnectionHandle handle) throws ConnectionException, ServiceNotAvailableException {
 		cfs = handle.getServiceOrThrow(ICommercialFeaturesService.class);
+
 		if (!isDynamicFlightRecorderSupported(handle) && isFlightRecorderDisabled(handle)) {
 			throw new ServiceNotAvailableException(""); //$NON-NLS-1$
 		}
@@ -418,8 +416,7 @@ public class FlightRecorderServiceV2 implements IFlightRecorderService {
 			Long id = (Long) helper.invokeOperation("cloneRecording", //$NON-NLS-1$
 					descriptor.getId(), Boolean.TRUE);
 			IMutableConstrainedMap<String> options = getEmptyRecordingOptions();
-			options.put(RecordingOptionsBuilder.KEY_NAME,
-					String.format("Clone of %s", descriptor.getName()));
+			options.put(RecordingOptionsBuilder.KEY_NAME, String.format("Clone of %s", descriptor.getName()));
 			helper.invokeOperation("setRecordingOptions", id, toTabularData(options)); //$NON-NLS-1$
 			return getUpdatedRecordingDescriptor(id);
 		} catch (Exception e) {
@@ -480,7 +477,7 @@ public class FlightRecorderServiceV2 implements IFlightRecorderService {
 	@Override
 	public boolean isEnabled() {
 		if (!wasEnabled) {
-			boolean isEnabled = isFlightRecorderCommercial() ? cfs.isCommercialFeaturesEnabled()
+			boolean isEnabled = cfs.hasCommercialFeatures() ? cfs.isCommercialFeaturesEnabled()
 					: isAvailable(connection);
 			if (isEnabled) {
 				wasEnabled = true;
@@ -500,3 +497,4 @@ public class FlightRecorderServiceV2 implements IFlightRecorderService {
 		}
 	}
 }
+

I prepared this patch by simply copy-pasting the JMC8 version of the FlightRecorderServiceV2 file into the embedded sources, and then fixing up a couple of small compilation failures that came as a result to do with another change to the ICommercialFeaturesService. This is just patching that one particular class up to the JMC8+ version.

After a rebuild:

image

I appear to be able to connect to @roberttoyonaga 's native image application and read some basic JFR information.

@mwangggg mwangggg marked this pull request as ready for review May 26, 2023 15:05
@mwangggg mwangggg force-pushed the 217-NPE-when-connecting-native-image branch from 3fc20a3 to f61cd54 Compare May 26, 2023 19:11
andrewazores
andrewazores previously approved these changes May 26, 2023
Copy link
Member

@andrewazores andrewazores left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change looks good, but we should wait for #223 before merging this.

@andrewazores andrewazores merged commit 752b986 into cryostatio:main Jun 7, 2023
5 checks passed
mergify bot pushed a commit that referenced this pull request Jun 13, 2023
andrewazores pushed a commit that referenced this pull request Jun 13, 2023
@mwangggg mwangggg deleted the 217-NPE-when-connecting-native-image branch July 4, 2023 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Status: Done
Development

Successfully merging this pull request may close these issues.

NPE when connecting to native image
2 participants