Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot find autostart amx option in Glassfish 7 #24762

Closed
escay opened this issue Jan 24, 2024 · 8 comments · Fixed by #24800
Closed

Cannot find autostart amx option in Glassfish 7 #24762

escay opened this issue Jan 24, 2024 · 8 comments · Fixed by #24800
Milestone

Comments

@escay
Copy link
Contributor

escay commented Jan 24, 2024

Environment Details

  • GlassFish Version (and build number): 7.0.10
  • JDK version: jdk-17.0.8.1_1-openjdk-adoptium
  • OS: Windows 10
  • Database: Postgresql

Problem Description

Coming from Payara 5 and 6 I can enable amx automatically on startup using
<amx-configuration enabled="true"></amx-configuration>
in domain.xml

However in Glassfish 7 I cannot seem to find such an option.
Via jconsole I can see the amx-support MBean and I can select the bootAMX() operation to start amx.

In the administration guide:
https://glassfish.org/docs/7.0.10/administration-guide.html#using-rest-interfaces-to-administer-glassfish-server
I can see a minimal mention of amx.

The goal I would like to achieve:

  • from within an application / ear file I would like to ask some server state / settings from an M-Bean or REST call
  • example: ask amx:pp=/domain/configs/config[server-config],type=transaction-service for attribute TimeoutInSeconds

Steps to reproduce

Deploy ear file, call java.lang.management.ManagementFactory.getPlatformMBeanServer();
Perform query to "amx" object.
Object amx:pp=/domain/configs/config[server-config],type=transaction-service not found due to amx not being started.

Workaround start amx from the ear file using MBeanServer like:

final MBeanServer platformMBeanServer = ManagementFactory.getPlatformMBeanServer();
try {
	ObjectName objectName = new ObjectName("amx-support:type=boot-amx");
	if (platformMBeanServer.isRegistered(objectName)) {
		platformMBeanServer.invoke(objectName, "bootAMX", new Object[0], new String[0]);
	} else {
		log.warn("amx-support:type=boot-amx not available");
	}
} catch (Exception e) {
	log.error("Unable to start AMX", e);
}

Impact of Issue

Main question really is:

  • can amx be started automatically like in Payara via domain.xml? if so: then the bug would be: some documentation is missing about the existence of the bootAmx operation for this purpose
  • is amx still valid to use? or is it deprecated and should the management REST Api be used instead?
    -- But it adds quite some overhead to get some server information from within the ear file.

Logging issue

When starting AMX in a clean domain you get errors / exceptions in INFO messages:
IllegalArgumentException: ResourceRef refers to non-existent resource: GlassFishConfigBean.com.sun.enterprise.config.serverbeans.ResourceRef

[#|2024-01-24T13:26:57.296964+01:00|INFO|GF 7.0.10|jakarta.enterprise.system.tools.amx|_ThreadID=244;_ThreadName=Thread-34;_LevelValue=800;_MessageID=NCLS-COM-00008;|
  AMX Startup Service: AMXLoader failed to load {0}
javax.management.RuntimeMBeanException: RuntimeException thrown in postRegister method: rethrowing <java.lang.RuntimeException: javax.management.RuntimeMBeanException: RuntimeException thrown in postRegister method: rethrowing <java.lang.IllegalArgumentException: ResourceRef refers to non-existent resource: GlassFishConfigBean.com.sun.enterprise.config.serverbeans.ResourceRef>, but keeping the MBean registered>, but keeping the MBean registered
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1023)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:968)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:895)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320)
        at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:523)
        at com.sun.enterprise.v3.admin.DynamicInterceptor.registerMBean(DynamicInterceptor.java:396)
        at org.glassfish.admin.amx.impl.j2ee.loader.AMXJ2EEStartupService.loadAMXMBeans(AMXJ2EEStartupService.java:216)
        at org.glassfish.admin.amx.impl.AMXStartupService$AMXLoaderThread.run(AMXStartupService.java:265)
Caused by: java.lang.RuntimeException: javax.management.RuntimeMBeanException: RuntimeException thrown in postRegister method: rethrowing <java.lang.IllegalArgumentException: ResourceRef refers to non-existent resource: GlassFishConfigBean.com.sun.enterprise.config.serverbeans.ResourceRef>, but keeping the MBean registered
        at org.glassfish.admin.amx.impl.mbean.AMXImplBase.registerChild(AMXImplBase.java:954)
        at org.glassfish.admin.amx.impl.j2ee.J2EEDomainImpl.registerChildren(J2EEDomainImpl.java:82)
        at org.glassfish.admin.amx.impl.mbean.AMXImplBase.postRegisterHook(AMXImplBase.java:898)
        at org.glassfish.admin.amx.impl.mbean.MBeanImplBase.postRegister(MBeanImplBase.java:327)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1018)
        ... 7 more
Caused by: javax.management.RuntimeMBeanException: RuntimeException thrown in postRegister method: rethrowing <java.lang.IllegalArgumentException: ResourceRef refers to non-existent resource: GlassFishConfigBean.com.sun.enterprise.config.serverbeans.ResourceRef>, but keeping the MBean registered
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1023)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:968)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:895)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320)
        at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:523)
        at com.sun.enterprise.v3.admin.DynamicInterceptor.registerMBean(DynamicInterceptor.java:396)
        at org.glassfish.admin.amx.impl.mbean.AMXImplBase.registerChild(AMXImplBase.java:950)
        ... 11 more
Caused by: java.lang.IllegalArgumentException: ResourceRef refers to non-existent resource: GlassFishConfigBean.com.sun.enterprise.config.serverbeans.ResourceRef
        at org.glassfish.admin.amx.impl.j2ee.RegistrationSupport.processResourceRef(RegistrationSupport.java:554)
        at org.glassfish.admin.amx.impl.j2ee.RegistrationSupport$RefListener.startListening(RegistrationSupport.java:673)
        at org.glassfish.admin.amx.impl.j2ee.RegistrationSupport.start(RegistrationSupport.java:136)
        at org.glassfish.admin.amx.impl.j2ee.J2EEServerImpl.registerChildren(J2EEServerImpl.java:98)
        at org.glassfish.admin.amx.impl.j2ee.DASJ2EEServerImpl.registerChildren(DASJ2EEServerImpl.java:49)
        at org.glassfish.admin.amx.impl.mbean.AMXImplBase.postRegisterHook(AMXImplBase.java:898)
        at org.glassfish.admin.amx.impl.mbean.MBeanImplBase.postRegister(MBeanImplBase.java:327)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1018)
        ... 17 more
|#]
[#|2024-01-24T14:21:04.131441+01:00|INFO|GF 7.0.10|jakarta.enterprise.system.tools.amx|_ThreadID=137;_ThreadName=AutoDeployer;_LevelValue=800;_MessageID=NCLS-COM-00009;|
  AMX Startup Service: AMX ready for use, DomainRoot amx:pp=,type=domain-root|#]

however AMX Startup completed and amx MBeans are available and can be used.

Workaround to hide these messages is to change logging.properties to change existing line:
jakarta.enterprise.system.tools.amx.level=INFO
to
jakarta.enterprise.system.tools.amx.level=WARNING
but this will hide the
AMX Startup Service: AMX ready for use
message as well.

And I get some System.out log lines 8 times the same __defaultManagedThreadFactory already exists exception, which I would not expect:

[#|2024-01-24T14:45:28.562217+01:00|INFO|GF 7.0.10|jakarta.enterprise.logging.stdout|_ThreadID=224;_ThreadName=AMXConfigLoader.AMXConfigLoaderThread;_LevelValue=800;|
  javax.management.InstanceAlreadyExistsException: "amx:pp=/domain/servers/server[server],type=resource-ref,name=concurrent/__defaultManagedThreadFactory"
java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1865)
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:960)
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:895)
java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320)
java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:523)
com.sun.enterprise.v3.admin.DynamicInterceptor.registerMBean(DynamicInterceptor.java:396)
org.glassfish.admin.amx.impl.config.AMXConfigLoader.createAndRegister(AMXConfigLoader.java:478)
org.glassfish.admin.amx.impl.config.AMXConfigLoader._registerConfigBeanAsMBean(AMXConfigLoader.java:456)
org.glassfish.admin.amx.impl.config.AMXConfigLoader.registerConfigBeanAsMBean(AMXConfigLoader.java:428)
org.glassfish.admin.amx.impl.config.AMXConfigLoader$AMXConfigLoaderThread.registerOne(AMXConfigLoader.java:357)
org.glassfish.admin.amx.impl.config.AMXConfigLoader$AMXConfigLoaderThread.doRun(AMXConfigLoader.java:398)
org.glassfish.admin.amx.impl.config.AMXConfigLoader$AMXConfigLoaderThread.run(AMXConfigLoader.java:371)|#]

Ideally these should be behind some log level.

@escay
Copy link
Contributor Author

escay commented Jan 25, 2024

About the 8 times system out __defaultContextService, it is coming from:

com.sun.jmx.mbeanserver.Repository.addMBean(DynamicMBean, ObjectName, RegistrationContext)
 -> com.sun.jmx.mbeanserver.Repository.addMBean(DynamicMBean, ObjectName, RegistrationContext)
  -> moiTb already contains "amx:pp=/domain/resources,type=context-service,name=concurrent/__defaultContextService"

throw new InstanceAlreadyExistsException(name.toString());

the stack:

Daemon System Thread [AMXConfigLoader.AMXConfigLoaderThread] (Suspended (exception InstanceAlreadyExistsException))	
	Repository.addMBean(DynamicMBean, ObjectName, Repository$RegistrationContext) line: 436	
	DefaultMBeanServerInterceptor.registerWithRepository(Object, DynamicMBean, ObjectName) line: 1865	
	DefaultMBeanServerInterceptor.registerDynamicMBean(String, DynamicMBean, ObjectName) line: 960	
	DefaultMBeanServerInterceptor.registerObject(String, Object, ObjectName) line: 895	
	DefaultMBeanServerInterceptor.registerMBean(Object, ObjectName) line: 320	
	JmxMBeanServer.registerMBean(Object, ObjectName) line: 523	
	DynamicInterceptor.registerMBean(Object, ObjectName) line: 396	
	AMXConfigLoader.createAndRegister(ConfigBean, ObjectName) line: 478	
	AMXConfigLoader._registerConfigBeanAsMBean(ConfigBean, ConfigBean) line: 456	
	AMXConfigLoader.registerConfigBeanAsMBean(ConfigBean) line: 428	
	AMXConfigLoader$AMXConfigLoaderThread.registerOne(PendingConfigBeanJob) line: 357	
	AMXConfigLoader$AMXConfigLoaderThread.doRun() line: 398	
	AMXConfigLoader$AMXConfigLoaderThread.run() line: 371	

JmxMBeanServer just returns the InstanceAlreadyExistsException to glassfish code:

org.glassfish.admin.amx.impl.config.AMXConfigLoader.createAndRegister(ConfigBean, ObjectName) does:

        } catch (final JMException e) {
            debug(ExceptionUtil.toString(e));

which does:

    private static void debug(final String s) {
        System.out.println(s);
    }

this could use some logger instead.

AMXConfigImpl -> uses java.util.logging.Logger
AMXConfigStartupService-> uses System.out as well via debug
AMXProxyHandler -> uses System.out but via sdebug unused method
there are also a lot of specifics like org.glassfish.admin.amx.util.AMXDebug internal

Not sure how this should be fixed. I would use the AMXConfigImpl approach in AMXConfigLoader as well.

@dmatej
Copy link
Contributor

dmatej commented Jan 25, 2024

AMX in GF will need quite radical maintenance, it seems to me that the code is in a "prototype state" for some 20 years.
I want to "start a buldozer and go through it", but it will need some time.
Thanks for reporting issues like this, to answer ...

  • Yes, the startup could be improved.
  • Yes, I believe AMX (JMX) is still existing and maintained protocol in Java, but for management I would really recommend REST, personally I always used JMX just for monitoring (Nagios, JVisualVM, JConsole, jcmd, etc.)

@OndroMih
Copy link
Contributor

OndroMih commented Feb 2, 2024

Hi, I'll answer the initial question whether AMX can be enabled on boot. GlassFish doesn't support the boot AMX at startup option. This was added to Payara but haven't been added back to GlassFish yet. You can boot AMX by accessing the JMX server remotely, then AMX starts automatically. If you connect to the JMX server locally, you also need to run the bootAMX action on a GlassFish JMX bean via the JMX connection, for example with this Java code:

final MBeanServer server = ManagementFactory.getPlatformMBeanServer();
try {
    final ObjectName bootMBean = new ObjectName("amx-support:type=boot-amx");
    server.invoke(bootMBean, "bootAMX", null, null);
} catch (final MBeanException | ReflectionException | MalformedObjectNameException e) {
    ...
} catch (final InstanceNotFoundException e) {
    ...
}

@OndroMih
Copy link
Contributor

OndroMih commented Feb 4, 2024

Hi @escay , the "IllegalArgumentException: ResourceRef refers to non-existent resource: " message is logged because it seems there's an error in your domain.xml file. Some server in the <servers> element probably refers to a resources with a resource-ref element, but the resource doesn't exist in the domain resources ( in the top-level resources element). It doesn't have a huge impact on the JMX server because the exception is thrown at the end of the initialization, but it's still possible that MX beans for some server resources that are defined after are not registered.

P.S. I admit that the error message could be improved to explain what actually is wrong and how to fix it. And it should be logged as a Warning, not an INFO message.

@escay
Copy link
Contributor Author

escay commented Feb 4, 2024

I tried a breakpoint at the "ResourceRef refers to non-existent resource" in org.glassfish.admin.amx.impl.j2ee.RegistrationSupport.processResourceRef(ResourceRef) but I could not figure out which resource was being looked at. I will have a look again with this knowledge.

The manual startup in my code using the bootAMX invoke is working fine for last 2 weeks, it is a good workaround.
So in general this issue could be closed.

I will make a pull request for solving the System out logging improvement.

@escay
Copy link
Contributor Author

escay commented Feb 5, 2024

I can confirm the "IllegalArgumentException: ResourceRef refers to non-existent resource" was coming from an incorrect resource-ref: <resource-ref ref="jdbc/sample"></resource-ref> in my domain.xml.
Cleaning up the domain.xml solved the exception problem.

Changing the org.glassfish.config.support.GlassFishConfigBean code to list the attributes:

    public String toString() {
        StringJoiner attributeJoiner = new StringJoiner(",");
        for (String attributeName : getAttributeNames()) {
            attributeJoiner.add(attributeName + "=" + attribute(attributeName));
        }
        return "GlassFishConfigBean." + getProxyType().getName() + "(" + attributeJoiner.toString() + ")";
    }

Makes the exception a bit more informative to log the ref="jdbc/sample" and maybe gives a hint this way to the domain.xml.
Example result in my situation:

[#|2024-02-05T13:59:40.691232+01:00|INFO|GF 7.0.10|jakarta.enterprise.system.tools.amx|_ThreadID=163;_ThreadName=Thread-21;_LevelValue=800;_MessageID=NCLS-COM-00008;|
  AMX Startup Service: AMXLoader failed to load {0}
javax.management.RuntimeMBeanException: RuntimeException thrown in postRegister method: rethrowing <java.lang.RuntimeException: javax.management.RuntimeMBeanException: RuntimeException thrown in postRegister method: rethrowing <java.lang.IllegalArgumentException: ResourceRef refers to non-existent resource: GlassFishConfigBean.com.sun.enterprise.config.serverbeans.ResourceRef(ref=jdbc/sample)>, but keeping the MBean registered>, but keeping the MBean registered
Caused by: java.lang.RuntimeException: javax.management.RuntimeMBeanException: RuntimeException thrown in postRegister method: rethrowing <java.lang.IllegalArgumentException: ResourceRef refers to non-existent resource: GlassFishConfigBean.com.sun.enterprise.config.serverbeans.ResourceRef(ref=jdbc/sample)>, but keeping the MBean registered
Caused by: javax.management.RuntimeMBeanException: RuntimeException thrown in postRegister method: rethrowing <java.lang.IllegalArgumentException: ResourceRef refers to non-existent resource: GlassFishConfigBean.com.sun.enterprise.config.serverbeans.ResourceRef(ref=jdbc/sample)>, but keeping the MBean registered
Caused by: java.lang.IllegalArgumentException: ResourceRef refers to non-existent resource: GlassFishConfigBean.com.sun.enterprise.config.serverbeans.ResourceRef(ref=jdbc/sample)
|#]

I do not know what the impact is of changing the GlassFishConfigBean toString method.

escay added a commit to escay/glassfish that referenced this issue Feb 9, 2024
…oggerInfo

instead instead of System.out. Removed all debug System out methods to
avoid using them in the future. No system.out used anymore in amx-core
module.
escay added a commit to escay/glassfish that referenced this issue Feb 9, 2024
…istent

resource" exception by logging the attributes of the ResourceRef that
was incorrect.
@escay
Copy link
Contributor Author

escay commented Feb 9, 2024

I created 2 commits.

  1. 1e9626d - upgrades System out to the logger and removes unused System out debug methods
  2. 2d9c87d - changes toString in case of incorrect ResourceRef, so the exception logs some context

Both do not solve or implement the autostart, they only improve the error messages I encountered.
Perhaps I should have created 2 separate new small issues?

Both do not contain tests. Ideally I would have added a unit test for the 2nd item. But I could not figure out how to load a domain.xml in a unit test and perform the same code as AMX is doing in the stacktrace above. could not figure out how Junit5 extension org.glassfish.config.api.test.ConfigApiJunit5Extension was supposed to work, the code does not seem to be in use, but it seems a basis to load a domain.xml.

Let me know if I should make pull requests for these two commits or not.

@dmatej
Copy link
Contributor

dmatej commented Feb 9, 2024

Let me know if I should make pull requests for these two commits or not.

You can put both commits into a single PR, they are both related and if we would create a PR for every commit, GlassFish would never move - any help is welcome. The PR will be checked by Jenkins and if it would not break any of existing thousands of tests and it would pass also manual review, we will merge it. I quickly looked over both commits and I think it is certainly a step forward, thank you!

escay added a commit to escay/glassfish that referenced this issue Feb 9, 2024
…oggerInfo

instead instead of System.out. Removed all debug System out methods to
avoid using them in the future. No system.out used anymore in amx-core
module.
Improves "ResourceRef refers to non-existent
resource" exception by logging the attributes of the ResourceRef that
was incorrect.
arjantijms added a commit that referenced this issue Feb 10, 2024
Fixes issue #24762 Use AMXLoggerInfo io System.out and add attribute info in toString method.
@dmatej dmatej added this to the 7.0.13 milestone Feb 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants