Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

All JMX Beans disappear when single Hazelcast instance is shut down in the same JVM #7467

Closed
mirkosertic opened this issue Feb 5, 2016 · 7 comments

Comments

Projects
None yet
5 participants
@mirkosertic
Copy link

commented Feb 5, 2016

I am running multiple Hazelcast 3.5.5 instances in the same JVM. When i programatically shutdown one instance, all other existing instances are also deregistered as JMX Beans, but they are still alive and running. Creating a new instance programatically then only registers the new instance, all others are no longer available via JMX.

All running Hazelcast Instances should always be visible by JMX (if JMX is enabled).

@mirkosertic

This comment has been minimized.

Copy link
Author

commented Feb 5, 2016

This seems to happen if two applications using Hazelcast are deployed in one Tomcat running in the same JVM. Shutting down the last Instance of one application also unregisters the still living instances in the other application.

@mirkosertic

This comment has been minimized.

Copy link
Author

commented Feb 5, 2016

This should do the trick:

    public static void shutdownAll() {
        MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
        try {
            Set<ObjectName> entries = mbs.queryNames(new ObjectName(DOMAIN + ":*"), null);
            for (ObjectName name : entries) {
                if (mbs.getClassLoaderFor(name) == ManagementService.class.getClassLoader()) {
                    if (mbs.isRegistered(name)) {
                        mbs.unregisterMBean(name);
                    }
                }
            }
        } catch (Exception e) {
            Logger.getLogger(ManagementService.class.getName())
                    .log(Level.WARNING, "Error while shutting down all jmx services...", e);
        }
    }
@emrahkocaman

This comment has been minimized.

Copy link
Contributor

commented Feb 9, 2016

Hi @mirkosertic ,

I'm not able to reproduce your case. When HazelcastInstance#shutdown method is called,
it'll call ManagementService#destroy (shutdownAll should not be called) method which unregisters the MBeans of only related instance. (by instance name). Not all of MBeans are unregistered.

Please see;

https://github.com/hazelcast/hazelcast/blob/master/hazelcast/src/main/java/com/hazelcast/internal/jmx/ManagementService.java#L92

Here is the test that I've created and not able to reproduce your issue.

https://gist.github.com/emrahkocaman/92755f326c186fbf2579

@mirkosertic

This comment has been minimized.

Copy link
Author

commented Feb 9, 2016

@emrahkocaman

This comment has been minimized.

Copy link
Contributor

commented Feb 22, 2016

@mirkosertic, but this happens only if INSTANCE_MAP size is 0. All instances are needed to be shutdown for this to work, am I missing something?

@emrahkocaman

This comment has been minimized.

Copy link
Contributor

commented Mar 11, 2016

@mirkosertic Can you please check my previous comment?

@mirkosertic

This comment has been minimized.

Copy link
Author

commented Mar 21, 2016

I have checked the comment. Indeed, it is called if INSTANCE_MAP size is zero.

We are running two web applications on the same tomcat in the same JVM.
In each web application a single Hazelcast instance is created.
There is only one PlatformMBeanServer for the JVM, and the two Hazelcast instances are registered as MBeans on the single PlatformMBeanServer.

The tricky part is the ClassLoader. The Hazelcast JAR is distributed in every WAR file, and hence every web application has its own Classloader. And from the point of view of a single web application, INSTANCE_MAP is one and not two. And now if the single Hazelcast instance is shutdown, INSTANCE_MAP will become 0, and this triggers the shutdown of EVERY registered instance MBean in the current JVM, and this will also kill the remaining instance.

I hope this makes the Bug clearer...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.