Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

StoreClientFactory leaves mbeans around in JMX after close() is called. #129

Open
aaronjwhiteside opened this Issue · 13 comments

3 participants

@aaronjwhiteside

StoreClientFactory leaves mbeans around in JMX after close() is called. (Voldemort 1.2.0)

The culprits seem to be the mbeans under the object names

voldemort.store.stats:
voldemort.store.routed:

If I create a new SCF and close it, 10 times. I end up with 10 stores in JMX..

Please see the attached screen shot...

Screen Shot 2013-01-25 at 1 54 28 PM

The other issue caused by this is that the previous class instances will not be garbage collected because they are still referenced by the mbean server..

@aaronjwhiteside

Also I am not sure why there is a count appended to the mbeans when there is only one instance of the SCF around? Is there a static variable being used?

I only ask this because I am running the voldemort client in OSGi and this could cause unexpected consequences ..

@vinothchandar
Collaborator

Hi aaron,

Let me explain why we need the counts. Sometimes people create multiple storeClientFactories for the same cluster within a single jvm process. In these cases, we needed to distinguish between the mbeans from different factories, since they could be configured differently. I think creating named factories might allieviate the confusion. But the problem I am afraid will still exist..

The best practice is to use one factory per cluster. What version are you running? I committed a bunch of fixes that cleaned up stuff around this.. eg all mbeans from the same factory will have the same number. Let me know and we can debug this together.

@aaronjwhiteside

1.2.0

What you say about multiple StoreClientFactory's makes sense.

Although in my case I only have one, It's just the previous instances don't clean up after them selves.

This should be pretty easy to reproduce in a unit test, create a SCF then close it.. keep JVM running and check JMX with jconsole.

@aaronjwhiteside

Would it be better to put a count/hashcode in the path versus the actual mbean name. Think of the case when you have a store called "test2" the second instance would be called "test21"..

@vinothchandar
Collaborator

We could add a "-" so that it appears as "test2-1".. Let me try a simple test to confirm that latest behavior because I definitely see the unregister call in PipelineRoutedStore::close()
https://github.com/voldemort/voldemort/blob/release-1.2.0/src/java/voldemort/store/routed/PipelineRoutedStore.java

@aaronjwhiteside

Let me know if there is anything I can do to help debug the issue.

@vinothchandar
Collaborator

I reproduced the problem and seems like, the voldemort.store.routed and voldemort.store.stats mbeans are not unregistered after factory.close(). Given the unregister code is there, I think somehow the close() is not being invoked.

@aaronjwhiteside

Could I also recommend you use something like a semaphore for the count? So that if you create 10 SCF's the count is 10, if you close 5 of them, the count does back down to 5. This way when another SCF is create the count goes up to 6 and not 11.

This will makes it more intuitive when looking at the stores in JMX. Especially when things are dynamically deployed and un-deployed in JMX.

@vinothchandar
Collaborator

Makes sense. We could simply use a AtomicLong.. But the problem here seems to be that some code path does not call close().

I have limited cycles now. So, wish to know if this is a blocker for you.

@ctasada

Hi guys,

I had the same problem. I've some code that fixes partially the problem. Take a look here: ctasada@cf56864

My 2 cents.

@aaronjwhiteside

It's not that big of a deal.

If we were to keep the container running for a long time and redeploy bundles that use voldemort often enough I would expect to see perm gen space issues (classes from the old bundles not being unloaded)..

But thankfully we don't do that much redeploying, at least in production.

But it is really annoying when testing stuff in dev, with tens of stores in JMX..

@vinothchandar
Collaborator

Thanks for the pointer Carlos. Will chase down the disconnect in the code path when I free up a bit. Assigning this to me

@vinothchandar
Collaborator

Carlos/Aaron, btw let me know if you want to take this up though. I would like to first understand why the close() is not being invoked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.