StoreClientFactory leaves mbeans around in JMX after close() is called. (Voldemort 1.2.0)
The culprits seem to be the mbeans under the object names
If I create a new SCF and close it, 10 times. I end up with 10 stores in JMX..
Please see the attached screen shot...
The other issue caused by this is that the previous class instances will not be garbage collected because they are still referenced by the mbean server..
Also I am not sure why there is a count appended to the mbeans when there is only one instance of the SCF around? Is there a static variable being used?
I only ask this because I am running the voldemort client in OSGi and this could cause unexpected consequences ..
Let me explain why we need the counts. Sometimes people create multiple storeClientFactories for the same cluster within a single jvm process. In these cases, we needed to distinguish between the mbeans from different factories, since they could be configured differently. I think creating named factories might allieviate the confusion. But the problem I am afraid will still exist..
The best practice is to use one factory per cluster. What version are you running? I committed a bunch of fixes that cleaned up stuff around this.. eg all mbeans from the same factory will have the same number. Let me know and we can debug this together.
What you say about multiple StoreClientFactory's makes sense.
Although in my case I only have one, It's just the previous instances don't clean up after them selves.
This should be pretty easy to reproduce in a unit test, create a SCF then close it.. keep JVM running and check JMX with jconsole.
Would it be better to put a count/hashcode in the path versus the actual mbean name. Think of the case when you have a store called "test2" the second instance would be called "test21"..
We could add a "-" so that it appears as "test2-1".. Let me try a simple test to confirm that latest behavior because I definitely see the unregister call in PipelineRoutedStore::close()
Let me know if there is anything I can do to help debug the issue.
I reproduced the problem and seems like, the voldemort.store.routed and voldemort.store.stats mbeans are not unregistered after factory.close(). Given the unregister code is there, I think somehow the close() is not being invoked.
Could I also recommend you use something like a semaphore for the count? So that if you create 10 SCF's the count is 10, if you close 5 of them, the count does back down to 5. This way when another SCF is create the count goes up to 6 and not 11.
This will makes it more intuitive when looking at the stores in JMX. Especially when things are dynamically deployed and un-deployed in JMX.
Makes sense. We could simply use a AtomicLong.. But the problem here seems to be that some code path does not call close().
I have limited cycles now. So, wish to know if this is a blocker for you.
I had the same problem. I've some code that fixes partially the problem. Take a look here: ctasada@cf56864
My 2 cents.
It's not that big of a deal.
If we were to keep the container running for a long time and redeploy bundles that use voldemort often enough I would expect to see perm gen space issues (classes from the old bundles not being unloaded)..
But thankfully we don't do that much redeploying, at least in production.
But it is really annoying when testing stuff in dev, with tens of stores in JMX..
Thanks for the pointer Carlos. Will chase down the disconnect in the code path when I free up a bit. Assigning this to me
Carlos/Aaron, btw let me know if you want to take this up though. I would like to first understand why the close() is not being invoked.