Client and server memory leaks while destroying FencedLocks [HZ-3039]

[arodionov]

The local FencedLock proxy object is not removed from LockService#proxies map while the Lock#destroy() called

Steps to reproduce:

        HazelcastInstance instance = Hazelcast.newHazelcastInstance(config);
        Hazelcast.newHazelcastInstance(config);
        Hazelcast.newHazelcastInstance(config);
        CPSubsystem cpSubsystem = instance.getCPSubsystem();

        for (int i = 0; i < 100; i++) {
            FencedLock lock = cpSubsystem.getLock("iLock_" + i);
            lock.lock();
            lock.unlock();
            lock.destroy();
        }

The LockService#proxies map contains 100 FencedLockProxy objects:

This behaviour causes memory leaks.

The same behaviour is also on the client side: the ClientRaftProxyFactory#lockProxies tracks all created FencedLockProxy and not remove from the map.

The same client-side _lock_proxies map can be found in the Python client https://github.com/hazelcast/hazelcast-python-client/blob/master/hazelcast/cp.py#L198

[gbarnett-hz]

From code we only tidy LockService.proxies up under the following scenarios:

1. CP system restart, LockService#onCPSubsystemRestart(); and
2. When a proxy with the same name already exists but is associated with an unexpected CP group ID

Doesn't look like proxies mapping is critical to other semantics.

If the current semantics of not permitting the same object to be reused post-destruction are to be preserved and we don't want the removal from proxies to deviate from current semantics, then @arodionov is it possible the safest (read: most conservative) place to perform the removal would be after we add the object name to ResourceRegistry.destroyedNames? Looking at the code I think this could be done earlier but not sure if that will have some unknown side effect.

As a sketch:

class LockService ... {
    // ...
    @Override
    public boolean destroyRaftObject(CPGroupId groupId, String name) {
        boolean result = super.destroyRaftObject(groupId, name);
        FencedLockProxy proxy = proxies.get(name);
        proxies.remove(name, proxy);
        return result;
    }
}

[github-actions]

Internal Jira issue: HZ-3039