Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IllegalThreadStateException when creating Map with configured indexes #8492

Closed
vbekiaris opened this issue Jul 4, 2016 · 8 comments

Comments

Projects
None yet
3 participants
@vbekiaris
Copy link
Contributor

commented Jul 4, 2016

An IllegalThreadStateException may be thrown sometimes on a member of a cluster with 2 or more members when a hazelcast client creates a new map with configured indexes. This exception is not thrown when indexes are created programmatically.
Thanks @noctarius for reporting this with the following reproducer:

import com.hazelcast.client.HazelcastClient;
import com.hazelcast.config.Config;
import com.hazelcast.config.MapConfig;
import com.hazelcast.config.MapIndexConfig;
import com.hazelcast.config.XmlConfigBuilder;
import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IMap;
import java.util.List;

public class Test {
    public static void main(String[] args) {
        Config config = new XmlConfigBuilder().build();

        MapConfig mapConfig = config.getMapConfig("test");
        List<MapIndexConfig> mapIndexConfigs = mapConfig.getMapIndexConfigs();

        MapIndexConfig mapIndexConfig = new MapIndexConfig();
        mapIndexConfig.setAttribute("foo");
        mapIndexConfig.setOrdered(true);
        mapIndexConfigs.add(mapIndexConfig);

        Hazelcast.newHazelcastInstance(config);
        Hazelcast.newHazelcastInstance(config);
        HazelcastInstance hazelcastInstance = HazelcastClient.newHazelcastClient();
        IMap<String, String> test = hazelcastInstance.getMap("test");

        test.put("foo", "bar");
    }
}

@vbekiaris vbekiaris added this to the 3.7 milestone Jul 4, 2016

@vbekiaris vbekiaris self-assigned this Jul 4, 2016

@vbekiaris

This comment has been minimized.

Copy link
Contributor Author

commented Jul 4, 2016

This issue affects 3.6[.x] versions as well as current master branch. The series of events that lead to manifestation of the problem are:

  1. client sends a ClientCreateProxy request to member A, with invocation target member B
  2. member A on its client thread sees it’s a remote operation and invokes remotely on member B
  3. member B deserializes and executes InitializeDistributedObjectOperation in generic operation thread. Subsequently IllegalThreadStateException is thrown because the generic operation thread is not allowed to invokeOnAllPartitions the AddIndexOperation (invoked from MapProxySupport.initialize()).
@vbekiaris

This comment has been minimized.

Copy link
Contributor Author

commented Jul 6, 2016

Sample stack trace of IllegalThreadStateException thrown:

WARNING: [192.168.2.4]:5701 [dev] [3.7-SNAPSHOT] Error while initializing proxy: IMap{name='test'}
java.lang.IllegalThreadStateException: Thread[hz._hzInstance_1_dev.generic-operation.thread-1,5,_hzInstance_1_dev] cannot make invocation on multiple partitions!
    at com.hazelcast.spi.impl.operationservice.impl.InvokeOnPartitions.ensureNotCallingFromOperationThread(InvokeOnPartitions.java:74)
    at com.hazelcast.spi.impl.operationservice.impl.InvokeOnPartitions.invoke(InvokeOnPartitions.java:61)
    at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.invokeOnAllPartitions(OperationServiceImpl.java:360)
    at com.hazelcast.map.impl.proxy.MapProxySupport.addIndex(MapProxySupport.java:1109)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.addIndex(MapProxyImpl.java:82)
    at com.hazelcast.map.impl.proxy.MapProxySupport.initializeIndexes(MapProxySupport.java:238)
    at com.hazelcast.map.impl.proxy.MapProxySupport.initialize(MapProxySupport.java:221)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.initialize(MapProxyImpl.java:82)
    at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.doCreateProxy(ProxyRegistry.java:180)
    at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.createProxy(ProxyRegistry.java:170)
    at com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.initializeDistributedObject(ProxyServiceImpl.java:124)
    at com.hazelcast.spi.impl.proxyservice.impl.operations.InitializeDistributedObjectOperation.run(InitializeDistributedObjectOperation.java:45)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:182)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:396)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:117)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:102)

Interestingly, indexes are eventually created despite the original invocation target node having failed:

  • other nodes in the cluster are notified with DistributedObjectEvent of type CREATED
  • nodes create the map proxy on their event thread; this thread is allowed to invokeOnAllPartitions
  • AddIndexOperations are delivered to the original invocation target node and executed on its partition threads, creating the index properly
@vbekiaris

This comment has been minimized.

Copy link
Contributor Author

commented Jul 11, 2016

Making PartitionIteratingOperations non-blocking will be addressed in #4889. Apart from that, it seems that we are doing more work than required here, as AddIndexOperations are triggered both from the original map proxy creation operation as well as during processing of the DistributedObject created event on other members of the cluster. Need to investigate if the extra work can be avoided or not.
Since this does not currently cause an issue with correctness or performance, removing the Critical label and slipping to 3.8.

@vbekiaris vbekiaris modified the milestones: 3.8, 3.7 Jul 11, 2016

@engrun

This comment has been minimized.

Copy link

commented Oct 27, 2016

We have run into this bug and when it happens the cluster does not perform, and seems to be in some kind of freeze state. The result is that we now are not confident when deploying (automatically) to production, as we cannot be sure wether the cluster actually starts correctly.

@vbekiaris

This comment has been minimized.

Copy link
Contributor Author

commented Oct 27, 2016

Sorry to hear this affects you @engrun , while attempting to reproduce this issue I have not seen the cluster being unresponsive (I guess that's what you mean by "freeze state", not literally in FROZEN cluster status, right?). Apart from the IllegalThreadStateException mentioned above, do you see any other suspicious log entries when this occurs? Could you share some of your logs at the time when this happens?

@engrun

This comment has been minimized.

Copy link

commented Oct 27, 2016

I'll try to get a thread-dump the next time we experience this. And have a closer look in the logs, but as I can recall, nothing else "suspicious" is being reported. And by "freeze" I mean unresponsive, that is correct. We have also looked at this issue #4889 , I am not sure if they somehow are related?

@engrun

This comment has been minimized.

Copy link

commented Oct 27, 2016

Sorry, I was a bit too quick to report. We have identified some issues in our own code that may cause the problem which is more likely related to this issue #4889

@vbekiaris

This comment has been minimized.

Copy link
Contributor Author

commented Oct 31, 2016

Thanks for the heads up @engrun ! Since #4889 is already fixed in master, you could test with 3.8-SNAPSHOT artifact from snapshots maven repository if your issue is resolved.
Cheers!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.