Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamically Adding Configuration #12222

Closed
jevel0per opened this issue Jan 30, 2018 · 10 comments
Closed

Dynamically Adding Configuration #12222

jevel0per opened this issue Jan 30, 2018 · 10 comments

Comments

@jevel0per
Copy link

@jevel0per jevel0per commented Jan 30, 2018

Hi guys. I'm using hazelcast 3.9.2 and one of new features as 'Dynamically Adding Configuration'. My application has a module structure: there are several modules and core. Instance of the hazelcast is created by core, others modules are able to add some maps dinamically after that. Also i use spring 5.0.

here is sample of creating a map in a module:

    @Bean
    public ReplicatedMap<String, Compilation> compilations(HazelcastInstance hazelcastInstance) {
        ReplicatedMapConfig config = new ReplicatedMapConfig(COMPILATION_MAP);
        config.setInMemoryFormat(InMemoryFormat.OBJECT);
        config.setAsyncFillup(false);
        hazelcastInstance.getConfig().addReplicatedMapConfig(config);
        return hazelcastInstance.getReplicatedMap(COMPILATION_MAP);
    }

when i only start one node, all works perfectly, but when second node is started i get an exception on him:

Failed to instantiate [com.hazelcast.core.ReplicatedMap]: Factory method 'compilations' threw exception; nested exception is com.hazelcast.config.ConfigurationException: Cannot add a dynamic configuration 'ReplicatedMapConfig{name='compilation.map'', inMemoryFormat=OBJECT', concurrencyLevel=32, replicationDelayMillis=100, asyncFillup=false, statisticsEnabled=true, mergePolicy='com.hazelcast.replicatedmap.merge.PutIfAbsentMapMergePolicy'}' as there is already a conflicting configuration 'ReplicatedMapConfig{name='compilation.map'', inMemoryFormat=OBJECT', concurrencyLevel=32, replicationDelayMillis=100, asyncFillup=false, statisticsEnabled=true, mergePolicy='com.hazelcast.replicatedmap.merge.PutIfAbsentMapMergePolicy'}'

But both configs are equals into text of exception and you can see that.

Is it a bug or i do something wrong?

@vbekiaris
Copy link
Contributor

@vbekiaris vbekiaris commented Jan 30, 2018

@jevel0per thanks the report. A dynamically added config should not cause a ConfigurationException to be thrown as long as it is equal to the existing one. For example, my attempt at reproducing the issue (start 2 Hazelcast instances and add the config to each one of them like I guess happens with your spring app), works as expected (no exception thrown):

public static void main(String[] args) throws FileNotFoundException, UnsupportedEncodingException {
        HazelcastInstance hz = Hazelcast.newHazelcastInstance();
        hz.getConfig().addReplicatedMapConfig(getReplicatedMapConfig());
        HazelcastInstance hz2 = Hazelcast.newHazelcastInstance();
        hz2.getConfig().addReplicatedMapConfig(getReplicatedMapConfig());
}

private static ReplicatedMapConfig getReplicatedMapConfig() {
        ReplicatedMapConfig config = new ReplicatedMapConfig("compilations");
        config.setInMemoryFormat(InMemoryFormat.OBJECT);
        config.setAsyncFillup(false);
        return config;
    }

Could you share a minimal reproducer?

As a side-note, with dynamic config, you only need to add a dynamic config once in the cluster. New cluster members become aware of the dynamic config's as they join the cluster.

@jevel0per
Copy link
Author

@jevel0per jevel0per commented Jan 30, 2018

@vbekiaris thanks for answer. yes, i was able to reproduce the exception.
i've choosen another way, i created 2 simple spring-boot-app with equals hazelcast config and when i started second node(just copy jar and run) i got the same error.
How could i share the reprocuder?

As a side-note, with dynamic config, you only need to add a dynamic config once in the cluster.

Do you mean i should add IF operation like this:

 @Bean
    public ReplicatedMap<String, Compilation> compilations(HazelcastInstance hazelcastInstance) {
        //check i am first
        if (hazelcastInstance.getCluster().getMembers().size() == 1) {
              ReplicatedMapConfig config = new ReplicatedMapConfig(COMPILATION_MAP);
              config.setInMemoryFormat(InMemoryFormat.OBJECT);
              config.setAsyncFillup(false);
             hazelcastInstance.getConfig().addReplicatedMapConfig(config);
        }
        return hazelcastInstance.getReplicatedMap(COMPILATION_MAP);
    }

because i have the same code on each node.
BTW i'm using hazelcast in embedded mode

@vbekiaris
Copy link
Contributor

@vbekiaris vbekiaris commented Jan 30, 2018

How could i share the reprocuder?

Can you share the spring boot app on github? If you prefer, zip it and email it to my first name [at] hazelcast.com .

Do you mean i should add IF operation like this:

I meant it's not necessary to add the dynamic config on each joining member, however it's not a problem either (or shouldn't be a problem, unless your reproducer proves otherwise ;-) ).

@jevel0per
Copy link
Author

@jevel0per jevel0per commented Jan 30, 2018

i've added the reproducer on github: https://github.com/jevel0per/hazelcast_test.git
i do next: making a jar through maven install and start one jar, then second jar

@jevel0per
Copy link
Author

@jevel0per jevel0per commented Jan 31, 2018

@vbekiaris is there any news?

@mmedenjak mmedenjak added this to the 3.10 milestone Jan 31, 2018
@vbekiaris
Copy link
Contributor

@vbekiaris vbekiaris commented Jan 31, 2018

@jevel0per thanks for the reproducer, here's the thing: while creating the ReplicatedMap proxy, a read-only copy of the config is made. However this results in mutating the listenerConfigs field of the original ReplicatedMapConfig because getListenerConfigs() has get-or-create semantics. As a result, when the second member starts and the same configuration is added again, on the first member equality check fails because the existing config's listenerConfigs is an empty list while incoming one is null.

When dynamic config was introduced, a config-check kill switch was also added: starting your JVM with -Dhazelcast.dynamicconfig.ignore.conflicts=true will work around the issue and pretend there is never a conflict while adding dynamic configs (and also hide real conflict issues!). This will workaround the issue until we properly fix this; we just entered code freeze for 3.9.3 today so most probably the fix should go into 3.9.4.

Once again, many thanks for an excellent issue report & reproducer.

@vbekiaris
Copy link
Contributor

@vbekiaris vbekiaris commented Mar 16, 2018

Fixed by #12591

@jevel0per if you are keen to test your use case, a new 3.9.4-SNAPSHOT should be published shortly in the snapshots repository including the fix (see below for the repository config). Thanks again for reporting this issue!

<repository>
    <id>sonatype-snapshots</id>
    <name>Sonatype Snapshot Repository</name>
    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    <releases>
        <enabled>false</enabled>
    </releases>
    <snapshots>
        <enabled>true</enabled>
    </snapshots>
</repository>
@vbekiaris vbekiaris closed this Mar 16, 2018
@benze
Copy link

@benze benze commented Aug 2, 2019

I am running into a similar issue today with HC 3.12. Is this a regression bug that has reappeared? I've tried stepping through the logic, but am having difficulties seeing how/where/why this is failing. I am unable to attach the debugger via IntelliJ in order to catch the offending portion of the SimpleCacheConfig.equals() method to determine why it is failing.

My cache is created as follows:

    @Produces
    @ApplicationScoped
    @UserRolesCache
    public Cache<String, String[]> createUserRoleCache(HazelcastInstance hazelcastInstance, @UserRolesCache Factory<? extends CacheEntryListener<? super String, ? super String[]>> entryListenerFactory) {
        int ssoSessionTimeoutInSeconds = 15;

        LOG.debug("Creating cache with expiration set to {} minutes", ssoSessionTimeoutInSeconds / 60);
        // Add cache used by adams
            CacheSimpleConfig cacheSimpleConfig = new CacheSimpleConfig()
                    .setName(CACHE_NAME)
                    .setKeyType(String.class.getName())
                    .setValueType((new String[0]).getClass().getName())
                    .setStatisticsEnabled(false)
                    .setManagementEnabled(false)
                    .setReadThrough(true)
                    .setWriteThrough(true)
                    .setInMemoryFormat(InMemoryFormat.OBJECT)
                    .setBackupCount(1)
                    .setAsyncBackupCount(1)
                    .setEvictionConfig(new EvictionConfig()
                            .setEvictionPolicy(EvictionPolicy.LRU)
                            .setSize(1000)
                            .setMaximumSizePolicy(EvictionConfig.MaxSizePolicy.ENTRY_COUNT))
                    .setExpiryPolicyFactoryConfig(
                            new ExpiryPolicyFactoryConfig(
                                    new TimedExpiryPolicyFactoryConfig(ACCESSED,
                                            new DurationConfig(
                                                    ssoSessionTimeoutInSeconds,
                                                    TimeUnit.SECONDS))));
            hazelcastInstance.getConfig().addCacheConfig(cacheSimpleConfig);

        ICache<String, String[]> cache = hazelcastInstance.getCacheManager().getCache(CACHE_NAME);
        cache.registerCacheEntryListener(new MutableCacheEntryListenerConfiguration<>(entryListenerFactory, null, false, false));
        return cache;
    }

Can the issue be due to the CacheEntryListener being a JSR107 listener instead of an HC implementation?

@vbekiaris
Copy link
Contributor

@vbekiaris vbekiaris commented Aug 2, 2019

@benze thanks for the cache configuration code, can you also post the stack trace of the exception?

@benze
Copy link

@benze benze commented Aug 2, 2019

@vbekiaris This is the stack trace I'm encountering:

SEVERE: [127.0.0.1]:5701 [hazelcast-poc-cluster] [3.12] Cannot add a dynamic configuration 'CacheSimpleConfig{name='UserRoles', asyncBackupCount=1, backupCount=1, inMemoryFormat=OBJECT, keyType=java.lang.String, valueType=[Ljava.lang.String;, statisticsEnabled=false, managementEnabled=false, readThrough=true, writeThrough=true, cacheLoaderFactory='null', cacheWriterFactory='null', cacheLoader='null', cacheWriter='null', expiryPolicyFactoryConfig=ExpiryPolicyFactoryConfig{className='null', timedExpiryPolicyFactoryConfig=TimedExpiryPolicyFactoryConfig{expiryPolicyType=ACCESSED, durationConfig=DurationConfig{durationAmount=15, timeUnitSECONDS}}}, cacheEntryListeners=null, evictionConfig=EvictionConfig{size=1000, maxSizePolicy=ENTRY_COUNT, evictionPolicy=LRU, comparatorClassName=null, comparator=null}, wanReplicationRef=null, quorumName=null, partitionLostListenerConfigs=null, mergePolicy=com.hazelcast.spi.merge.PutIfAbsentMergePolicy, hotRestartConfig=HotRestartConfig{enabled=false, fsync=false}}' as there is already a conflicting configuration 'CacheSimpleConfig{name='UserRoles', asyncBackupCount=1, backupCount=1, inMemoryFormat=OBJECT, keyType=java.lang.String, valueType=[Ljava.lang.String;, statisticsEnabled=false, managementEnabled=false, readThrough=true, writeThrough=true, cacheLoaderFactory='null', cacheWriterFactory='null', cacheLoader='null', cacheWriter='null', expiryPolicyFactoryConfig=ExpiryPolicyFactoryConfig{className='null', timedExpiryPolicyFactoryConfig=TimedExpiryPolicyFactoryConfig{expiryPolicyType=ACCESSED, durationConfig=DurationConfig{durationAmount=15, timeUnitSECONDS}}}, cacheEntryListeners=null, evictionConfig=EvictionConfig{size=1000, maxSizePolicy=ENTRY_COUNT, evictionPolicy=LRU, comparatorClassName=null, comparator=null}, wanReplicationRef=null, quorumName=null, partitionLostListenerConfigs=[], mergePolicy=com.hazelcast.spi.merge.PutIfAbsentMergePolicy, hotRestartConfig=HotRestartConfig{enabled=false, fsync=false}}'
com.hazelcast.config.ConfigurationException: Cannot add a dynamic configuration 'CacheSimpleConfig{name='UserRoles', asyncBackupCount=1, backupCount=1, inMemoryFormat=OBJECT, keyType=java.lang.String, valueType=[Ljava.lang.String;, statisticsEnabled=false, managementEnabled=false, readThrough=true, writeThrough=true, cacheLoaderFactory='null', cacheWriterFactory='null', cacheLoader='null', cacheWriter='null', expiryPolicyFactoryConfig=ExpiryPolicyFactoryConfig{className='null', timedExpiryPolicyFactoryConfig=TimedExpiryPolicyFactoryConfig{expiryPolicyType=ACCESSED, durationConfig=DurationConfig{durationAmount=15, timeUnitSECONDS}}}, cacheEntryListeners=null, evictionConfig=EvictionConfig{size=1000, maxSizePolicy=ENTRY_COUNT, evictionPolicy=LRU, comparatorClassName=null, comparator=null}, wanReplicationRef=null, quorumName=null, partitionLostListenerConfigs=null, mergePolicy=com.hazelcast.spi.merge.PutIfAbsentMergePolicy, hotRestartConfig=HotRestartConfig{enabled=false, fsync=false}}' as there is already a conflicting configuration 'CacheSimpleConfig{name='UserRoles', asyncBackupCount=1, backupCount=1, inMemoryFormat=OBJECT, keyType=java.lang.String, valueType=[Ljava.lang.String;, statisticsEnabled=false, managementEnabled=false, readThrough=true, writeThrough=true, cacheLoaderFactory='null', cacheWriterFactory='null', cacheLoader='null', cacheWriter='null', expiryPolicyFactoryConfig=ExpiryPolicyFactoryConfig{className='null', timedExpiryPolicyFactoryConfig=TimedExpiryPolicyFactoryConfig{expiryPolicyType=ACCESSED, durationConfig=DurationConfig{durationAmount=15, timeUnitSECONDS}}}, cacheEntryListeners=null, evictionConfig=EvictionConfig{size=1000, maxSizePolicy=ENTRY_COUNT, evictionPolicy=LRU, comparatorClassName=null, comparator=null}, wanReplicationRef=null, quorumName=null, partitionLostListenerConfigs=[], mergePolicy=com.hazelcast.spi.merge.PutIfAbsentMergePolicy, hotRestartConfig=HotRestartConfig{enabled=false, fsync=false}}'
	at com.hazelcast.internal.dynamicconfig.ClusterWideConfigurationService.checkCurrentConfigNullOrEqual(ClusterWideConfigurationService.java:410)
	at com.hazelcast.internal.dynamicconfig.ClusterWideConfigurationService.registerConfigLocally(ClusterWideConfigurationService.java:368)
	at com.hazelcast.internal.dynamicconfig.AddDynamicConfigOperation.run(AddDynamicConfigOperation.java:48)
	at com.hazelcast.spi.Operation.call(Operation.java:170)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:210)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:199)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:416)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:153)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:123)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:110)

I've pushed my simple POC code to https://github.com/benze/hazelcast-poc
The startup class is CDIApp.main().

It is a POC to see how I can use CDI to setup/configure HC and use JSR-107 annotations/etc.
If I run multiple instances of CDIApp, the second fails.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.