Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

com.hazelcast.core.OperationTimeoutException: [CONCURRENT_MAP_SET] Redo threshold[90] exceeded! Last redo cause: REDO_MAP_OVER_CAPACITY #1685

Closed
patriziomunzi opened this issue Jan 31, 2014 · 6 comments

Comments

Projects
None yet
3 participants
@patriziomunzi
Copy link

commented Jan 31, 2014

Hazelcast 2.6.5.
In a map with max-size 10000. I see a lot of warning like:

WARNING: [127.0.0.1]:5703 [htgamingdatagrid] Caller -> RedoLog{name=c:com.hypertv.gaming.model.HTUserNotificationBean, redoType=REDO_MAP_OVER_CAPACITY, operation=CONCURRENT_MAP_SET, target=Address[127.0.0.1]:5703 / connected=false, redoCount=91, migrating=null
partition=Partition [170]{
        0:Address[127.0.0.1]:5703
}

and then the following exception

Exception in thread "pool-5-thread-172" com.hazelcast.core.OperationTimeoutException: [CONCURRENT_MAP_SET] Redo threshold[90] exceeded! Last redo cause: REDO_MAP_OVER_CAPACITY, Name: c:com.hypertv.gaming.model.HTUserNotificationBean
        at com.hazelcast.impl.BaseManager$ResponseQueueCall.getRedoAwareResult(BaseManager.java:639)
        at com.hazelcast.impl.BaseManager$ResponseQueueCall.getResult(BaseManager.java:626)
        at com.hazelcast.impl.BaseManager$RequestBasedCall.getResultAsBoolean(BaseManager.java:440)
        at com.hazelcast.impl.BaseManager$ResponseQueueCall.getResultAsBoolean(BaseManager.java:547)
        at com.hazelcast.impl.ConcurrentMapManager$MPut.txnalPut(ConcurrentMapManager.java:1894)
        at com.hazelcast.impl.ConcurrentMapManager$MPut.txnalPut(ConcurrentMapManager.java:1837)
        at com.hazelcast.impl.ConcurrentMapManager$MPut.set(ConcurrentMapManager.java:1743)
        at com.hazelcast.impl.MProxyImpl$MProxyReal.set(MProxyImpl.java:659)
        at com.hazelcast.impl.MProxyImpl$MProxyReal.set(MProxyImpl.java:645)
        at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.hazelcast.impl.MProxyImpl$DynamicInvoker.invoke(MProxyImpl.java:66)
        at com.sun.proxy.$Proxy9.set(Unknown Source)
        at com.hazelcast.impl.MProxyImpl.set(MProxyImpl.java:246)
        at com.hazelcast.cache.HazelcastCacheWrapper.put(HazelcastCacheWrapper.java:160)
        at ...
@enesakar

This comment has been minimized.

Copy link
Member

commented Feb 1, 2014

Have decided defined an eviction policy?
On Jan 31, 2014 7:37 PM, "patriziomunzi" notifications@github.com wrote:

In a map with max-size 10000. I see a lot of warning like:

WARNING: [127.0.0.1]:5703 [htgamingdatagrid] Caller -> RedoLog{name=c:com.hypertv.gaming.model.HTUserNotificationBean, redoType=REDO_MAP_OVER_CAPACITY, operation=CONCURRENT_MAP_SET, target=Address[127.0.0.1]:5703 / connected=false, redoCount=91, migrating=null
partition=Partition [170]{
0:Address[127.0.0.1]:5703
}

and then the following exception

Exception in thread "pool-5-thread-172" com.hazelcast.core.OperationTimeoutException: [CONCURRENT_MAP_SET] Redo threshold[90] exceeded! Last redo cause: REDO_MAP_OVER_CAPACITY, Name: c:com.hypertv.gaming.model.HTUserNotificationBean
at com.hazelcast.impl.BaseManager$ResponseQueueCall.getRedoAwareResult(BaseManager.java:639)
at com.hazelcast.impl.BaseManager$ResponseQueueCall.getResult(BaseManager.java:626)
at com.hazelcast.impl.BaseManager$RequestBasedCall.getResultAsBoolean(BaseManager.java:440)
at com.hazelcast.impl.BaseManager$ResponseQueueCall.getResultAsBoolean(BaseManager.java:547)
at com.hazelcast.impl.ConcurrentMapManager$MPut.txnalPut(ConcurrentMapManager.java:1894)
at com.hazelcast.impl.ConcurrentMapManager$MPut.txnalPut(ConcurrentMapManager.java:1837)
at com.hazelcast.impl.ConcurrentMapManager$MPut.set(ConcurrentMapManager.java:1743)
at com.hazelcast.impl.MProxyImpl$MProxyReal.set(MProxyImpl.java:659)
at com.hazelcast.impl.MProxyImpl$MProxyReal.set(MProxyImpl.java:645)
at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.hazelcast.impl.MProxyImpl$DynamicInvoker.invoke(MProxyImpl.java:66)
at com.sun.proxy.$Proxy9.set(Unknown Source)
at com.hazelcast.impl.MProxyImpl.set(MProxyImpl.java:246)
at com.hazelcast.cache.HazelcastCacheWrapper.put(HazelcastCacheWrapper.java:160)
at ...

Reply to this email directly or view it on GitHubhttps://github.com//issues/1685
.

@patriziomunzi

This comment has been minimized.

Copy link
Author

commented Feb 1, 2014

Nope, I haven't set any eviction policy.
Shouldn't it default to something?

iPAT@iphone

On 01/feb/2014, at 12:18, Enes Akar notifications@github.com wrote:

Have decided defined an eviction policy?
On Jan 31, 2014 7:37 PM, "patriziomunzi" notifications@github.com wrote:

In a map with max-size 10000. I see a lot of warning like:

WARNING: [127.0.0.1]:5703 [htgamingdatagrid] Caller -> RedoLog{name=c:com.hypertv.gaming.model.HTUserNotificationBean, redoType=REDO_MAP_OVER_CAPACITY, operation=CONCURRENT_MAP_SET, target=Address[127.0.0.1]:5703 / connected=false, redoCount=91, migrating=null
partition=Partition [170]{
0:Address[127.0.0.1]:5703
}

and then the following exception

Exception in thread "pool-5-thread-172" com.hazelcast.core.OperationTimeoutException: [CONCURRENT_MAP_SET] Redo threshold[90] exceeded! Last redo cause: REDO_MAP_OVER_CAPACITY, Name: c:com.hypertv.gaming.model.HTUserNotificationBean
at com.hazelcast.impl.BaseManager$ResponseQueueCall.getRedoAwareResult(BaseManager.java:639)
at com.hazelcast.impl.BaseManager$ResponseQueueCall.getResult(BaseManager.java:626)
at com.hazelcast.impl.BaseManager$RequestBasedCall.getResultAsBoolean(BaseManager.java:440)
at com.hazelcast.impl.BaseManager$ResponseQueueCall.getResultAsBoolean(BaseManager.java:547)
at com.hazelcast.impl.ConcurrentMapManager$MPut.txnalPut(ConcurrentMapManager.java:1894)
at com.hazelcast.impl.ConcurrentMapManager$MPut.txnalPut(ConcurrentMapManager.java:1837)
at com.hazelcast.impl.ConcurrentMapManager$MPut.set(ConcurrentMapManager.java:1743)
at com.hazelcast.impl.MProxyImpl$MProxyReal.set(MProxyImpl.java:659)
at com.hazelcast.impl.MProxyImpl$MProxyReal.set(MProxyImpl.java:645)
at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.hazelcast.impl.MProxyImpl$DynamicInvoker.invoke(MProxyImpl.java:66)
at com.sun.proxy.$Proxy9.set(Unknown Source)
at com.hazelcast.impl.MProxyImpl.set(MProxyImpl.java:246)
at com.hazelcast.cache.HazelcastCacheWrapper.put(HazelcastCacheWrapper.java:160)
at ...

Reply to this email directly or view it on GitHubhttps://github.com//issues/1685
.


Reply to this email directly or view it on GitHub.

@enesakar

This comment has been minimized.

Copy link
Member

commented Feb 1, 2014

Default is NONE. Set it to either LRU or LFU to make eviction effective.

On Sat, Feb 1, 2014 at 2:18 PM, patriziomunzi notifications@github.comwrote:

Nope, I haven't set any eviction policy.
Shouldn't it default to something?

iPAT@iphone

On 01/feb/2014, at 12:18, Enes Akar notifications@github.com wrote:

Have decided defined an eviction policy?
On Jan 31, 2014 7:37 PM, "patriziomunzi" notifications@github.com
wrote:

In a map with max-size 10000. I see a lot of warning like:

WARNING: [127.0.0.1]:5703 [htgamingdatagrid] Caller ->
RedoLog{name=c:com.hypertv.gaming.model.HTUserNotificationBean,
redoType=REDO_MAP_OVER_CAPACITY, operation=CONCURRENT_MAP_SET,
target=Address[127.0.0.1]:5703 / connected=false, redoCount=91,
migrating=null
partition=Partition [170]{
0:Address[127.0.0.1]:5703
}

and then the following exception

Exception in thread "pool-5-thread-172"
com.hazelcast.core.OperationTimeoutException: [CONCURRENT_MAP_SET] Redo
threshold[90] exceeded! Last redo cause: REDO_MAP_OVER_CAPACITY, Name:
c:com.hypertv.gaming.model.HTUserNotificationBean
at
com.hazelcast.impl.BaseManager$ResponseQueueCall.getRedoAwareResult(BaseManager.java:639)

at
com.hazelcast.impl.BaseManager$ResponseQueueCall.getResult(BaseManager.java:626)

at
com.hazelcast.impl.BaseManager$RequestBasedCall.getResultAsBoolean(BaseManager.java:440)

at
com.hazelcast.impl.BaseManager$ResponseQueueCall.getResultAsBoolean(BaseManager.java:547)

at
com.hazelcast.impl.ConcurrentMapManager$MPut.txnalPut(ConcurrentMapManager.java:1894)

at
com.hazelcast.impl.ConcurrentMapManager$MPut.txnalPut(ConcurrentMapManager.java:1837)

at
com.hazelcast.impl.ConcurrentMapManager$MPut.set(ConcurrentMapManager.java:1743)

at com.hazelcast.impl.MProxyImpl$MProxyReal.set(MProxyImpl.java:659)
at com.hazelcast.impl.MProxyImpl$MProxyReal.set(MProxyImpl.java:645)
at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)
at
com.hazelcast.impl.MProxyImpl$DynamicInvoker.invoke(MProxyImpl.java:66)
at com.sun.proxy.$Proxy9.set(Unknown Source)
at com.hazelcast.impl.MProxyImpl.set(MProxyImpl.java:246)
at
com.hazelcast.cache.HazelcastCacheWrapper.put(HazelcastCacheWrapper.java:160)

at ...

Reply to this email directly or view it on GitHub<
https://github.com/hazelcast/hazelcast/issues/1685>
.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1685#issuecomment-33870085
.

Enes Akar
Hazelcast | Open source in-memory data grid
Mobile: +90.507.150.56.71

@patriziomunzi

This comment has been minimized.

Copy link
Author

commented Feb 2, 2014

I've set the eviction policy to LRU
<hz:map name="com.hypertv.datagrid.test.model.ValueObject" max-size="10" eviction-policy="LRU"/>
but I've still got the same warnings. They happen in two situations:

  1. when I put the objects in the map
  2. at the startup if I load all the objects from the DB.

Patrizio Munzi
email: patrizio.munzi@gmail.com
mobile: +39 393 7195164
skype: patrizio.munzi

On 01 Feb 2014, at 22:51, Enes Akar notifications@github.com wrote:

Default is NONE. Set it to either LRU or LFU to make eviction effective.

On Sat, Feb 1, 2014 at 2:18 PM, patriziomunzi notifications@github.comwrote:

Nope, I haven't set any eviction policy.
Shouldn't it default to something?

iPAT@iphone

On 01/feb/2014, at 12:18, Enes Akar notifications@github.com wrote:

Have decided defined an eviction policy?
On Jan 31, 2014 7:37 PM, "patriziomunzi" notifications@github.com
wrote:

In a map with max-size 10000. I see a lot of warning like:

WARNING: [127.0.0.1]:5703 [htgamingdatagrid] Caller ->
RedoLog{name=c:com.hypertv.gaming.model.HTUserNotificationBean,
redoType=REDO_MAP_OVER_CAPACITY, operation=CONCURRENT_MAP_SET,
target=Address[127.0.0.1]:5703 / connected=false, redoCount=91,
migrating=null
partition=Partition [170]{
0:Address[127.0.0.1]:5703
}

and then the following exception

Exception in thread "pool-5-thread-172"
com.hazelcast.core.OperationTimeoutException: [CONCURRENT_MAP_SET] Redo
threshold[90] exceeded! Last redo cause: REDO_MAP_OVER_CAPACITY, Name:
c:com.hypertv.gaming.model.HTUserNotificationBean
at
com.hazelcast.impl.BaseManager$ResponseQueueCall.getRedoAwareResult(BaseManager.java:639)

at
com.hazelcast.impl.BaseManager$ResponseQueueCall.getResult(BaseManager.java:626)

at
com.hazelcast.impl.BaseManager$RequestBasedCall.getResultAsBoolean(BaseManager.java:440)

at
com.hazelcast.impl.BaseManager$ResponseQueueCall.getResultAsBoolean(BaseManager.java:547)

at
com.hazelcast.impl.ConcurrentMapManager$MPut.txnalPut(ConcurrentMapManager.java:1894)

at
com.hazelcast.impl.ConcurrentMapManager$MPut.txnalPut(ConcurrentMapManager.java:1837)

at
com.hazelcast.impl.ConcurrentMapManager$MPut.set(ConcurrentMapManager.java:1743)

at com.hazelcast.impl.MProxyImpl$MProxyReal.set(MProxyImpl.java:659)
at com.hazelcast.impl.MProxyImpl$MProxyReal.set(MProxyImpl.java:645)
at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)
at
com.hazelcast.impl.MProxyImpl$DynamicInvoker.invoke(MProxyImpl.java:66)
at com.sun.proxy.$Proxy9.set(Unknown Source)
at com.hazelcast.impl.MProxyImpl.set(MProxyImpl.java:246)
at
com.hazelcast.cache.HazelcastCacheWrapper.put(HazelcastCacheWrapper.java:160)

at ...

Reply to this email directly or view it on GitHub<
https://github.com/hazelcast/hazelcast/issues/1685>
.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1685#issuecomment-33870085
.

Enes Akar
Hazelcast | Open source in-memory data grid
Mobile: +90.507.150.56.71

Reply to this email directly or view it on GitHub.

@patriziomunzi

This comment has been minimized.

Copy link
Author

commented Feb 12, 2014

as an additional info I've verified that this happens only when the objects are put into the map, not when the objects are loaded from the DB.

This test passes:

    @Test
    public void testMaxSizeOnlyGet() throws InterruptedException {
        for(int i = 1; i <= 2000; i++) {
            ValueObjectKey key = new ValueObjectKey(i * 1L, "name100");
            ValueObject v = ValueObjectCacheUtil.getById(key);
            Assert.assertNotNull(v);
        }
        [...]
    }

The following throw exceptions

    @Test
    public void testMaxSize() throws InterruptedException {
        for(int i = 1; i < 2000; i++) {
            ValueObjectKey key = new ValueObjectKey(i * 1L, "name100");
            ValueObject obj1 = new ValueObject(key);
            ValueObjectCacheUtil.put(obj1);         
        }
        [...]
    }

@enesakar enesakar added this to the 2.6.8 milestone Mar 5, 2014

@mesutcelik

This comment has been minimized.

Copy link
Contributor

commented Mar 18, 2014

Hi @patriziomunzi ,

Following Test is passing on maintenance-2.x branch.
https://github.com/hazelcast/hazelcast/blob/maintenance-2.x/hazelcast/src/test/java/com/hazelcast/impl/ClusterTest.java#L1020

Can you please try with 2.6.8-SNAPSHOT?

@mesutcelik mesutcelik closed this Mar 18, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.