Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Excessive logs on target side when `cache` config missing for WAN replication #12826

Closed
lazerion opened this issue Apr 11, 2018 · 1 comment
Closed

Comments

@lazerion
Copy link
Contributor

@lazerion lazerion commented Apr 11, 2018

For instance, when replication A [active] -> B [passive] if target B cluster is missing similar config for cache replication, within 3 mins of run time ~150MB log is generated by target cluster, it also affects other on going replication for instance Map to fail or linger at least given period of time for verification (under 1 min), as observed from the tests.
Logs repeat below pattern during execution:

...
2018-04-11T08:54:58.702805328Z Apr 11, 2018 8:54:58 AM com.hazelcast.enterprise.wan.EnterpriseWanReplicationService
2018-04-11T08:54:58.702833776Z SEVERE: [hazelcast-C-2]:5701 [dc-3] [3.8.6] null
2018-04-11T08:54:58.702840300Z java.lang.NullPointerException
2018-04-11T08:54:58.702844553Z 	at com.hazelcast.cache.impl.AbstractCacheService.putCacheConfigIfAbsent(AbstractCacheService.java:323)
2018-04-11T08:54:58.702848678Z 	at com.hazelcast.cache.EnterpriseCacheService.putCacheConfigIfAbsent(EnterpriseCacheService.java:549)
2018-04-11T08:54:58.702853111Z 	at com.hazelcast.cache.wan.CacheReplicationSupportingService.onReplicationEvent(CacheReplicationSupportingService.java:55)
2018-04-11T08:54:58.702856826Z 	at com.hazelcast.cache.EnterpriseCacheService.onReplicationEvent(EnterpriseCacheService.java:582)
2018-04-11T08:54:58.702859462Z 	at com.hazelcast.enterprise.wan.EnterpriseWanReplicationService$BatchWanEventRunnable.run(EnterpriseWanReplicationService.java:401)
2018-04-11T08:54:58.702862158Z 	at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:217)
2018-04-11T08:54:58.702873626Z 	at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:200)
2018-04-11T08:54:58.702876403Z 
2018-04-11T08:54:58.708100590Z Apr 11, 2018 8:54:58 AM com.hazelcast.enterprise.wan.EnterpriseWanReplicationService
2018-04-11T08:54:58.708125793Z SEVERE: [hazelcast-C-2]:5701 [dc-3] [3.8.6] null
2018-04-11T08:54:58.708130266Z java.lang.NullPointerException
2018-04-11T08:54:58.708132628Z 	at com.hazelcast.cache.impl.AbstractCacheService.putCacheConfigIfAbsent(AbstractCacheService.java:323)
2018-04-11T08:54:58.708135156Z 	at com.hazelcast.cache.EnterpriseCacheService.putCacheConfigIfAbsent(EnterpriseCacheService.java:549)
2018-04-11T08:54:58.708137501Z 	at com.hazelcast.cache.wan.CacheReplicationSupportingService.onReplicationEvent(CacheReplicationSupportingService.java:55)
2018-04-11T08:54:58.708139943Z 	at com.hazelcast.cache.EnterpriseCacheService.onReplicationEvent(EnterpriseCacheService.java:582)
2018-04-11T08:54:58.708142241Z 	at com.hazelcast.enterprise.wan.EnterpriseWanReplicationService$BatchWanEventRunnable.run(EnterpriseWanReplicationService.java:401)
2018-04-11T08:54:58.708144670Z 	at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:217)
2018-04-11T08:54:58.708147023Z 	at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:200)
2018-04-11T08:54:58.708149337Z 
2018-04-11T08:54:58.709786983Z Apr 11, 2018 8:54:58 AM com.hazelcast.enterprise.wan.EnterpriseWanReplicationService
2018-04-11T08:54:58.709811912Z SEVERE: [hazelcast-C-2]:5701 [dc-3] [3.8.6] null
2018-04-11T08:54:58.709817994Z java.lang.NullPointerException
2018-04-11T08:54:58.709822139Z 	at com.hazelcast.cache.impl.AbstractCacheService.putCacheConfigIfAbsent(AbstractCacheService.java:323)
2018-04-11T08:54:58.709825695Z 	at com.hazelcast.cache.EnterpriseCacheService.putCacheConfigIfAbsent(EnterpriseCacheService.java:549)
2018-04-11T08:54:58.709828069Z 	at com.hazelcast.cache.wan.CacheReplicationSupportingService.onReplicationEvent(CacheReplicationSupportingService.java:55)
2018-04-11T08:54:58.709830447Z 	at com.hazelcast.cache.EnterpriseCacheService.onReplicationEvent(EnterpriseCacheService.java:582)
2018-04-11T08:54:58.709832769Z 	at com.hazelcast.enterprise.wan.EnterpriseWanReplicationService$BatchWanEventRunnable.run(EnterpriseWanReplicationService.java:401)
2018-04-11T08:54:58.709835169Z 	at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:217)
2018-04-11T08:54:58.709837490Z 	at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:200)
...
@mmedenjak

This comment has been minimized.

Copy link
Contributor

@mmedenjak mmedenjak commented Sep 6, 2018

Caused by the local cache config lookup to fail: https://github.com/hazelcast/hazelcast-enterprise/blob/7404caf54c767b5d15f7bb30d98c68ba88feb98f/hazelcast-enterprise/src/main/java/com/hazelcast/cache/wan/CacheReplicationSupportingService.java#L118-L136

We can throw a more meaningful exception but otherwise, we can't address the problem as the source cluster may not be aware that the config is missing and will continue to send WAN events, causing more errors.

@mmedenjak mmedenjak modified the milestones: 3.11, 3.12 Sep 6, 2018
@mmedenjak mmedenjak removed this from the 3.12 milestone Feb 25, 2019
@mmedenjak mmedenjak added this to the 4.0 milestone Apr 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.