Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flood in log when quorum is not present #9781

Closed
vmnikulin opened this issue Jan 30, 2017 · 3 comments

Comments

Projects
None yet
2 participants
@vmnikulin
Copy link

commented Jan 30, 2017

Hello!

When quorum is not present operations like MapSizeOperation produces 2 * 271 (542) log records on every client request. So when quorum is broken log made by MapSizeOperation quick fill my disk.

Log file looks like:
2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-5] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-0] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-4] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-0] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-5] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-1] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-7] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-1] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-0] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-1] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-4] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-1] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-4] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-1] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-0] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-1] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1 2017-01-30 17:35:14,676 [hz._hzInstance_1_dev.partition-operation.thread-0] WARN [MapSizeOperation] [192.168.1.151]:5701 [dev] [3.6.7] Cluster quorum failed, quorum minimum size: 2, current size: 1

This is true for every operation than invokes on all partitions (see InvokeOnPartitions )

As a temporary decision I set log level to ERROR for com.hazelcast.map.impl.operation package. But I'm afraid I miss some important WARN log record in future.

So my question is this a bug or feature? What should I do?

Version: 3.6.7, but if I understand correctly reasons of this behaviour, there are no changes in 3.8-EA

@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Jan 31, 2017

hi @vmnikulin,

you are right, there is no need to log this as the exception is propagate to the caller anyway. As a temporary measure you could set log level to error just to com.hazelcast.map.impl.operation.MapSizeOperation operation.

@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Jan 31, 2017

related to #9051

@jerrinot jerrinot added this to the 3.8 milestone Jan 31, 2017

jerrinot added a commit to jerrinot/hazelcast that referenced this issue Jan 31, 2017

Do not log business exceptions
Exceptions such as StaleSequenceException are part of a normal
flow. They are propagated to user code and there is no reason
why Hazelcast should log them by default.

Fixes hazelcast#9051, hazelcast#9781

jerrinot added a commit to jerrinot/hazelcast that referenced this issue Jan 31, 2017

Do not log business exceptions
Exceptions such as StaleSequenceException are part of a normal
flow. They are propagated to user code and there is no reason
why Hazelcast should log them by default.

Fixes hazelcast#9051, hazelcast#9781

jerrinot added a commit to jerrinot/hazelcast that referenced this issue Feb 2, 2017

Do not log business exceptions
Exceptions such as StaleSequenceException are part of a normal
flow. They are propagated to user code and there is no reason
why Hazelcast should log them by default.

Fixes hazelcast#9051, hazelcast#9781

jerrinot added a commit to jerrinot/hazelcast that referenced this issue Feb 2, 2017

Do not log business exceptions
Exceptions such as StaleSequenceException are part of a normal
flow. They are propagated to user code and there is no reason
why Hazelcast should log them by default.

Fixes hazelcast#9051, hazelcast#9781
@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Feb 2, 2017

closed by #9790

@jerrinot jerrinot closed this Feb 2, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.