Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix isMaster() guards in migration system #15617

Merged
merged 1 commit into from Sep 27, 2019

Conversation

@mdogan
Copy link
Member

mdogan commented Sep 26, 2019

node.isMaster() was being used in migration mechanism
to detect whether or not a node is master. Depending on this
check, a process executes or rejects to execute.

But while a migration method is executing under partition service
lock, the node may become master (when former master leaves the cluster).
This may cause inconsistenct behaviours during execution of the specific method.

For example, master node increments partition version when a replica ownership
changes. If a node becomes master while appyling partition table updates,
the race mentioned above can cause multiple increments of partition version.

To fix this issue, instead of checking node.isMaster(), we check “is master?” by using the latest observed master by the partition service under partition lock, so we order observation of master and partition update operations.

Fixes #hazelcast/hazelcast-enterprise#2837

`node.isMaster()` was being used in migration mechanism
to detect whether or not a node is master. Depending on this
check, a process executes or rejects to execute.

But while a migration method is executing under partition service
lock, the node may become master (when former master leaves the cluster).
This may cause inconsistenct behaviours during execution of the specific method.

For example, master node increments partition version when a replica ownership
changes. If a node becomes master while appyling partition table updates,
the race mentioned above can cause multiple increments of partition version.
Copy link
Contributor

metanet left a comment

🥇

Copy link
Contributor

mmedenjak left a comment

💯

@mdogan

This comment has been minimized.

Copy link
Member Author

mdogan commented Sep 27, 2019

Thanks for reviews @metanet & @mmedenjak! And thanks again @mmedenjak for the great analysis in the issue #hazelcast/hazelcast-enterprise#2837 🙏

@mdogan mdogan merged commit fbda622 into hazelcast:master Sep 27, 2019
1 check passed
1 check passed
default Test PASSed.
Details
@mdogan mdogan deleted the mdogan:parititon-version-race-fix branch Sep 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.