Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed-Up Partition Migrations #14351

Merged
merged 1 commit into from Feb 14, 2019

Conversation

Projects
None yet
5 participants
@mdogan
Copy link
Member

commented Jan 8, 2019

When partition count becomes higher, cost of publishing
the new partition state increases dramatically.
And if partition count is high but average data size
in partitions is low, then committing migration
and publishing new partition state dominate most of the
total migration time.

To fix that, instead of sending the whole partition table
after each migration, only the latest completed migrations are
sent. Members receiving completed migrations list, apply those
to their partition state and generate the new partition table
theirselves.

Additionally, there's no need to send completed migrations
to all cluster members after each migration. Instead, completed migrations
are sent to source and destination members of the migration
inside migration operations. Remaining members receive
completed migrations in batches asynchronously.

There's a minor change in periodic partition table publishing task too.
Instead of publishing partition table periodically to all members,
master asks whether their partition table is stale when compared to
master's version. If one responds as its partition table is stale,
master sends partition table to that specific member only.

@mdogan mdogan added this to the 3.12 milestone Jan 8, 2019

@mdogan mdogan force-pushed the mdogan:migration-speedup branch from de93475 to 8b7d248 Jan 8, 2019

@mdogan mdogan changed the title [WIP] Speed-Up Partition Migrations Speed-Up Partition Migrations Jan 8, 2019

@mdogan mdogan force-pushed the mdogan:migration-speedup branch from 8b7d248 to fece1ef Jan 8, 2019

@metanet metanet self-requested a review Jan 8, 2019

@mdogan mdogan force-pushed the mdogan:migration-speedup branch from fece1ef to b81908a Jan 9, 2019

@Danny-Hazelcast

This comment has been minimized.

Copy link
Member

commented Jan 16, 2019

EE master will not compile on top of this branch.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project hazelcast-enterprise: Compilation failure
[ERROR] /disk1/jenkins/hazelcast-enterprise/hazelcast-enterprise/src/main/java/com/hazelcast/internal/serialization/impl/NativeMemoryData.java:[170,4] error: method does not override or implement a method from a supertype

@mdogan

This comment has been minimized.

Copy link
Member Author

commented Jan 16, 2019

@Danny-Hazelcast: While on this branch, you can rebase it on latest master:

git fetch origin master
git rebase origin/master
@Danny-Hazelcast

This comment has been minimized.

Copy link
Member

commented Jan 16, 2019

I still get the same EE complication issue.

@Danny-Hazelcast

This comment has been minimized.

Copy link
Member

commented Jan 20, 2019

@mdogan i have created this issue hazelcast/hazelcast-enterprise#2683

as I was running some test with your branch.

the
git fetch origin master
git rebase origin/master
steps gives a new commit hash which is reported in the hazelcast logs,

Speed up partition migrations
When partition count becomes higher, cost of publishing
the new partition state increases dramatically.
And if partition count is high but average data size
in partitions is low, then committing migration
and publishing new partition state dominate most of the
total migration time.

To fix that, instead of sending the whole partition table
after each migration, only the latest completed migrations are
sent. Members receiving completed migrations list, apply those
to their partition state and generate the new partition table
theirselves.

Additionally, there's no need to send completed migrations
to all cluster members after each migration. Instead, completed migrations
are sent to source and destination members of the migration
inside migration operations. Remaining members receive
completed migrations in batches asynchronously.

There's a minor change in periodic partition table publishing task too.
Instead of publishing partition table periodically to all members,
master asks whether their partition table is state when compared to
master's version. If one responds as its partition table is stale,
master sends partition table to that specific member only.

@mdogan mdogan force-pushed the mdogan:migration-speedup branch from b81908a to 7b62376 Jan 21, 2019

@metanet
Copy link
Contributor

left a comment

pure perfection

@mmedenjak mmedenjak requested a review from ahmetmircik Feb 14, 2019

@mmedenjak mmedenjak merged commit 7ce31e7 into hazelcast:master Feb 14, 2019

1 check passed

default Test PASSed.
Details
@mmedenjak

This comment has been minimized.

Copy link
Contributor

commented Feb 14, 2019

Great work and thanks for the reviews guys!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.