Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cluster sharding state warmup with health check #27168

Closed
chbatey opened this issue Jun 18, 2019 · 4 comments
Closed

Cluster sharding state warmup with health check #27168

chbatey opened this issue Jun 18, 2019 · 4 comments
Assignees
@johanandren
Labels
3 - in progress Someone is working on this ticket t:cluster t:cluster-sharding
Milestone
2.6.18

Comments

@chbatey
Copy link
Member

chbatey commented Jun 18, 2019 •
edited by raboof
Loading

When a new node is added to a cluster using cluster sharding, if user requests are initially routed to it then each of them will incur a latency penalty as the ShardRegion gets the Shard location from the ShardCoordinator. This is for all Shards, not just the ones that will be moved.

The ShardCoordinator could instead send the current regions per shardId to the ShardRegion when it registers and a health check could be exposed (https://doc.akka.io/docs/akka-management/current/healthchecks.html) to prevent user traffic being routed to the node in environments like Kubernetes until this has happened.

This might help #30315.

/cc @jroper
@chbatey chbatey added t:cluster discuss Tickets that need some discussion before proceeding. Not decided if it's a good idea. labels Jun 18, 2019
@jroper
Copy link
Contributor

jroper commented Jun 19, 2019

I think this would be really helpful in elastic environments, thanks for raising!

@milanvdm
Copy link

Is there any progress on this? Since we deploy on ECS (which shuts down the oldest containers first), we have a lot of movement of our ShardCoordinator during a redeploy.
We would like to find a way to know when to route requests to the new nodes.

@patriknw patriknw mentioned this issue Sep 11, 2020
Open
@patriknw patriknw added 1 - triaged Tickets that are safe to pick up for contributing in terms of likeliness of being accepted and removed discuss Tickets that need some discussion before proceeding. Not decided if it's a good idea. labels Sep 11, 2020
@patriknw
Copy link
Member

One way would be to reply with all shard locations when the region register to the coordinator. I think it's better to add a separate message for this. Then it can maybe be used in other situations like #29589

It could be many shards, and might not fit in one single message, but we can split it several smaller.

@patriknw
Copy link
Member

The health check was added in #29638

@johanandren johanandren self-assigned this Nov 29, 2021
@johanandren johanandren added 3 - in progress Someone is working on this ticket and removed 1 - triaged Tickets that are safe to pick up for contributing in terms of likeliness of being accepted labels Nov 29, 2021
johanandren added a commit to johanandren/akka that referenced this issue Nov 29, 2021
@johanandren
Quick-push all allocated shards to region/proxy when registering akka…
a404334 
…#27168

This should shorten latencies per new shard that is addressed from
a newly joined region/proxy
@johanandren johanandren mentioned this issue Nov 29, 2021
Merged
johanandren added a commit that referenced this issue Dec 3, 2021
@johanandren
Quick-push all allocated shards to region/proxy when registering #27168
9c0e624 
This should shorten latencies per new shard that is addressed from
a newly joined region/proxy
@johanandren johanandren added this to the 2.6.18 milestone Dec 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@johanandren johanandren

Labels
3 - in progress Someone is working on this ticket t:cluster t:cluster-sharding
Projects
None yet
Milestone
2.6.18
Development

No branches or pull requests
5 participants
@jroper @patriknw @johanandren @chbatey @milanvdm