New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sharded message left buffered until next retry #24191

Closed
t3hnar opened this Issue Dec 21, 2017 · 2 comments

Comments

Projects
None yet
3 participants
@t3hnar
Contributor

t3hnar commented Dec 21, 2017

I can see often the situation in sharding mechanism that the message which came in the middle of hand off process (caused by rebalance) is stuck in the shardBuffers until next Retry extends ShardRegionCommand which is by default set to akka.cluster.sharding.retry-interval = 2 s.

We have workaround this issue by lowering the retry-interval.

But would be nice to have it properly fixed by delivering buffered message upon hand off completed

Akka 2.5.6

patriknw added a commit that referenced this issue Dec 26, 2017

Reply to GetShardHome requests after rebalance, #24191
* Some GetShardHome requests were ignored (by design) during
  rebalance and they would be retried later.
* This optimization keeps tracks of such requests and reply
  to them immediately after rebalance has been completed and
  thereby the buffered messages in the region don't have to
  wait for next retry tick.
@t3hnar

This comment has been minimized.

Show comment
Hide comment
@t3hnar

t3hnar Dec 27, 2017

Contributor

@patriknw I can confirm that issue is fixed by your changes

Contributor

t3hnar commented Dec 27, 2017

@patriknw I can confirm that issue is fixed by your changes

patriknw added a commit that referenced this issue Jan 2, 2018

Reply to GetShardHome requests after rebalance, #24191
* Some GetShardHome requests were ignored (by design) during
  rebalance and they would be retried later.
* This optimization keeps tracks of such requests and reply
  to them immediately after rebalance has been completed and
  thereby the buffered messages in the region don't have to
  wait for next retry tick.

patriknw added a commit that referenced this issue Jan 9, 2018

Reply to GetShardHome requests after rebalance, #24191
* Some GetShardHome requests were ignored (by design) during
  rebalance and they would be retried later.
* This optimization keeps tracks of such requests and reply
  to them immediately after rebalance has been completed and
  thereby the buffered messages in the region don't have to
  wait for next retry tick.

patriknw added a commit that referenced this issue Jan 9, 2018

Reply to GetShardHome requests after rebalance, #24191
* Some GetShardHome requests were ignored (by design) during
  rebalance and they would be retried later.
* This optimization keeps tracks of such requests and reply
  to them immediately after rebalance has been completed and
  thereby the buffered messages in the region don't have to
  wait for next retry tick.
* use regionTerminationInProgress also during the update since
  all GetShardHome requests are not stashed

patriknw added a commit that referenced this issue Jan 9, 2018

Merge pull request #24209 from akka/wip-24191-sharding-opt2-patriknw
Reply to GetShardHome requests after rebalance, #24191

@patriknw patriknw added this to the 2.5.9 milestone Jan 9, 2018

@ktoso ktoso removed the 2 - pick next label Jan 11, 2018

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Jan 11, 2018

Member

Was resolved in #24209
To be released today in 2.5.9

Member

ktoso commented Jan 11, 2018

Was resolved in #24209
To be released today in 2.5.9

@ktoso ktoso closed this Jan 11, 2018

@ktoso ktoso removed the 1 - triaged label Jan 11, 2018

manonthegithub added a commit to manonthegithub/akka that referenced this issue Jan 31, 2018

Reply to GetShardHome requests after rebalance, akka#24191
* Some GetShardHome requests were ignored (by design) during
  rebalance and they would be retried later.
* This optimization keeps tracks of such requests and reply
  to them immediately after rebalance has been completed and
  thereby the buffered messages in the region don't have to
  wait for next retry tick.
* use regionTerminationInProgress also during the update since
  all GetShardHome requests are not stashed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment