Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read execution stage can grow without bounds. #4749

Closed
gleb-cloudius opened this issue Jul 22, 2019 · 1 comment
Closed

Read execution stage can grow without bounds. #4749

gleb-cloudius opened this issue Jul 22, 2019 · 1 comment

Comments

@gleb-cloudius
Copy link
Contributor

@gleb-cloudius gleb-cloudius commented Jul 22, 2019

Installation details
Scylla version (or git commit hash): 7604980

We observe that under heavy read load read execution stage grows to several million entries:

execstage

The problem is that nothing is limiting the queue size. If works comes in faster than it can be dequeued the queue will grow until all memory is consumed. The graph of preempted execution stage flushes shows that indeed the queue cannot be depleted by a single flush task:

execpreempt

avikivity added a commit that referenced this issue Nov 13, 2019
* seastar 75488f6ef2...cfc082207c (2):
  > core: fix a race in execution stages
  > execution_stage: prevent unbounded growth

Fixes #4749.
Fixes #4856.
avikivity added a commit that referenced this issue Nov 13, 2019
* seastar 3920dcb3f8...083dc0875e (2):
  > core: fix a race in execution stages
  > execution_stage: prevent unbounded growth

Fixes #4749.
Fixes #4856.
@avikivity

This comment has been minimized.

Copy link
Contributor

@avikivity avikivity commented Nov 13, 2019

Backported to 3.0 and 3.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.