Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

shard_reader: background reads out-living the parent reader can access freed range and slice #9719

Closed
denesb opened this issue Dec 2, 2021 · 3 comments
Closed

shard_reader: background reads out-living the parent reader can access freed range and slice #9719

denesb opened this issue Dec 2, 2021 · 3 comments
Milestone
5.0

Comments

@denesb
Copy link
Contributor

denesb commented Dec 2, 2021 •
edited

The shard reader can outlive its parent reader (the multishard reader). This creates a problem for lifecycle management: readers take the range and slice parameters by reference and users keep these alive until the reader is alive. The shard reader outliving the top-level reader means that any background read-ahead that it has to wait on will potentially have stale references to the range and the slice. This was seen in the wild recently when the evictable reader wrapped by the shard reader hit a use-after-free while wrapping up a background read-ahead. This problem was solved by fa43d76 but any previous versions are susceptible to it.

Affected versions: < 4.6
avikivity pushed a commit that referenced this issue Dec 2, 2021
@denesb @avikivity
mutation_reader: shard_reader: ensure referenced objects are kept alive
417e853 
The shard reader can outlive its parent reader (the multishard reader).
This creates a problem for lifecycle management: readers take the range
and slice parameters by reference and users keep these alive until the
reader is alive. The shard reader outliving the top-level reader means
that any background read-ahead that it has to wait on will potentially
have stale references to the range and the slice. This was seen in the
wild recently when the evictable reader wrapped by the shard reader hit
a use-after-free while wrapping up a background read-ahead.
This problem was solved by fa43d76 but any previous versions are
susceptible to it.

This patch solves this problem by having the shard reader copy and keep
the range and slice parameters in stable storage, before passing them
further down.

Fixes: #9719

Tests: unit(dev)
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20211202113910.484591-1-bdenes@scylladb.com>
denesb added a commit that referenced this issue Dec 6, 2021
@denesb
mutation_reader: shard_reader: ensure referenced objects are kept alive
cc92856 
The shard reader can outlive its parent reader (the multishard reader).
This creates a problem for lifecycle management: readers take the range
and slice parameters by reference and users keep these alive until the
reader is alive. The shard reader outliving the top-level reader means
that any background read-ahead that it has to wait on will potentially
have stale references to the range and the slice. This was seen in the
wild recently when the evictable reader wrapped by the shard reader hit
a use-after-free while wrapping up a background read-ahead.
This problem was solved by fa43d76 but any previous versions are
susceptible to it.

This patch solves this problem by having the shard reader copy and keep
the range and slice parameters in stable storage, before passing them
further down.

Fixes: #9719

Tests: unit(dev)
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20211202113910.484591-1-bdenes@scylladb.com>
(cherry picked from commit 417e853)
@denesb
Copy link
Contributor Author

denesb commented Dec 6, 2021

Closed by 417e853 (not merged to master as master is not vulnerable).

@denesb denesb closed this as completed Dec 6, 2021
@denesb
Copy link
Contributor Author

denesb commented Dec 6, 2021

Backported to 4.5 (initial merge) 4.4 and 2021.1

@denesb
Copy link
Contributor Author

denesb commented Dec 7, 2021

For posterity: this problem was observed when repairing a cluster which has mismatching shard counts and hence repair falls-back to the non-local reader (which uses the multishard reader, wrapped by a multi-range reader).

benipeled pushed a commit to benipeled/scylla that referenced this issue May 31, 2022
@denesb
mutation_reader: shard_reader: ensure referenced objects are kept alive
b1ae3af 
The shard reader can outlive its parent reader (the multishard reader).
This creates a problem for lifecycle management: readers take the range
and slice parameters by reference and users keep these alive until the
reader is alive. The shard reader outliving the top-level reader means
that any background read-ahead that it has to wait on will potentially
have stale references to the range and the slice. This was seen in the
wild recently when the evictable reader wrapped by the shard reader hit
a use-after-free while wrapping up a background read-ahead.
This problem was solved by fa43d76 but any previous versions are
susceptible to it.

This patch solves this problem by having the shard reader copy and keep
the range and slice parameters in stable storage, before passing them
further down.

Fixes: scylladb#9719

Tests: unit(dev)
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20211202113910.484591-1-bdenes@scylladb.com>
(cherry picked from commit 417e853)
@DoronArazii DoronArazii added this to the 5.0 milestone Nov 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
5.0
Development

No branches or pull requests
2 participants
@denesb @DoronArazii