Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scylla hangs or even segfaults when querying system.size_estimates #4689

Closed
kbr- opened this issue Jul 10, 2019 · 2 comments · Fixed by #4702

Comments

@kbr-
Copy link
Contributor

commented Jul 10, 2019

Installation details
Scylla version (or git commit hash): branch master as of time of writing this issue, commit a0499bb. But it's been there since at least 256b7d3 (4 months ago). I weren't able to compile previous commits because of the old fmt dependency.
Cluster size: 1
OS (RHEL/CentOS/Ubuntu/AWS AMI): Fedora 29, kernel 5.1.16-200.fc29.x86_64

Easy to reproduce on a fresh database:
select * from system.size_estimates
timeouts, Scylla starts using 100% CPU and can't be closed with usual interrupt (needs to be killed). Most probably goes into an infinite loop somewhere.

I even managed to create a database in which the above results in a segfault. Unsurprisingly, the trace points to the function db::size_estimates::size_estimates_mutation_reader::estimates_for_current_keyspace.

@kbr- kbr- self-assigned this Jul 10, 2019
@kbr- kbr- added the bug label Jul 10, 2019
@slivne slivne added this to the 3.2 milestone Jul 11, 2019
kbr- added a commit to kbr-/scylla that referenced this issue Jul 12, 2019
…imates.

Queries to system.size_estimates table which are not single parition queries
caused Scylla to go into an infinite loop inside multishard_combining_reader::fill_buffer.
This happened because multishard_combinind_reader assumes that shards return rows belonging
to separate partitions, which was not the case for size_estimates_mutation_reader.
This commit fixes the issue and closes scylladb#4689.
tgrabiec added a commit that referenced this issue Jul 15, 2019
Fixes a segfault when querying for an empty keyspace.

Also, fixes an infinite loop on smp > 1. Queries to
system.size_estimates table which are not single-partition queries
caused Scylla to go into an infinite loop inside
multishard_combining_reader::fill_buffer. This happened because
multishard_combinind_reader assumes that shards return rows belonging
to separate partitions, which was not the case for
size_estimates_mutation_reader.

Fixes #4689.
avikivity added a commit that referenced this issue Aug 7, 2019
Fixes a segfault when querying for an empty keyspace.

Also, fixes an infinite loop on smp > 1. Queries to
system.size_estimates table which are not single-partition queries
caused Scylla to go into an infinite loop inside
multishard_combining_reader::fill_buffer. This happened because
multishard_combinind_reader assumes that shards return rows belonging
to separate partitions, which was not the case for
size_estimates_mutation_reader.

Fixes #4689.

(cherry picked from commit 14700c2)
@avikivity

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2019

Backported to 3.1. @kbr- please prepare backports for 3.0 and 2.3 (if vulnerable).

@avikivity

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2019

@kbr- ^^^

kbr- added a commit to kbr-/scylla that referenced this issue Aug 14, 2019
…imates.

Queries to system.size_estimates table which are not single parition queries
caused Scylla to go into an infinite loop inside multishard_combining_reader::fill_buffer.
This happened because multishard_combinind_reader assumes that shards return rows belonging
to separate partitions, which was not the case for size_estimates_mutation_reader.
This commit fixes the issue and closes scylladb#4689.
kbr- added a commit to kbr-/scylla that referenced this issue Aug 14, 2019
…imates.

Queries to system.size_estimates table which are not single parition queries
caused Scylla to go into an infinite loop inside multishard_combining_reader::fill_buffer.
This happened because multishard_combinind_reader assumes that shards return rows belonging
to separate partitions, which was not the case for size_estimates_mutation_reader.
This commit fixes the issue and closes scylladb#4689.
tgrabiec added a commit that referenced this issue Aug 14, 2019
Fixes a segfault when querying for an empty keyspace.

Also, fixes an infinite loop on smp > 1. Queries to
system.size_estimates table which are not single-partition queries
caused Scylla to go into an infinite loop inside
multishard_combining_reader::fill_buffer. This happened because
multishard_combinind_reader assumes that shards return rows belonging
to separate partitions, which was not the case for
size_estimates_mutation_reader.

Fixes #4689
tgrabiec added a commit that referenced this issue Aug 14, 2019
Fixes a segfault when querying for an empty keyspace.

Also, fixes an infinite loop on smp > 1. Queries to
system.size_estimates table which are not single-partition queries
caused Scylla to go into an infinite loop inside
multishard_combining_reader::fill_buffer. This happened because
multishard_combinind_reader assumes that shards return rows belonging
to separate partitions, which was not the case for
size_estimates_mutation_reader.

Fixes #4689
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.