Skip to content
This repository has been archived by the owner on Mar 31, 2022. It is now read-only.

Reaper pausing on too many compaction > 20 #139

Open
dasmohapatrak opened this issue May 4, 2016 · 4 comments
Open

Reaper pausing on too many compaction > 20 #139

dasmohapatrak opened this issue May 4, 2016 · 4 comments

Comments

@dasmohapatrak
Copy link

Hi,
On our Cassandra cluster, i see the below message no of times when the reaper is running.

SegmentRunner - SegmentRunner declined to repair segment 556369 because of too many pending compactions (> 20) on host.

But when i run the "nodetool compactionstats" on that node, most of the time, i see only one compaction task is running. Though the pending tasks show more than 20 .
Now, i understand the "pending tasks" is actually only an estimate, not the actual no. of tasks pending.

So my question is, why should Reaper wait on the pending tasks, even though there are less than 20 pending tasks running on that node.

Thanks

Kishore

@deniszh
Copy link

deniszh commented May 9, 2016

IIRC "pending tasks" includes all tasks, not only compaction tasks.

@Bj0rnen
Copy link
Contributor

Bj0rnen commented May 9, 2016

It's the responsibility of the compaction strategy to estimate how many remaining compaction tasks it has, in effect, got "queued up". This, for all tables summed up is what you see as "pending tasks".

If we would go ahead and repair just because there are few ongoing repairs right now, there would be more SSTables piling up, and the node would fall behind on compactions. The pending compactions are what matters, because we don't want to overwhelm the nodes with SSTables.

@dasmohapatrak
Copy link
Author

When you mention all tables, does that mean it includes the keyspaces for OpsCenter, system_traces and system also.
We do not run reaper against these keyspaces.

Thanks

Kishore Mohapatra
Principal Database Administrator
Seattle, WA
Ph : 425-691-6417 (cell)
Email : kishore.mohapatra@nuance.commailto:kishore.mohapatra@nuance.com

From: Björn Hegerfors [mailto:notifications@github.com]
Sent: Monday, May 09, 2016 9:25 AM
To: spotify/cassandra-reaper
Cc: Mohapatra, Kishore; Author
Subject: [EXTERNAL] Re: [spotify/cassandra-reaper] Reaper pausing on too many compaction > 20 (#139)

It's the responsibility of the compaction strategy to estimate how many remaining compaction taskshttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_blob_e16d8a7a667d50271a183a95be894126cb2a5414_src_java_org_apache_cassandra_db_compaction_AbstractCompactionStrategy.java-23L209&d=CwMCaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=O20_rcIS1QazTO3_J10I1cPIygxnuBZ4sUCz1TS16XE&m=0OeAb06DF5CHvF-hfe4C1baanC0t7TODYltmJ-KNdwQ&s=_f_MdZ7EZIBN1sXpG7vbgn_7pQAxxMx-rPGC4xozEMI&e= it has, in effect, got "queued up". This, for all tables summed up is what you see as "pending tasks".

If we would go ahead and repair just because there are few ongoing repairs right now, there would be more SSTables piling up, and the node would fall behind on compactions. The pending compactions are what matters, because we don't want to overwhelm the nodes with SSTables.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_spotify_cassandra-2Dreaper_issues_139-23issuecomment-2D217914486&d=CwMCaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=O20_rcIS1QazTO3_J10I1cPIygxnuBZ4sUCz1TS16XE&m=0OeAb06DF5CHvF-hfe4C1baanC0t7TODYltmJ-KNdwQ&s=xdWVKg4ccyYzrIG07_VfnoFKzr25ijKvQ-Zf-9qIepQ&e=

@Bj0rnen
Copy link
Contributor

Bj0rnen commented May 9, 2016

I don't actually know. In practice, at least the tables in system and system_traces don't seem to make a meaningful impact, if any at all. It's probably because they are all very small tables (except hints sometimes), orders of magnitude smaller than user tables. Compactions take seconds and don't happen often. Or maybe they aren't even tracked. I don't have any experience with OpsCenter, so I can't give you an answer on that.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants