compaction_manager::perform_cleanup does not handle condition_variable_timed_out

[bhalevy]

As seen in SCT (#15461 (comment)), nodetool cleanup occasionally fails with exit status 2.

nodetool-cleanup-5-2-cleanup--db-node-d57e5c04-5/messages.log, for example, reveals this:

2023-09-28T17:32:50+00:00 nodetool-cleanup-5-2-cleanup--db-node-d57e5c04-5     !INFO | scylla[5490]:  [shard 10] compaction - [Cleanup keyspace1.standard1 3650c280-5e24-11ee-9514-0470f5d93d1d] Cleaned 64 sstables to [/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12260-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12274-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12288-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12302-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12316-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12330-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12344-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12358-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12372-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12386-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12400-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12414-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12428-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12442-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12456-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12470-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12484-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12498-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12512-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12526-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12540-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12554-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12568-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12582-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12596-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12610-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12624-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12638-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12652-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12666-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12680-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12694-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12708-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12722-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12736-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12750-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12764-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12778-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12792-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12806-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12820-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12834-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12848-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12862-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12876-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12890-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12904-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12918-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12932-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12946-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12960-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12974-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-12988-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-13016-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-13030-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-13044-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-13058-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-13072-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-13086-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-13100-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-13114-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-13128-big-Data.db:level=2,/var/lib/scylla/data/keyspace1/standard1-9d63c6405df011ee909e1266ed13bf99/me-13142-big-Data.db:level=2]. 10GB to 10GB (~98% of original) in 358081ms = 30MB/s. ~9814272 total partitions merged to 9660309.
2023-09-28T17:32:50+00:00 nodetool-cleanup-5-2-cleanup--db-node-d57e5c04-5     !INFO | scylla[5490]:  [shard 10] compaction - [Compact keyspace1.standard1_c1_nemesis_index 0bbfe3b0-5e25-11ee-9514-0470f5d93d1d] Compacting [/var/lib/scylla/data/keyspace1/standard1_c1_nemesis_index-d4ff27105dfb11ee8d4c884f632f0ea8/me-1144-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/keyspace1/standard1_c1_nemesis_index-d4ff27105dfb11ee8d4c884f632f0ea8/me-1158-big-Data.db:level=0:origin=memtable]
...
2023-09-28T17:33:00+00:00 nodetool-cleanup-5-2-cleanup--db-node-d57e5c04-5      !ERR | scylla[5490]:  [shard 10] api - Failed force_keyspace_cleanup of keyspace1.table{name=standard1, id=9d63c640-5df0-11ee-909e-1266ed13bf99}: seastar::condition_variable_timed_out (Condition variable timed out)

The conditional variable timeout originates from

scylladb/compaction/compaction_manager.cc

Line 1792 in 4e6fe34

co_await t.get_staging_done_condition().when(sleep_duration, [&] { return has_sstables_eligible_for_compaction(); });

that was supposed to be benign.
A longer timeout is checked at

scylladb/compaction/compaction_manager.cc

Lines 1775 to 1780 in 4e6fe34

	auto idle = seastar::lowres_clock::now() - last_idle;
	if (idle >= max_idle_duration) {
	auto msg = ::format("Cleanup timed out after {} seconds of no progress", std::chrono::duration_cast<std::chrono::seconds>(idle).count());
	cmlog.warn("{}", msg);
	co_await coroutine::return_exception(std::runtime_error(msg));
	}

That said, it is possible that if regular compaction will pick up a huge sstable that requires cleanup and it takes more than 5 minutes to compact it cleanup may still fail.

@Deexie in the above case maybe perform_cleanup can identify the respective task and wait on it.
But let's leave it for a follow-up.

[denesb]

@bhalevy please evaluate for backport.

[bhalevy]

@bhalevy please evaluate for backport.

Backport is required to 5.4 and 2024.1

[mykaul]

@bhalevy please evaluate for backport.

It seem to be needed.
@scylladb/scylla-maint - please backport (in the hope it's straightforward!)

[denesb]

Backported to 5.4.