SYSTEM FLUSH LOGS query_log hangs until the 180 s timeout in the Stateless tests (amd_tsan, flaky check) job, failing whatever test happens to flush the query log. The failure is not specific to any one test or pull request: since 2026-07-24 it has hit at least 5 unrelated pull requests and 12 different tests.
Symptom (client-visible):
Code: 159. DB::Exception: Timeout exceeded (180 s) while flushing system log 'DB::SystemLogQueue<DB::QueryLogElement>'. (TIMEOUT_EXCEEDED)
(query: SYSTEM FLUSH LOGS query_log;)
with the waiting side in DB::SystemLogQueue<DB::QueryLogElement>::waitFlush (src/Common/SystemLogBase.cpp:212) via SystemLogs::flushImpl → InterpreterSystemQuery::execute. Once it starts happening in a job, every subsequent SYSTEM FLUSH LOGS query_log in the same server run also times out (three consecutive 180 s failures three minutes apart), so the flushing thread looks stuck rather than merely slow.
Evidence that this is job-wide and not caused by any of the affected pull requests (play.clickhouse.com, checks table, test_context_raw LIKE '%while flushing system log%', check_name LIKE '%tsan, flaky%', last 14 days):
| pull request |
tests affected |
| #110084 |
04552_uuid2_parquet_stats_pushdown |
| #96130 |
03735_async_insert_mergetree, 02770_async_buffer_ignore, 01343_min_bytes_to_use_mmap_io, 03735_async_insert_mergetree_count, 03168_query_log_privileges_not_empty, 03008_optimize_equal_ranges, 03315_query_log_privileges_backup_restore, 01344_min_bytes_to_use_mmap_io_index |
| #104591 |
01475_read_subcolumns |
| #110015 |
03164_s3_settings_for_queries_and_merges, 00167_read_bytes_from_fs, 03274_prewarm_primary_index_cache, 03707_parquet_metadata_cache, 04103_deltalake_parquet_metadata_cache, 03310_index_hints_read_columns |
| #76867 |
04611_join_runtime_filters_index_analysis_skip_index |
Per-day counts of such failures in tsan jobs over 60 days: isolated singletons on 2026-05-31, 06-01, 06-09, 06-25, 06-27, 06-30, then 4 on 2026-07-24, 11 on 2026-07-25, 13 on 2026-07-27 — i.e. a step change starting the evening of 2026-07-24 (first occurrence 2026-07-24 23:29:14 UTC).
The affected tests take a few seconds normally (for example 01475_read_subcolumns on the same commit: 22 runs OK with a maximum of 8.3 s, 3 runs failing at 180–190 s), so this is a hang in system-log flushing, not test slowness.
Example report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=104591&sha=8892769dcf2ae9324f84bbe4ed3b298a707f7e72&name_0=PR&name_1=Stateless%20tests%20%28amd_tsan%2C%20flaky%20check%29
Candidate leads (both merged into master shortly before the first occurrence, both present in every failing head, so ancestry alone does not discriminate between them):
The flaky check is the job that surfaces it because it runs a changed test ~25 times against one long-lived server, which is exactly the load pattern that lets the flush queue fall behind or deadlock.
Version info
SYSTEM FLUSH LOGS query_loghangs until the 180 s timeout in theStateless tests (amd_tsan, flaky check)job, failing whatever test happens to flush the query log. The failure is not specific to any one test or pull request: since 2026-07-24 it has hit at least 5 unrelated pull requests and 12 different tests.Symptom (client-visible):
with the waiting side in
DB::SystemLogQueue<DB::QueryLogElement>::waitFlush(src/Common/SystemLogBase.cpp:212) viaSystemLogs::flushImpl→InterpreterSystemQuery::execute. Once it starts happening in a job, every subsequentSYSTEM FLUSH LOGS query_login the same server run also times out (three consecutive 180 s failures three minutes apart), so the flushing thread looks stuck rather than merely slow.Evidence that this is job-wide and not caused by any of the affected pull requests (
play.clickhouse.com,checkstable,test_context_raw LIKE '%while flushing system log%',check_name LIKE '%tsan, flaky%', last 14 days):04552_uuid2_parquet_stats_pushdown03735_async_insert_mergetree,02770_async_buffer_ignore,01343_min_bytes_to_use_mmap_io,03735_async_insert_mergetree_count,03168_query_log_privileges_not_empty,03008_optimize_equal_ranges,03315_query_log_privileges_backup_restore,01344_min_bytes_to_use_mmap_io_index01475_read_subcolumns03164_s3_settings_for_queries_and_merges,00167_read_bytes_from_fs,03274_prewarm_primary_index_cache,03707_parquet_metadata_cache,04103_deltalake_parquet_metadata_cache,03310_index_hints_read_columns04611_join_runtime_filters_index_analysis_skip_indexPer-day counts of such failures in
tsanjobs over 60 days: isolated singletons on 2026-05-31, 06-01, 06-09, 06-25, 06-27, 06-30, then 4 on 2026-07-24, 11 on 2026-07-25, 13 on 2026-07-27 — i.e. a step change starting the evening of 2026-07-24 (first occurrence 2026-07-24 23:29:14 UTC).The affected tests take a few seconds normally (for example
01475_read_subcolumnson the same commit: 22 runs OK with a maximum of 8.3 s, 3 runs failing at 180–190 s), so this is a hang in system-log flushing, not test slowness.Example report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=104591&sha=8892769dcf2ae9324f84bbe4ed3b298a707f7e72&name_0=PR&name_1=Stateless%20tests%20%28amd_tsan%2C%20flaky%20check%29
Candidate leads (both merged into
mastershortly before the first occurrence, both present in every failing head, so ancestry alone does not discriminate between them):system-log-self-contained-elements, merged 2026-07-24 16:22 UTC) — changes how system-log elements are held before flushing.CompressionCodecAdaptivewith per-block codec selection #111134 (adaptive-codec, merged 2026-07-24 16:08 UTC) — adaptive codec selection could make writing system-log parts much more expensive underTSan.The flaky check is the job that surfaces it because it runs a changed test ~25 times against one long-lived server, which is exactly the load pattern that lets the flush queue fall behind or deadlock.
Version info
26.8.1.268(included in26.8and later)