Skip to content

Backport #107119 to 25.8: Fix flaky test_async_load_databases::test_materialized_views_replicated#108917

Merged
alexey-milovidov merged 2 commits into
25.8from
backport/25.8/107119
Jul 1, 2026
Merged

Backport #107119 to 25.8: Fix flaky test_async_load_databases::test_materialized_views_replicated#108917
alexey-milovidov merged 2 commits into
25.8from
backport/25.8/107119

Conversation

@alexey-milovidov

@alexey-milovidov alexey-milovidov commented Jun 30, 2026

Copy link
Copy Markdown
Member

Closes: #108914
Related: #107119
Related: #107889

Backport of the test-only fix #107119 (commit 6166517952c1) to the 25.8 release branch. The fix landed on master on 2026-06-18 but was never backported, so test_async_load_databases::test_materialized_views_replicated and ::test_materialized_views_cascaded_multiple keep flaking on the 25.8 base and on every backport/25.8/* pull request built on it (e.g. #107889).

On 25.8 the test still uses disconnect_event.wait(90) (return value ignored) and un-timed INSERTs. Under sanitizers the server stop+start budget alone is ~120s, so a restart slower than the ignored 90s wait lets the main thread race ahead, and an INSERT to a half-restarted server can block up to 600s and exhaust the 900s pytest session timeout. The fix waits 180s and asserts it, bounds each INSERT with timeout=60, catches QueryTimeoutExceedException, and stops the warm-up INSERT loop once the restart finishes or its budget elapses.

Since #107119 merged, the CI database shows master clean (1879 OK / 0 FAIL across all sanitizers) while 25.8 still shows failures (59 OK / 7 FAIL) — full breakdown in #108914.

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

...

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

Version info

  • Merged into: 25.8.26.9

…aterialized_views_replicated

Cherry-pick of the test-only fix merged to `master` in #107119
(commit `6166517952c1`, change `e5479ff4869`). The fix is absent on
`25.8`, where `test_materialized_views_replicated` still uses
`disconnect_event.wait(90)` and un-timed INSERTs.

Under sanitizers the server stop+start budget alone is ~120s, so a
restart slower than the ignored 90s wait let the main thread race
ahead, and an INSERT to a half-restarted server could block up to 600s
and exhaust the 900s pytest session timeout. The fix waits 180s and
asserts it, bounds each INSERT with `timeout=60`, catches
`QueryTimeoutExceedException`, and stops the warm-up INSERT loop once
the restart finishes or its budget elapses.

This clears the `test_async_load_databases` failures observed on the
`25.8` base and on `backport/25.8/*` PRs (e.g. #107889). True `master`
has been clean since #107119 merged (2026-06-18).

Closes: #108914

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@clickhouse-gh

clickhouse-gh Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Workflow [BackportPR], commit [7154c69]

@alexey-milovidov

Copy link
Copy Markdown
Member Author

The only CI failure (Integration tests (amd_tsan, 6/6)) is unrelated to this backport.

The backported tests themselves pass: test_async_load_databases::test_materialized_views_replicated and ::test_materialized_views_cascaded_multiple are OK in both the amd_asan, old analyzer, 4/6 and amd_tsan, 4/6 shards.

The failing test is test_polymorphic_parts::test_polymorphic_parts_non_adaptive, which actually passed; the job failed because ThreadSanitizer reported a data race in the Keeper instance (zoo1) during the cluster teardown:

WARNING: ThreadSanitizer: data race
  Read of size 1 ... DB::KeeperStorageBase::AuthID::operator==
    ... DB::preprocess<...>(Coordination::ZooKeeperAuthRequest const&, ...)  (under raft lock)
  Previous write of size 8 ... DB::KeeperStorageBase::AuthID::AuthID(AuthID&&)
    ... DB::KeeperStorage<...>::commit(...)::operator()<DB::AddAuthDelta>     (under auth_mutex)

This is the known race tracked in #82719 and fixed on master by #100396 (copy instead of move in KeeperStorage::commit), which was never backported. 25.8 still has the racy std::move(*operation.auth_id) in KeeperStorage::commit, so it keeps flaking on TSan integration runs.

I opened the backport of that fix to 25.8: #108965. Once it lands and 25.8 is merged in here, this flake should stop. Re-running the failed job meanwhile.

@alexey-milovidov

Copy link
Copy Markdown
Member Author

Update: the re-run of Integration tests (amd_tsan, 6/6) passed (completed at 19:42 UTC), so this PR is now fully green and MERGEABLE.

This confirms the earlier analysis: the failure was the flaky Keeper TSan data race (#82719), not anything in this test-only backport. The backported tests test_async_load_databases::test_materialized_views_replicated and ::test_materialized_views_cascaded_multiple pass.

The durable fix for the underlying race is the 25.8 backport #108965 (copy instead of move in KeeperStorage::commit); it does not block this PR, which is ready to merge.

@alexey-milovidov

Copy link
Copy Markdown
Member Author

Merged the base `25.8` into this branch (`3e51fbda` → `7154c699`, tests/integration/test_async_load_databases/test.py is the only PR-content change — verified unchanged).

The branch was ~1 day / 21 commits behind `25.8` and still showed a stale `FAILURE` commit status for `Integration tests (amd_tsan, 6/6)` (the flaky Keeper TSan AuthID race #82719, re-run to green earlier but the legacy status lingered → mergeStateStatus: UNSTABLE).

The durable fix #108965 (backport of #100396 to `25.8`) merged into the base at 05:43 UTC today, so this merge pulls it in. The fresh BackportPR run now builds on a base that no longer has the racy `std::move(*operation.auth_id)` in `KeeperStorage::commit`, so the flake cannot recur and the new commit statuses should come back clean.

@alexey-milovidov alexey-milovidov merged commit da1d740 into 25.8 Jul 1, 2026
33 checks passed
@alexey-milovidov alexey-milovidov deleted the backport/25.8/107119 branch July 1, 2026 21:31
@alexey-milovidov alexey-milovidov self-assigned this Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant