fix(migrations): Wait for ON CLUSTER DDL instead of failing at 5 min - #8416
Open
phacops wants to merge 2 commits into
Open
fix(migrations): Wait for ON CLUSTER DDL instead of failing at 5 min#8416phacops wants to merge 2 commits into
phacops wants to merge 2 commits into
Conversation
distributed_ddl_task_timeout=300 threw after 5 minutes and continued the DDL async. Quiet DROP then IncompleteRead the HTTP body (SNUBA-C3Y). Wait for every host (-1), raise http_send_timeout, and re-send the 15s progress interval per request so clickhouse-connects 120s cap does not win. Client timeout is seconds (None to 1h), not leftover milliseconds. Refs SNUBA-C3Y
phacops
marked this pull request as ready for review
August 28, 2026 21:50
Drop the extra helper and overlapping tests. Progress keys are copied in _build_query_settings, which JSONCompact now reuses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ON CLUSTER migrations now wait for ClickHouse to finish instead of throwing at 5 minutes (and continuing the DDL in the background). Quiet DROP/ALTER no longer dies as
IncompleteReadafter that wait.distributed_ddl_task_timeoutis-1(wait for every host; real errors still return immediately).http_send_timeoutis 1h so a lock wait with no body does not close HTTP. The 15s progress interval from #8413 is re-sent on eachcommand()/execute()because clickhouse-connect overwrites constructor settings with a 120s cap. The client timeout isNone(1h urllib3 fallback) — the old300000was leftover milliseconds, not 5 minutes.Tradeoff: a replica that never acks hangs the migrate job instead of failing at 5 min while ClickHouse keeps going. That matches waiting for processing.
Refs SNUBA-C3Y