Skip to content

Add timeout for minio restart in CI to avoid hanging forever#97655

Merged
alexey-milovidov merged 1 commit intomasterfrom
fix-minio-restart-timeout
Feb 23, 2026
Merged

Add timeout for minio restart in CI to avoid hanging forever#97655
alexey-milovidov merged 1 commit intomasterfrom
fix-minio-restart-timeout

Conversation

@alexey-milovidov
Copy link
Copy Markdown
Member

@alexey-milovidov alexey-milovidov commented Feb 22, 2026

Summary

  • The /mc admin service restart clickminio --wait command in create_minio_log_tables uses Shell.get_output which calls subprocess.run with no timeout, so it can hang indefinitely
  • When this happens, the entire CI test job fails without running any tests (as seen in the linked CI report where a 35-minute hang occurred)
  • Added a 60-second timeout to the minio restart; if exceeded, kill the minio process with -9 and start it fresh (the webhook config persists in minio's data directory)

CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=97647&sha=e8326de84a9cb6eb786a3ddf7d28bd1b8d48f171&name_0=PR&name_1=Stateless%20tests%20%28amd_tsan%2C%20sequential%2C%202%2F2%29
#97647

Changelog category (leave one):

  • CI Fix or improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Add timeout for minio restart in CI to avoid test jobs hanging forever.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

🤖 Generated with Claude Code

The `/mc admin service restart clickminio --wait` command can hang
indefinitely during CI setup, causing the entire test job to fail
without running any tests. Add a 60-second timeout; if exceeded,
kill the minio process with `-9` and start it fresh.

CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=97647&sha=e8326de84a9cb6eb786a3ddf7d28bd1b8d48f171&name_0=PR&name_1=Stateless%20tests%20%28amd_tsan%2C%20sequential%2C%202%2F2%29

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh Bot commented Feb 22, 2026

Workflow [PR], commit [098d9d0]

Summary:

@alexey-milovidov alexey-milovidov self-assigned this Feb 23, 2026
@alexey-milovidov alexey-milovidov added this pull request to the merge queue Feb 23, 2026
Merged via the queue into master with commit d64f376 Feb 23, 2026
148 checks passed
@alexey-milovidov alexey-milovidov deleted the fix-minio-restart-timeout branch February 23, 2026 01:19
@robot-ch-test-poll robot-ch-test-poll added the pr-synced-to-cloud The PR is synced to the cloud repo label Feb 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants