feat: improve degraded readiness reporting and shutdown handling#3089
Merged
Conversation
Contributor
|
CLA requirements are satisfied for this pull request. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR continues the readiness/shutdown unification work by improving degraded readiness observability (structured reasons surfaced in health payloads + metrics) and by making server shutdown more deterministic via awaitable shutdown handles (replacing sleep-based guesses).
Changes:
- Add structured degraded readiness reporting (reasons + metrics) and separate startup-ready publication from cached health readiness.
- Introduce
ShutdownHandleand update HTTP/protocol servers to support signal-and-await shutdown semantics. - Update dependencies (remove git
mysql_async, bump several crates) and propagate health payload signature changes across endpoint/console/admin paths.
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| rustfs/src/server/service_state.rs | Add ShutdownSignal::log_label() for consistent shutdown logging + unit test. |
| rustfs/src/server/readiness.rs | Implement dependency readiness reporting, caching, metrics, and startup-ready publication helper. |
| rustfs/src/server/mod.rs | Add ShutdownHandle and re-export readiness helpers/types for internal consumers and the binary. |
| rustfs/src/server/layer.rs | Include degraded readiness reasons in public health endpoint payloads. |
| rustfs/src/server/http.rs | Return ShutdownHandle from start_http_server and await graceful drain via hyper-util’s graceful watcher. |
| rustfs/src/main.rs | Publish ready only after runtime readiness, and await shutdown of protocol/http tasks instead of sleeping. |
| rustfs/src/init.rs | Return ShutdownHandle from FTP/FTPS/WebDAV/SFTP init and spawn tasks with join handles. |
| rustfs/src/embedded.rs | Use ShutdownHandle and runtime-readiness publishing during embedded server startup and shutdown. |
| rustfs/src/app/admin_usecase.rs | Remove duplicated readiness computation; delegate to server readiness collection. |
| rustfs/src/admin/handlers/health.rs | Add degradedReasons to health payload/response and update tests. |
| rustfs/src/admin/console.rs | Include degraded readiness reasons in console health responses. |
| Cargo.toml | Move mysql_async/russh off git deps and bump several crate versions. |
| Cargo.lock | Lockfile updates reflecting dependency changes. |
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF ScorecardScorecard details
Scanned Files
|
cxymds
approved these changes
May 27, 2026
loverustfs
approved these changes
May 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
Continues rustfs/backlog#647
Summary of Changes
This follow-up PR continues the work from the merged readiness/shutdown fix and focuses on two areas:
It:
READY=1Verification
cargo check -p rustfs --bin rustfs --quietcargo test -p rustfs wait_for_runtime_readiness_with_does_not_publish_ready_when_runtime_readiness_is_not_reached --quietmake pre-commitImpact
Operators can now distinguish degraded readiness states more clearly instead of only seeing
ready=false.Startup readiness publication is also less likely to be delayed by stale cached readiness results, and shutdown behavior continues moving toward explicit signal-and-wait semantics instead of timing heuristics.
Additional Notes
crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs.