Skip to content

fix(tiering): gate remote version state writes - #5382

Merged
cxymds merged 27 commits into
mainfrom
cxymds/fix-1358-fleet-version-gate
Jul 29, 2026
Merged

fix(tiering): gate remote version state writes#5382
cxymds merged 27 commits into
mainfrom
cxymds/fix-1358-fleet-version-gate

Conversation

@cxymds

@cxymds cxymds commented Jul 28, 2026

Copy link
Copy Markdown
Member

Related Issues

Related to rustfs/backlog#1358.

Summary of Changes

  • Add a default-off local writer opt-in for persisting suspended-null and opaque remote-tier version states.
  • Require a separate operator-attested fleet-upgrade confirmation before the writer opt-in can take effect.
  • Preserve UUID remote versions without the gate.
  • Preserve empty PUT responses as KnownDisabled, matching the remote-tier invariant that an empty version means an unversioned tier bucket and must omit versionId on GET/DELETE.
  • Keep nil UUID responses rejected and require the fleet gate for suspended null and opaque version tokens.
  • Cover default, missing-confirmation, explicit-false, rollback, fully-upgraded, null, opaque, empty, and nil behavior.

Verification

  • cargo fmt --all --check
  • git diff --check
  • cargo test -p rustfs-config remote_version_state_gate_uses_stable_environment_names --lib -- --nocapture
  • cargo test -p rustfs-ecstore transition_version_id_tests --lib -- --nocapture
  • make pre-commit

The branch was synchronized with current main after #5374 merged. The merge resolution retained empty -> KnownDisabled, the dual-flag gate, and its regression assertions. Full CI on the synchronized branch remains authoritative.

Impact

This adds two default-off environment variables: RUSTFS_TIER_REMOTE_VERSION_STATE_WRITE and RUSTFS_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED. The second variable is an operator attestation, not automatic node capability discovery. Operators must upgrade every node that can read or write transitioned-object metadata before enabling both variables on writer nodes. To roll back, disable the local writer opt-in on every node before rolling binaries back. Incorrectly confirming a mixed-version fleet can expose old nodes to metadata they cannot interpret safely.

Unversioned tier buckets continue to work without either flag. Exact UUID semantics remain compatible. Only suspended-null and opaque provider versions require both flags.

Additional Notes

This PR follows merged #5374. It deliberately adds no GET/DELETE hot-path probes and no per-object peer RPCs. Cluster snapshot and runtime capability views are observational, while the tier-mutation peer protocol covers configuration mutations rather than every node that may write object metadata; neither is a safe automatic fleet gate for this behavior.

High-risk adversarial review:

  • Correctness: attacked default, missing/false confirmation, rollback, empty, nil UUID, null, and opaque states; empty remains KnownDisabled, nil remains rejected, and gated states require both flags.
  • Simplicity: compared the change with an inline-only gate and checked existing config helpers/constants; the small policy helpers keep production and tests on one decision path.
  • Security: malformed/default environment values remain default-off and remote IDs retain provider validation.
  • Concurrency/durability: either false flag disables new gated writes; no lock, await, RPC, or partial metadata write is added.
  • Compatibility: defaults preserve existing UUID/unversioned behavior; mixed-version deployment and rollback ordering are explicit.
  • Performance: only two local boolean config reads are added on transition commit; no GET/DELETE hot-path work.
  • Test coverage: reverting the AND gate or null/opaque mapping fails focused tests; stable environment names and default-off constants are pinned.

@github-actions

Copy link
Copy Markdown
Contributor

CLA requirements are satisfied for this pull request.

cxymds added 18 commits July 28, 2026 17:49
…e-transition-metadata

# Conflicts:
#	crates/ecstore/src/set_disk/ops/object.rs
…/fix-1358-fleet-version-gate

# Conflicts:
#	crates/ecstore/src/set_disk/ops/object.rs
…n-metadata' into cxymds/fix-1358-fleet-version-gate
…n-metadata' into cxymds/fix-1358-fleet-version-gate
…n-metadata' into cxymds/fix-1358-fleet-version-gate
…n-metadata' into cxymds/fix-1358-fleet-version-gate
…n-metadata' into cxymds/fix-1358-fleet-version-gate
Base automatically changed from cxymds/fix-1358-opaque-transition-metadata to main July 29, 2026 02:43
@cxymds
cxymds marked this pull request as ready for review July 29, 2026 02:59
@cxymds

cxymds commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

The synchronized branch exposed a Clippy-only layout failure: remote_version_state_tests appeared before the remainder of object.rs constants, triggering clippy::items_after_test_module.

Commit 908111e7a moves the unchanged test module to the end of the constants file. No runtime behavior or assertion changed.

Verification:

  • cargo fmt --all --check
  • git diff --check
  • CARGO_TARGET_DIR=/tmp/fleet-config-clippy cargo clippy -p rustfs-config --all-targets -- -D warnings

@cxymds

cxymds commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

CI follow-up for head 908111e7a:

  • The only failed check was Test and Lint job 90471230201.
  • Clippy completed successfully. Run tests started at 03:20 UTC but never received a step conclusion; the job ended at 03:43 UTC with the remaining anchor, JUnit upload, and migration-proof steps still pending.
  • The completed job log endpoint returns 404 and the run has no JUnit artifact, so there is no failing test assertion or branch-specific failure evidence.
  • This is the evidence-loss baseline fixed by merged ci: preserve timeout evidence for workspace tests #5402 (358af6a8d), which bounds the workspace test commands and uploads timeout diagnostics before the job deadline.

The branch has been normally merged with current origin/main at merge commit 4a007249c; there were no conflicts and the PR's own diff remains limited to the two tier-gate files. No retry, timeout increase, or gate relaxation was added.

Post-merge local verification:

  • cargo test -p rustfs-ecstore transition_version_id_tests — 4 passed
  • cargo test -p rustfs-config remote_version_state_gate_uses_stable_environment_names — 1 passed
  • cargo fmt --all --check
  • git diff --check

The PR remains Ready for review; the new CI run on the #5402 workflow is authoritative.

@cxymds
cxymds merged commit 2801b25 into main Jul 29, 2026
19 checks passed
@cxymds
cxymds deleted the cxymds/fix-1358-fleet-version-gate branch July 29, 2026 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant