Skip to content

feat(rc-admin)!: add non-exporting SSE-KMS round-trip diagnostic - #281

Merged
cxymds merged 20 commits into
mainfrom
cxymds/issue-1397-kms-roundtrip-diagnostic
Jul 23, 2026
Merged

feat(rc-admin)!: add non-exporting SSE-KMS round-trip diagnostic#281
cxymds merged 20 commits into
mainfrom
cxymds/issue-1397-kms-roundtrip-diagnostic

Conversation

@cxymds

@cxymds cxymds commented Jul 21, 2026

Copy link
Copy Markdown
Member

Related issues

Resolves rustfs/backlog#1397
Part of rustfs/backlog#1378
Roadmap: rustfs/backlog#1361

Background

RustFS does not expose the encrypt/decrypt Admin API probe used by mc admin kms key status. Its legacy data-key route returns plaintext key material and must not be exposed. A safe diagnostic therefore needs to exercise the real S3 SSE-KMS object path without exporting keys or accepting user plaintext.

Solution

  • add confirmed rc admin kms roundtrip ALIAS BUCKET [--key-id KEY_ID] --yes
  • resolve the configured default key when --key-id is omitted
  • internally generate exactly 4 KiB of random test content
  • write one randomly named temporary object with explicit SSE-KMS headers
  • read at most 4 KiB, compare the decrypted bytes directly, and avoid reporting a digest
  • always attempt permanent cleanup, including after write, read, or verification failure
  • use RustFS x-rustfs-force-delete: true semantics so versioned buckets do not intentionally retain the probe object
  • classify write/encrypt, read/decrypt, mismatch, permission, and cleanup failures without retaining adapter details
  • report only bucket, key identifier, pass/cleanup status, and timing metadata
  • add typed output-v3 schema, reference documentation, core scenario tests, S3 transport tests, CLI unit tests, and signed HTTP coverage
  • explicitly keep KMS-specific metrics unsupported until RustFS publishes a route or selector contract

Security

The command refuses to mutate without --yes. It never calls /kms/generate-data-key, never accepts test content from the command line, and never emits plaintext, ciphertext, digests, generated data keys, or the randomized temporary object name. Application-owned content buffers and downloaded bytes use zeroizing storage. Download size is bounded by both declared length and streamed bytes. A --debug end-to-end test asserts that the temporary path appears in neither stdout nor stderr.

Known operational risks

  • cleanup is best-effort: network uncertainty, retention/Object Lock, an intermediary stripping the custom header, or older servers may leave the 4 KiB encrypted probe object behind
  • a successful DELETE response is trusted; the command does not issue a post-delete HEAD/LIST proof of physical removal
  • RustFS force-delete is prefix-delete semantics; the probe uses a 128-bit random path segment to make collision negligible, but --yes remains mandatory
  • non-success S3 error bodies are parsed by the AWS SDK, so their memory bound is controlled by the SDK rather than the diagnostic's 4 KiB success-payload bound
  • heterogeneous backends may map unsupported standard-object behavior to a network error rather than UnsupportedFeature

The command reports observable cleanup failures and never claims cryptographic key export, server-side physical deletion proof, or transactional cleanup.

Compatibility

BREAKING contract marker: this extends the protected CLI reference and output-v3 KMS contract with the roundtrip operation. Existing output-v1 and output-v2 contracts remain unchanged.

This is a RustFS-native S3 object diagnostic; it does not claim wire or semantic compatibility with the MinIO Admin API key-status probe.

Validation

  • cargo fmt --all --check
  • CARGO_BUILD_JOBS=2 cargo clippy --workspace -- -D warnings
  • CARGO_BUILD_JOBS=2 cargo test --workspace
  • core KMS tests: 12 passed
  • core roundtrip scenarios: 6 passed
  • S3 diagnostic transport tests: 3 passed
  • S3 KMS tests: 18 passed
  • CLI roundtrip integration tests: 3 passed
  • CLI configuration tests: 7 passed
  • CLI key/status tests: 11 passed
  • output-v3 schema tests: 13 passed

All checks pass locally on head ed21712 after synchronization with current main. Current RustFS source was checked for SSE-KMS headers and force-delete behavior; signed test servers cover transport and redaction. A configured Local/Vault KMS plus versioned-bucket cleanup exercise remains explicit release validation because the standard smoke environment does not provision a KMS backend.

@cxymds
cxymds changed the base branch from cxymds/issue-1396-kms-config-lifecycle to main July 21, 2026 14:25
@cxymds cxymds closed this Jul 21, 2026
@cxymds cxymds reopened this Jul 21, 2026
@cxymds
cxymds changed the base branch from main to cxymds/issue-1396-kms-config-lifecycle July 21, 2026 14:25
@cxymds
cxymds changed the base branch from cxymds/issue-1396-kms-config-lifecycle to main July 21, 2026 14:38
@cxymds cxymds closed this Jul 21, 2026
@cxymds cxymds reopened this Jul 21, 2026
@cxymds
cxymds changed the base branch from main to cxymds/issue-1396-kms-config-lifecycle July 21, 2026 14:39
@cxymds

cxymds commented Jul 21, 2026

Copy link
Copy Markdown
Member Author

Implementation and validation update:

The initial CI instability came from combining strict JSON parsing with --debug in one end-to-end test. The coverage is now split into:

  1. a strict JSON contract test; and
  2. a separate debug-output disclosure test that verifies the temporary object path is absent from stdout and stderr.

Local validation passed:

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace

The implementation remains mock-contract tested; validation against a live configured RustFS KMS deployment is still recommended before merge.

Base automatically changed from cxymds/issue-1396-kms-config-lifecycle to main July 23, 2026 01:08
@cxymds
cxymds requested a review from overtrue July 23, 2026 01:14
@cxymds
cxymds marked this pull request as ready for review July 23, 2026 01:14
@cxymds

cxymds commented Jul 23, 2026

Copy link
Copy Markdown
Member Author

Current head ed21712 is synchronized with main. Full local fmt, clippy, and workspace tests pass. Parent synchronization exposed four new core error variants; the diagnostic now exhaustively classifies them while zeroizing path/version/message details. Thread-aware audit found no reviews or unresolved threads. The PR description explicitly documents best-effort cleanup, prefix-delete, SDK error-body, heterogeneous-backend, and missing live configured-KMS validation risks.

@cxymds

cxymds commented Jul 23, 2026

Copy link
Copy Markdown
Member Author

The initial Windows failure was an unrelated transient IncompleteMessage in the existing versioned_objects mock HTTP test. The failed Windows job was rerun unchanged and passed all tests in 4m0s; no code or test assertions were modified.

@cxymds
cxymds merged commit a97d058 into main Jul 23, 2026
28 of 29 checks passed
@cxymds
cxymds deleted the cxymds/issue-1397-kms-roundtrip-diagnostic branch July 23, 2026 01:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant