refactor(storage): per-bucket S3 profiles instead of s3+<name> scheme#5
Merged
Merged
Conversation
…> scheme Replaces the named-backend design (which embedded a config label in the `s3+<name>://` URI scheme, persisting it into the metastore) with per-bucket profiles keyed by bucket name under `storage.s3.profiles.<bucket>`. URIs stay canonical `s3://bucket/path`, so nothing synthetic is persisted: routing is by bucket name, resolved at runtime from node-local config. - quickwit-common/uri.rs: revert all s3+<name> scheme handling to upstream; the Uri type no longer needs to carry a non-canonical scheme. - quickwit-config: `named` -> `profiles` (keyed by bucket), `NamedS3StorageConfig` -> `S3ProfileConfig`, `is_named_backend` -> `is_profile`. Drop the URL-scheme name validation (a bucket name is not a scheme). - quickwit-storage: resolver extracts the bucket via parse_s3_uri and looks it up in `profiles`; match -> that profile's config + per-bucket client cache, no match -> primary backend. - docs: rewrite as "Per-bucket S3 profiles", including the bucket-name uniqueness caveat (a bucket name maps to exactly one backend). Validated end-to-end through the REST stack: ingest -> split -> upload -> publish -> search against in-cluster SeaweedFS and external OVH backends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworks S3-compatible multi-backend support per review feedback: instead of embedding a config label in the URI scheme (
s3+<name>://), which gets persisted into the metastore, route by bucket name via per-bucket profiles understorage.s3.profiles.<bucket>. URIs stay canonicals3://bucket/path— nothing synthetic is persisted; routing is resolved at runtime from node-local config.Why
The previous
s3+<name>://design baked a config-time label into durable metastore state (theindex_uriand per-split metadata). That made profiles un-renameable, tied environments to a shared label convention, and required teaching theUritype a non-canonical scheme. Keying on bucket name (already intrinsic to the object location) avoids all of that.Changes
s3+<name>scheme handling to upstream —Uristays canonical.named→profiles(keyed by bucket),NamedS3StorageConfig→S3ProfileConfig,is_named_backend→is_profile. Drop URL-scheme name validation (a bucket name isn't a scheme).parse_s3_uriand looks it up inprofiles— match → that profile's config + per-bucket client cache; no match → primary backend.Trade-off
Bucket-keyed routing cannot disambiguate the same bucket name across two different endpoints (S3-compatible namespaces are per-endpoint). Documented; acceptable since bucket names are under our control.
Test plan
cargo +nightly fmt --all -- --checkcargo clippy -p quickwit-config -p quickwit-storage --all-features --tests(clean)cargo test -p quickwit-config storage_config(14 passed) +-p quickwit-common uri(9 passed)🤖 Generated with Claude Code