[improve][metadata] Add Option-set API to MetadataStore#25710
Merged
Conversation
Introduces a `MetadataStore.Option` sealed interface and a `Set<Option>`-taking overload on every read/write method (`get`, `getChildren`, `getChildrenFromStore`, `exists`, `put`, `delete`, `findByIndex` on `MetadataStore`; `put` on `MetadataStoreExtended` and `MetadataCache`). Existing methods are unchanged; the new overloads default to the existing forms, so this commit is purely additive — no callers need to change. Option subtypes: - `Ephemeral`, `Sequential` — replace `EnumSet<CreateOption>` flags - `SecondaryIndex(name, key)` — replaces the `Map<String, String>` argument - `PartitionKey(key)` — new; routing hint for sharded backends `OptionsHelper` provides typed accessors (`isEphemeral`, `partitionKey`, `secondaryIndexes`, ...) for backends and wrappers that need to consult the set. `MetadataStoreExtended` provides a default override for the new `put(path, value, version, Set<Option>)` that translates Option entries back into the legacy `EnumSet<CreateOption>` + `Map<String, String>` form and calls the existing extended-store put. This keeps every existing implementation correct without modification. Phase 1 of a multi-step migration: subsequent PRs wire each backend to override the new methods natively (Oxia for `PartitionKey` first), then migrate callers, and eventually remove the legacy forms.
Flip the direction of the Option-set <-> CreateOption translation so that Set<Option> is the abstract canonical method on every MetadataStore / MetadataStoreExtended / MetadataCache method, and the legacy no-opts / EnumSet<CreateOption> overloads are defaults that translate INTO the canonical form. Also flip the AbstractMetadataStore.storePut backend hook to take Set<Option> directly and update every backend (LocalMemory, RocksDB, ZK via OpPut, Oxia) to read options through OptionsHelper.
…ions # Conflicts: # pulsar-metadata/src/main/java/org/apache/pulsar/metadata/api/MetadataStore.java # pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java # pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/DualMetadataStore.java # pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/FaultInjectionMetadataStore.java # pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/LocalMemoryMetadataStore.java # pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/RocksdbMetadataStore.java # pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/oxia/OxiaMetadataStore.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The metadata-store API exposes per-operation flags through several disjoint mechanisms that have grown over time:
EnumSet<CreateOption>forEphemeral/SequentialMap<String, String> secondaryIndexesparameter for index hintsEach new flag has been added as an explicit method argument or a new overload, which does not scale. PIP-471 (metadata-driven transactions) needs PartitionKey routing on most read/write methods, which would otherwise require yet another argument across the surface.
Modifications
Introduce a single options bag based on a sealed
Optioninterface and makeSet<Option>the canonical form across the metadata API.API surface (
MetadataStore,MetadataStoreExtended,MetadataCache):Optionsealed interface with subtypesEphemeral,Sequential,SecondaryIndex(name, key),PartitionKey(key)OptionsHelperfor extracting typed values (isEphemeral,isSequential,secondaryIndexes,partitionKey)Set<Option>form is the abstract canonical method on every read/write entry point (get,getChildren,getChildrenFromStore,exists,put,delete,findByIndex)EnumSet<CreateOption>/Map<String,String>overloads are now defaults that translate into the canonicalSet<Option>form — old callers continue to work unchangedImplementation wiring:
AbstractMetadataStore.storePuthook signature flipped to(path, data, version, Set<Option>)— single method, the EnumSet+Map overload is goneLocalMemoryMetadataStore,RocksdbMetadataStore,OxiaMetadataStore(two overloads collapsed into one), andOpPut/AbstractBatchedMetadataStore(used byZKMetadataStore) all read options viaOptionsHelperDualMetadataStore,FaultInjectionMetadataStore,DualMetadataCache) override the canonicalSet<Option>methods; legacy callers route through the interface default forwardersMetadataEvent(sync replication payload) still carriesSet<CreateOption>—AbstractMetadataStorebridges to/fromSet<Option>at the event handler boundary;SecondaryIndex/PartitionKeyare not propagated through the legacy event payload, which matches existing behavior.Verifying this change
This change is already covered by existing tests since it preserves behavior of all current call sites — only the canonical override point moved.
Does this pull request potentially affect one of the following parts:
MetadataStore/MetadataStoreExtended/MetadataCache) — strictly additive: new abstractSet<Option>methods are added; legacy forms become defaults that forward, so existing callers and external implementations are unaffected.Matching PR in forked repository
PR in forked repository: https://github.com/merlimat/pulsar/pull/new/mmerli/metadata-options