[protocol] Stage storageMode on StoreMetaValue v44 and AdminOperation v99#2814
Merged
sixpluszero merged 3 commits intoMay 21, 2026
Merged
Conversation
… v99
Add a new storageMode int field on both StoreVersion (per-version state)
and UpdateStore (store-level admin op) to support the upcoming batch-only
store dual-write migration.
Values:
0 => INTERNAL (default, Venice-only — current behavior)
1 => DUAL_WRITE_FROM_LEADER (version partition leader synchronously
dual-writes each record to the configured external storage on the
consumer pipeline before producing to local VT)
2 => DUAL_WRITE_FROM_VPJ (the Venice Push Job emits each record to the
configured external storage in parallel with Kafka produce, fanning
out to every target region)
3 => EXTERNAL (external-storage-only; Venice's data partition becomes
NoOp while metadata partitions are still persisted locally for
checkpointing)
Both v44 and v99 are already pinned to v43/v98 in build.gradle (alongside
the staged targetRegionPromoted field), so this is a forward-compatible
schema-only change with no behavioral impact. Java wiring
(UpdateStoreQueryParams, controller, server/VPJ producers) will land in
follow-up PRs.
There was a problem hiding this comment.
Pull request overview
This PR stages a new forward-compatible storageMode field in the next (currently pinned/inactive) protocol versions so future work can wire “batch-only dual-write” storage behavior without needing another immediate schema bump.
Changes:
- Added
storageMode(int, default0) toStoreMetaValuev44StoreVersionto persist per-version storage mode in metadata. - Added
storageMode(int, default0) toAdminOperationv99UpdateStoreto propagate the setting via controller admin messages. - Updated
build.gradlecomments to document the staged fields while keeping the versionOverrides pinning unchanged (v43/v98 remain active).
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| services/venice-controller/src/main/resources/avro/AdminOperation/v99/AdminOperation.avsc | Stages storageMode on UpdateStore admin op for future controller wiring. |
| internal/venice-common/src/main/resources/avro/StoreMetaValue/v44/StoreMetaValue.avsc | Stages storageMode on StoreVersion to persist the per-version mode in store metadata. |
| build.gradle | Clarifies the versionOverrides comment to include storageMode among staged fields. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- AdminOperation v99 UpdateStore.storageMode: clarify that this is the store-level default, and that the controller copies it into the new StoreVersion.storageMode at version-creation time only — existing versions are unaffected. (Copilot review comment.) - build.gradle: trim the versionOverrides comment to just name the staged fields; the schema files themselves carry the long-form semantics.
ymuppala
reviewed
May 21, 2026
storageMode (StoreVersion + UpdateStore) now has a single dual-write
value:
0 => INTERNAL (default, Venice-only)
1 => DUAL_WRITE (both Venice and external storage; the specific
leader-vs-VPJ implementation is selected by separate
server/VPJ configuration, not by this enum)
2 => EXTERNAL (external-storage-only; data partition is NoOp)
Add externalStorageReadMode on StoreProperties (v44) and UpdateStore
(v99). This is a store-level-only knob (not per-version) that controls
how clients route reads between Venice and the external storage:
0 => VENICE_ONLY (default, current behavior)
1 => DUAL_MODE_CONSISTENCY_CHECK (read both, verify/report divergence)
2 => DUAL_MODE_EARLY_RETURN (read both, return first; fall back on miss)
3 => EXTERNAL_ONLY (reads served from external storage only)
Still staged: build.gradle keeps v44/v99 pinned to v43/v98 so this is
schema-only with no behavioral impact.
14 tasks
12 tasks
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…orageReadMode + external-table identity ## Problem Statement PR linkedin#2814 staged storageMode (per-version) and externalStorageReadMode (per-store) on StoreMetaValue v44 + AdminOperation v99 to support external-storage-backed stores. The schemas are pinned at v43/v98 in build.gradle so the generated StoreProperties / StoreVersion records do not yet carry the new fields, and no Java surface is wired through Store / Version yet. Downstream consumers (Fast Client metadata refresh, controller, server, VPJ producers) need typed Java accessors to start integrating against. ## Solution Add the Java-side surface for the staged schema fields, plus three follow-on fields for external-table identity / lifecycle that the read path will also need: - New enums (VeniceEnumValue, EnumUtils-backed valueOf): - `StorageMode { INTERNAL, DUAL_WRITE, EXTERNAL }` — mirrors StoreVersion.storageMode - `ExternalStorageReadMode { VENICE_ONLY, DUAL_MODE_CONSISTENCY_CHECK, DUAL_MODE_EARLY_RETURN, EXTERNAL_ONLY }` — mirrors StoreProperties.externalStorageReadMode - `ExternalTableStatus { NOT_CREATED, CREATED, INGESTING, ONLINE, ERROR }` — used by readers to gate external-table queries per version - `Store` interface gains `get/setExternalStorageReadMode`. Implemented on `ZKStore` (transient POJO field with TODO pointing at the schema-pin bump), delegated on `ReadOnlyStore` (setter throws UOE) and `SystemStore` (setter routes through throwUnsupportedOperationException). `ZKStore`'s copy constructor propagates the field; `StoreInfo.fromStore` mirrors it for JSON. - `Version` interface gains `get/setStorageMode`, `get/setExternalDbName`, `get/setExternalTableName`, `get/setExternalTableStatus`. Implemented on `VersionImpl` as transient POJO fields with null-coercion defaults; `cloneVersion` propagates them. `ReadOnlyVersion` (inner class of `ReadOnlyStore`) delegates getters and throws UOE on setters. All four fields are held as transient Java members rather than reading from the Avro record because the StoreMetaValue pin is still v43; the TODOs call out the transition path once the pin advances. ### Code changes - [ ] Added new code behind a config. (no — schema-pinned fields surfaced as Java POJOs only; no runtime behavior change) - [ ] Introduced new log lines. (no) ### Concurrency-Specific Checks - [x] No race conditions: new fields are plain instance state on Store/Version implementations, accessed under the same synchronization invariants as the existing fields they sit alongside. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections; only enum singletons and primitive / String fields. - [x] No new threading. ## How was this PR tested? - [x] New unit tests added: `ExternalStorageReadModeTest`, `StorageModeTest`, `ExternalTableStatusTest` (VeniceEnumValueTest-based int-mapping coverage); new test methods in `TestStoreInfo`, `TestZKStore`, `TestVersion` for defaults / round-trip / clone preservation / null coercion. - [x] Modified or extended existing tests: yes (TestStoreInfo / TestZKStore / TestVersion). - [x] Verified backward compatibility: yes — Avro schema pin unchanged (still v43), so on-the-wire format is unchanged. New methods are additions on the Store / Version interfaces; existing callers compile against the new shape, and StoreInfo JSON gains the new field with a default-value default. ## Does this PR introduce any user-facing or breaking changes? - [x] No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…orageReadMode ## Problem Statement PR linkedin#2814 staged `storageMode` (per-version) and `externalStorageReadMode` (per-store) on StoreMetaValue v44 + AdminOperation v99 to support external-storage-backed stores. The schemas are pinned at v43/v98 in `build.gradle` so the generated StoreProperties / StoreVersion records do not yet carry the new fields, and no Java surface is wired through `Store` / `Version` yet. Downstream consumers (Fast Client metadata refresh, controller, server, VPJ producers) need typed Java accessors to start integrating against. ## Solution Add the Java-side surface for the two staged schema fields: - New enums (`VeniceEnumValue`, `EnumUtils`-backed `valueOf`): - `StorageMode { INTERNAL, DUAL_WRITE, EXTERNAL }` — mirrors `StoreVersion.storageMode` - `ExternalStorageReadMode { VENICE_ONLY, DUAL_MODE_CONSISTENCY_CHECK, DUAL_MODE_EARLY_RETURN, EXTERNAL_ONLY }` — mirrors `StoreProperties.externalStorageReadMode` - `Store` interface gains `get/setExternalStorageReadMode`. Implemented on `ZKStore` (transient POJO field with TODO pointing at the schema-pin bump), delegated on `ReadOnlyStore` (setter throws UOE) and `SystemStore` (setter routes through `throwUnsupportedOperationException`). `ZKStore`'s copy constructor propagates the field; `StoreInfo.fromStore` mirrors it for JSON. - `Version` interface gains `get/setStorageMode`. Implemented on `VersionImpl` as a transient POJO field with null-coercion default; `cloneVersion` propagates it. `ReadOnlyVersion` (inner class of `ReadOnlyStore`) delegates the getter and throws UOE on the setter. Both fields are held as transient Java members rather than reading from the Avro record because the StoreMetaValue pin is still v43; the TODOs call out the transition path once the pin advances to v44 (follow-up PR). ### Code changes - [ ] Added new code behind a config. (no — schema-pinned fields surfaced as Java POJOs only; no runtime behavior change) - [ ] Introduced new log lines. (no) ### Concurrency-Specific Checks - [x] No race conditions: new fields are plain instance state on Store/Version implementations, accessed under the same synchronization invariants as the existing fields they sit alongside. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections; only enum singletons and a primitive-or-enum field per type. - [x] No new threading. ## How was this PR tested? - [x] New unit tests added: `ExternalStorageReadModeTest`, `StorageModeTest` (`VeniceEnumValueTest`-based int-mapping coverage); new test methods in `TestStoreInfo`, `TestZKStore`, `TestVersion` for defaults / round-trip / clone preservation / null coercion. - [x] Modified or extended existing tests: yes (`TestStoreInfo` / `TestZKStore` / `TestVersion`). - [x] Verified backward compatibility: yes — Avro schema pin unchanged (still v43), so on-the-wire format is unchanged. New methods are additions on the `Store` / `Version` interfaces; existing callers compile against the new shape, and `StoreInfo` JSON gains the new field with a default-value default. ## Does this PR introduce any user-facing or breaking changes? - [x] No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
11 tasks
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
… v45 ## Problem Statement Beyond the storageMode + externalStorageReadMode fields already staged on StoreMetaValue v44 by PR linkedin#2814, the external-storage-backed read path needs two more identity fields that v44 does not yet describe: * Per-store: which external storage database to bind to. The external client is typically constructed against a single database per store, so this is a store-level knob. * Per-version: which table within that database to query for a given Venice version. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. ## Solution Schema-only PR — strictly additive. No Java code is touched. The build.gradle pin stays at v43 / v98, so v45 is staged but inactive (the Avro generator keeps producing Java for v43). A subsequent PR will advance the pin and add the Java accessors. - Stage `StoreMetaValue/v45/StoreMetaValue.avsc`: * Add `externalDbName` (string, default "") on `StoreProperties`. * Add `externalTableName` (string, default "") on `StoreVersion`. ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: yes — v45 is staged-but-not-pinned, so the on-the-wire format is unchanged. New fields have defaults; even once v45 is pinned in a future PR, readers on older schemas will observe the defaults and writers on older schemas will skip the unknown fields. - [x] Modified or extended existing tests: no — the existing `AvroCompatibility` suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding `AvroProtocolDefinition` constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…e + externalStorageReadMode ## Problem Statement PR linkedin#2814 staged StoreMetaValue v44 (storageMode + externalStorageReadMode + targetRegionPromoted) and AdminOperation v99 (UpdateStore.storageMode + externalStorageReadMode), but build.gradle still pins to v43/v98 so the generated StoreProperties / StoreVersion records do not yet carry the new fields. Downstream consumers (Fast Client metadata refresh, controller, server, VPJ producers) need typed Java accessors to start integrating against, and the schema must be active on the wire for those accessors to read/write durably. ## Solution Pin the active schema to v44/v99 and add Java accessors that read/write directly from the now-Avro-backed StoreProperties / StoreVersion records. - `build.gradle`: pin StoreMetaValue to v44 (was v43); pin AdminOperation to v99 (was v98). The generated `StoreProperties` / `StoreVersion` classes now include the v44 fields. - `AvroProtocolDefinition`: bump constants in lockstep so the in-process protocol version matches the compiled schema (`METADATA_SYSTEM_SCHEMA_STORE` 43 -> 44; `ADMIN_OPERATION` 98 -> 99). Required for the runtime guard in `UtilsTest.testGetAllSchemasFromResources` and the schema initializer to operate consistently. Add the Java-side surface backed directly by the Avro records (no transient POJO mirroring) so updates round-trip through ZK / metadata system store serialization: - New enums (`VeniceEnumValue`, `EnumUtils`-backed `valueOf`): * `StorageMode { INTERNAL, DUAL_WRITE, EXTERNAL }` — mirrors `StoreVersion.storageMode`. * `ExternalStorageReadMode { VENICE_ONLY, DUAL_MODE_CONSISTENCY_CHECK, DUAL_MODE_EARLY_RETURN, EXTERNAL_ONLY }` — mirrors `StoreProperties.externalStorageReadMode`. - `Store` interface gains `get/setExternalStorageReadMode`. `ZKStore` reads and writes directly from the Avro-generated `storeProperties.externalStorageReadMode` field (int <-> enum). `ReadOnlyStore` delegates the getter and throws UOE on the setter; `SystemStore` does the same via its existing `throwUnsupportedOperationException` helper. `StoreInfo.fromStore` mirrors the field for JSON. - `Version` interface gains `get/setStorageMode`. `VersionImpl` reads and writes directly from `storeVersion.storageMode` (int <-> enum); the existing `cloneVersion` propagates it through the Avro-record copy. `ReadOnlyVersion` delegates the getter and throws UOE on the setter. ### Code changes - [ ] Added new code behind a config. (no.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: new fields are plain Avro-record fields on the existing storeProperties / storeVersion containers, accessed under the same synchronization invariants as the fields they sit alongside. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections; only enum singletons and a primitive-or-enum field per type. - [x] No new threading. ## How was this PR tested? - [x] New unit tests added: `ExternalStorageReadModeTest`, `StorageModeTest` (`VeniceEnumValueTest`-based int-mapping coverage); new test methods in `TestStoreInfo`, `TestZKStore`, `TestVersion` for defaults / round-trip / clone preservation / null coercion; new "persists through Avro data model" tests on `TestZKStore` and `TestVersion` that guard against a regression to transient POJO fields by reading the Avro record directly via `dataModel()`. - [x] Modified or extended existing tests: yes (`TestStoreInfo` / `TestZKStore` / `TestVersion`). - [x] Verified backward compatibility: the v44 schema is forward-compatible with v43 because the new fields have defaults; downgrading a process that writes v44 records to one that reads v43 records works as long as both sides treat unknown fields as ignorable, which Avro does for record fields with defaults. ## Rollout note Pinning the active StoreMetaValue/AdminOperation schema changes what the controller writes to ZK and what controllers exchange via the admin topic. All Venice services must be redeployed at this pin before any caller starts setting `storageMode` / `externalStorageReadMode` to non-default values, otherwise a v43-only service performing a read-modify-write could silently drop the new fields. ## Does this PR introduce any user-facing or breaking changes? - [x] No (defaults preserve existing behavior; new fields are inert until callers opt in). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, and `externalStorageReadMode` on `StoreProperties`, along with `targetRegionPromoted`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98, so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-store: which external storage database to bind to. The external client is typically constructed against a single database per store at construction time, so this is a store-level knob. * Per-version: which table within that database to query for a given Venice version. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44: * Add `externalTableName` (string, default "") on `StoreVersion`. * Add `externalDbName` (string, default "") on `StoreProperties`. - AdminOperation v99 `UpdateStore`: * Add `externalDbName` (string, default "") — direct mirror. * Add `externalTableName` (string, default "") — per-version field also surfaced on `UpdateStore` so operators can override it via admin op for recovery / migration scenarios (e.g. point a store at a pre-existing external table outside the normal push-creates-table lifecycle). ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: yes — the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) will round-trip cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not-pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing AvroCompatibility test suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, and `externalStorageReadMode` on `StoreProperties`, along with `targetRegionPromoted`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98, so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-store: which external storage database to bind to. The external client is typically constructed against a single database per store at construction time, so this is a store-level knob. * Per-version: which table within that database to query for a given Venice version. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44: * Add `externalTableName` (string, default "") on `StoreVersion`. * Add `externalDbName` (string, default "") on `StoreProperties`. - AdminOperation v99 `UpdateStore`: * Add `externalDbName` (string, default "") — direct mirror. * Add `externalTableName` (string, default "") — per-version field also surfaced on `UpdateStore` so operators can override it via admin op for recovery / migration scenarios (e.g. point a store at a pre-existing external table outside the normal push-creates-table lifecycle). ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: yes — the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) will round-trip cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not-pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing AvroCompatibility test suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…n v99 + Java accessors ## Problem Statement Predecessor PRs (linkedin#2806, linkedin#2814, linkedin#2820) staged StoreMetaValue v44 and AdminOperation v99 with five external-storage-related fields (`targetRegionPromoted` and `storageMode` on `StoreVersion`; `externalStorageReadMode`, `externalDbName` on `StoreProperties`; plus `externalTableName` on `StoreVersion` and `externalDbName` / `externalTableName` on `UpdateStore`). build.gradle's `versionOverrides` list still forces the Avro compiler to use v43 / v98 instead of letting it pick the highest-numeric directory, so v44 / v99 are not yet wire-real. Downstream consumers (Fast Client metadata refresh, controller, server, VPJ producers) need typed Java accessors to start integrating against. ## Solution Drop the `versionOverrides` entries so the Avro compiler picks the latest schema directories (v44 / v99) naturally — that's exactly the remove-the-override step the comment above `versionOverrides` describes ("remove the override in a follow-up PR when actually using the new protocol"). Bump `AvroProtocolDefinition` constants in lockstep so the in-process protocol version matches the compiled schema. Add Java accessors that read/write directly from the now-Avro-backed `StoreProperties` / `StoreVersion` records (no transient POJO mirroring), so updates round-trip through ZK / metadata-system-store serialization. - `build.gradle`: remove the `versionOverrides` entries for StoreMetaValue and AdminOperation. The Avro generator now selects v44 / v99 by the default max-numeric rule. - `AvroProtocolDefinition`: bump constants so the in-process protocol version matches the compiled schema (`METADATA_SYSTEM_SCHEMA_STORE` 43 -> 44; `ADMIN_OPERATION` 98 -> 99). Required for the runtime guard in `UtilsTest.testGetAllSchemasFromResources` and the system schema initializer to operate consistently. - New enums (`VeniceEnumValue`, `EnumUtils`-backed `valueOf`): * `StorageMode { INTERNAL, DUAL_WRITE, EXTERNAL }` — mirrors `StoreVersion.storageMode`. * `ExternalStorageReadMode { VENICE_ONLY, DUAL_MODE_CONSISTENCY_CHECK, DUAL_MODE_EARLY_RETURN, EXTERNAL_ONLY }` — mirrors `StoreProperties.externalStorageReadMode`. - `Store` interface gains `get/setExternalStorageReadMode`. `ZKStore` reads/writes directly from `storeProperties.externalStorageReadMode` (int <-> enum). `ReadOnlyStore` delegates the getter and throws UOE on the setter; `SystemStore` does the same via its existing `throwUnsupportedOperationException` helper. `StoreInfo.fromStore` mirrors the field for JSON. - `Version` interface gains `get/setStorageMode`. `VersionImpl` reads/writes directly from `storeVersion.storageMode` (int <-> enum); the existing `cloneVersion` propagates it through the Avro-record copy. `ReadOnlyVersion` delegates the getter and throws UOE on the setter. ### Code changes - [ ] Added new code behind a config. (no.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: new fields are plain Avro-record fields on the existing storeProperties / storeVersion containers, accessed under the same synchronization invariants as the fields they sit alongside. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections; only enum singletons and a primitive-or-enum field per type. - [x] No new threading. ## How was this PR tested? - [x] New unit tests added: `ExternalStorageReadModeTest`, `StorageModeTest` (`VeniceEnumValueTest`-based int-mapping coverage); new test methods in `TestStoreInfo`, `TestZKStore`, `TestVersion` for defaults / round-trip / clone preservation / null coercion; new "persists through Avro data model" tests on `TestZKStore` and `TestVersion` that guard against a regression to transient POJO fields by reading the Avro record directly via `dataModel()`. - [x] Modified or extended existing tests: yes (`TestStoreInfo` / `TestZKStore` / `TestVersion`). - [x] Verified backward compatibility: the v44 schema is forward-compatible with v43 because the new fields have defaults; downgrading a process that writes v44 records to one that reads v43 records works as long as both sides treat unknown fields as ignorable, which Avro does for record fields with defaults. ## Rollout note Activating the v44 / v99 schemas changes what the controller writes to ZK and what controllers exchange via the admin topic. All Venice services must be redeployed at this version before any caller starts setting `storageMode` / `externalStorageReadMode` / `externalDbName` / `externalTableName` to non-default values, otherwise a v43-only service performing a read-modify-write could silently drop the new fields. ## Does this PR introduce any user-facing or breaking changes? - [x] No (defaults preserve existing behavior; new fields are inert until callers opt in). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…n v99 + Java accessors ## Problem Statement Predecessor PRs (linkedin#2806, linkedin#2814, linkedin#2820) staged StoreMetaValue v44 and AdminOperation v99 with five external-storage-related fields: `targetRegionPromoted`, `storageMode`, `externalTableName` on `StoreVersion` and `externalStorageReadMode`, `externalDbName` on `StoreProperties`; plus `storageMode`, `externalStorageReadMode`, `externalDbName`, `externalTableName` on `UpdateStore` (admin op). build.gradle's `versionOverrides` list still forces the Avro compiler to use v43 / v98 instead of letting it pick the highest-numeric directory, so v44 / v99 are not yet wire-real. Downstream consumers (Fast Client metadata refresh, controller, server, VPJ producers) need typed Java accessors to start integrating against. ## Solution Drop the `versionOverrides` entries so the Avro compiler picks the latest schema directories (v44 / v99) naturally — that's exactly the remove-the-override step the comment above `versionOverrides` describes ("remove the override in a follow-up PR when actually using the new protocol"). Bump `AvroProtocolDefinition` constants in lockstep so the in-process protocol version matches the compiled schema. Add Java accessors that read/write directly from the now-Avro-backed `StoreProperties` / `StoreVersion` records (no transient POJO mirroring), so updates round-trip through ZK / metadata-system-store serialization. - `build.gradle`: remove the `versionOverrides` entries for StoreMetaValue and AdminOperation. The Avro generator now selects v44 / v99 by the default max-numeric rule. - `AvroProtocolDefinition`: bump constants (`METADATA_SYSTEM_SCHEMA_STORE` 43 -> 44; `ADMIN_OPERATION` 98 -> 99) so the in-process protocol version matches the compiled schema. Required for the runtime guard in `UtilsTest.testGetAllSchemasFromResources` and the system schema initializer to operate consistently. - New enums (`VeniceEnumValue`, `EnumUtils`-backed `valueOf`): * `StorageMode { INTERNAL, DUAL_WRITE, EXTERNAL }` — mirrors `StoreVersion.storageMode`. * `ExternalStorageReadMode { VENICE_ONLY, DUAL_MODE_CONSISTENCY_CHECK, DUAL_MODE_EARLY_RETURN, EXTERNAL_ONLY }` — mirrors `StoreProperties.externalStorageReadMode`. - Store-level Java accessors backed by `StoreProperties` Avro record: * `Store.get/setExternalStorageReadMode` → `storeProperties.externalStorageReadMode`. * `Store.get/setExternalDbName` → `storeProperties.externalDbName`. Implemented on `ZKStore` (int <-> enum where applicable, with null-coercion on the string). `ReadOnlyStore` delegates the getter and throws UOE on the setter; `SystemStore` does the same via its existing `throwUnsupportedOperationException` helper. `StoreInfo.fromStore` mirrors both fields for JSON. - Version-level Java accessors backed by `StoreVersion` Avro record: * `Version.get/setStorageMode` → `storeVersion.storageMode`. * `Version.get/setExternalTableName` → `storeVersion.externalTableName`. Implemented on `VersionImpl`; `cloneVersion` propagates both through the Avro-record copy. `ReadOnlyVersion` delegates getters and throws UOE on setters. ### Code changes - [ ] Added new code behind a config. (no.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: new fields are plain Avro-record fields on the existing storeProperties / storeVersion containers, accessed under the same synchronization invariants as the fields they sit alongside. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections; only enum singletons and a primitive-or-enum-or-String field per type. - [x] No new threading. ## How was this PR tested? - [x] New unit tests added: `ExternalStorageReadModeTest`, `StorageModeTest` (`VeniceEnumValueTest`-based int-mapping coverage); new test methods in `TestStoreInfo`, `TestZKStore`, `TestVersion` for defaults / round-trip / clone preservation / null coercion for both the enum fields (storageMode, externalStorageReadMode) and the string fields (externalDbName, externalTableName); plus "persists through Avro data model" tests on `TestZKStore` and `TestVersion` for all four fields that read the Avro record directly via `dataModel()` to guard against a regression to transient POJO mirroring. - [x] Modified or extended existing tests: yes (`TestStoreInfo` / `TestZKStore` / `TestVersion`). - [x] Verified backward compatibility: the v44 schema is forward-compatible with v43 because the new fields have defaults; downgrading a process that writes v44 records to one that reads v43 records works as long as both sides treat unknown fields as ignorable, which Avro does for record fields with defaults. ## Rollout note Activating the v44 / v99 schemas changes what the controller writes to ZK and what controllers exchange via the admin topic. All Venice services must be redeployed at this version before any caller starts setting `storageMode` / `externalStorageReadMode` / `externalDbName` / `externalTableName` to non-default values, otherwise a v43-only service performing a read-modify-write could silently drop the new fields. ## Does this PR introduce any user-facing or breaking changes? - [x] No (defaults preserve existing behavior; new fields are inert until callers opt in). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…n v99 + Java accessors ## Problem Statement Predecessor PRs (linkedin#2806, linkedin#2814, linkedin#2820) staged StoreMetaValue v44 and AdminOperation v99 with five external-storage-related fields: `targetRegionPromoted`, `storageMode`, `externalTableName` on `StoreVersion` and `externalStorageReadMode`, `externalDbName` on `StoreProperties`; plus `storageMode`, `externalStorageReadMode`, `externalDbName`, `externalTableName` on `UpdateStore` (admin op). build.gradle's `versionOverrides` list still forces the Avro compiler to use v43 / v98 instead of letting it pick the highest-numeric directory, so v44 / v99 are not yet wire-real. Downstream consumers (Fast Client metadata refresh, controller, server, VPJ producers) need typed Java accessors to start integrating against. ## Solution Drop the `versionOverrides` entries so the Avro compiler picks the latest schema directories (v44 / v99) naturally — that's exactly the remove-the-override step the comment above `versionOverrides` describes ("remove the override in a follow-up PR when actually using the new protocol"). Bump `AvroProtocolDefinition` constants in lockstep so the in-process protocol version matches the compiled schema. Add Java accessors that read/write directly from the now-Avro-backed `StoreProperties` / `StoreVersion` records (no transient POJO mirroring), so updates round-trip through ZK / metadata-system-store serialization. - `build.gradle`: remove the `versionOverrides` entries for StoreMetaValue and AdminOperation. The Avro generator now selects v44 / v99 by the default max-numeric rule. - `AvroProtocolDefinition`: bump constants (`METADATA_SYSTEM_SCHEMA_STORE` 43 -> 44; `ADMIN_OPERATION` 98 -> 99) so the in-process protocol version matches the compiled schema. Required for the runtime guard in `UtilsTest.testGetAllSchemasFromResources` and the system schema initializer to operate consistently. - New enums (`VeniceEnumValue`, `EnumUtils`-backed `valueOf`): * `StorageMode { INTERNAL, DUAL_WRITE, EXTERNAL }` — mirrors `StoreVersion.storageMode`. * `ExternalStorageReadMode { VENICE_ONLY, DUAL_MODE_CONSISTENCY_CHECK, DUAL_MODE_EARLY_RETURN, EXTERNAL_ONLY }` — mirrors `StoreProperties.externalStorageReadMode`. - Store-level Java accessors backed by `StoreProperties` Avro record: * `Store.get/setExternalStorageReadMode` → `storeProperties.externalStorageReadMode`. * `Store.get/setExternalDbName` → `storeProperties.externalDbName`. Implemented on `ZKStore` (int <-> enum where applicable, with null-coercion on the string). `ReadOnlyStore` delegates the getter and throws UOE on the setter; `SystemStore` does the same via its existing `throwUnsupportedOperationException` helper. `StoreInfo.fromStore` mirrors both fields for JSON. - Version-level Java accessors backed by `StoreVersion` Avro record: * `Version.get/setStorageMode` → `storeVersion.storageMode`. * `Version.get/setExternalTableName` → `storeVersion.externalTableName`. Implemented on `VersionImpl`; `cloneVersion` propagates both through the Avro-record copy. `ReadOnlyVersion` delegates getters and throws UOE on setters. ### Code changes - [ ] Added new code behind a config. (no.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: new fields are plain Avro-record fields on the existing storeProperties / storeVersion containers, accessed under the same synchronization invariants as the fields they sit alongside. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections; only enum singletons and a primitive-or-enum-or-String field per type. - [x] No new threading. ## How was this PR tested? - [x] New unit tests added: `ExternalStorageReadModeTest`, `StorageModeTest` (`VeniceEnumValueTest`-based int-mapping coverage); new test methods in `TestStoreInfo`, `TestZKStore`, `TestVersion` for defaults / round-trip / clone preservation / null coercion for both the enum fields (storageMode, externalStorageReadMode) and the string fields (externalDbName, externalTableName); plus "persists through Avro data model" tests on `TestZKStore` and `TestVersion` for all four fields that read the Avro record directly via `dataModel()` to guard against a regression to transient POJO mirroring. - [x] Modified or extended existing tests: yes (`TestStoreInfo` / `TestZKStore` / `TestVersion`). - [x] Verified backward compatibility: the v44 schema is forward-compatible with v43 because the new fields have defaults; downgrading a process that writes v44 records to one that reads v43 records works as long as both sides treat unknown fields as ignorable, which Avro does for record fields with defaults. ## Rollout note Activating the v44 / v99 schemas changes what the controller writes to ZK and what controllers exchange via the admin topic. All Venice services must be redeployed at this version before any caller starts setting `storageMode` / `externalStorageReadMode` / `externalDbName` / `externalTableName` to non-default values, otherwise a v43-only service performing a read-modify-write could silently drop the new fields. ## Does this PR introduce any user-facing or breaking changes? - [x] No (defaults preserve existing behavior; new fields are inert until callers opt in). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, `externalStorageReadMode` on `StoreProperties`, and `targetRegionPromoted` on `StoreVersion`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98 so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-version external database — which external storage database holds this version's data. Captured per-version (alongside externalTableName) so that successive pushes can address distinct external databases when needed (e.g. cross-database migration without store migration). * Per-version external table — which table within that database holds this version's data. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44 `StoreVersion` (both fields per-version, adjacent to the existing `storageMode`): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). - AdminOperation v99 `UpdateStore` (operator-override surface for recovery/migration; successful application copies into the corresponding StoreVersion fields): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). The "NOT_SPECIFIED" sentinel matches the convention established by the `blobTransferInServerEnabled` and `blobDbEnabled` fields already present on UpdateStore — non-null default that makes the field safe to serialize even when not explicitly initialized at construction time. ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) round-trips cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not- pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing `AvroCompatibility` suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version (all four external-storage-identity fields now per-version on Version). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, `externalStorageReadMode` on `StoreProperties`, and `targetRegionPromoted` on `StoreVersion`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98 so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-version external database — which external storage database holds this version's data. Captured per-version (alongside externalTableName) so that successive pushes can address distinct external databases when needed (e.g. cross-database migration without store migration). * Per-version external table — which table within that database holds this version's data. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44 `StoreVersion` (both fields per-version, adjacent to the existing `storageMode`): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). - AdminOperation v99 `UpdateStore` (operator-override surface for recovery/migration; successful application copies into the corresponding StoreVersion fields): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). The "NOT_SPECIFIED" sentinel matches the convention established by the `blobTransferInServerEnabled` and `blobDbEnabled` fields already present on UpdateStore — non-null default that makes the field safe to serialize even when not explicitly initialized at construction time. ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) round-trips cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not- pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing `AvroCompatibility` suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version (all four external-storage-identity fields now per-version on Version). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, `externalStorageReadMode` on `StoreProperties`, and `targetRegionPromoted` on `StoreVersion`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98 so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-version external database — which external storage database holds this version's data. Captured per-version (alongside externalTableName) so that successive pushes can address distinct external databases when needed (e.g. cross-database migration without store migration). * Per-version external table — which table within that database holds this version's data. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44 `StoreVersion` (both fields per-version, adjacent to the existing `storageMode`): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). - AdminOperation v99 `UpdateStore` (operator-override surface for recovery/migration; successful application copies into the corresponding StoreVersion fields): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). The "NOT_SPECIFIED" sentinel matches the convention established by the `blobTransferInServerEnabled` and `blobDbEnabled` fields already present on UpdateStore — non-null default that makes the field safe to serialize even when not explicitly initialized at construction time. ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) round-trips cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not- pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing `AvroCompatibility` suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version (all four external-storage-identity fields now per-version on Version). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, `externalStorageReadMode` on `StoreProperties`, and `targetRegionPromoted` on `StoreVersion`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98 so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-version external database — which external storage database holds this version's data. Captured per-version (alongside externalTableName) so that successive pushes can address distinct external databases when needed (e.g. cross-database migration without store migration). * Per-version external table — which table within that database holds this version's data. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44 `StoreVersion` (both fields per-version, adjacent to the existing `storageMode`): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). - AdminOperation v99 `UpdateStore` (operator-override surface for recovery/migration; successful application copies into the corresponding StoreVersion fields): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). The "NOT_SPECIFIED" sentinel matches the convention established by the `blobTransferInServerEnabled` and `blobDbEnabled` fields already present on UpdateStore — non-null default that makes the field safe to serialize even when not explicitly initialized at construction time. ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) round-trips cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not- pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing `AvroCompatibility` suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version (all four external-storage-identity fields now per-version on Version). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, `externalStorageReadMode` on `StoreProperties`, and `targetRegionPromoted` on `StoreVersion`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98 so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-version external database — which external storage database holds this version's data. Captured per-version (alongside externalTableName) so that successive pushes can address distinct external databases when needed (e.g. cross-database migration without store migration). * Per-version external table — which table within that database holds this version's data. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44 `StoreVersion` (both fields per-version, adjacent to the existing `storageMode`): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). - AdminOperation v99 `UpdateStore` (operator-override surface for recovery/migration; successful application copies into the corresponding StoreVersion fields): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). The "NOT_SPECIFIED" sentinel matches the convention established by the `blobTransferInServerEnabled` and `blobDbEnabled` fields already present on UpdateStore — non-null default that makes the field safe to serialize even when not explicitly initialized at construction time. ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) round-trips cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not- pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing `AvroCompatibility` suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version (all four external-storage-identity fields now per-version on Version). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
11 tasks
11 tasks
This was referenced May 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Statement
The "Venice Batch-Only Store Dual-Write" design (separate doc) introduces a per-store-version knob that selects where data is persisted: Venice-only (today's behavior), dual-written to an external KV store, or external-storage-only. Once dual-write is enabled, client reads also need a routing knob to validate dual-write correctness, early-return from the faster side, or cut over to external-only reads. Both fields need to be forward-compatible additions to the schemas before any of the Java/server/VPJ/client wiring can be built.
PR #2806 already staged
StoreMetaValue v44andAdminOperation v99for the not-yet-activetargetRegionPromotedflag and pinned them viaversionOverrides. This PR piggybacks on those same not-yet-active versions to stage two more fields, avoiding the cost of a new protocol version for each not-yet-launched feature.Solution
Add two forward-compatible fields:
storageMode(per store version) —int, default0Added to:
StoreVersionininternal/venice-common/src/main/resources/avro/StoreMetaValue/v44/StoreMetaValue.avsc— per-version state.UpdateStoreinservices/venice-controller/src/main/resources/avro/AdminOperation/v99/AdminOperation.avsc— store-level default; the controller copies it intoStoreVersion.storageModeat new-version creation only. Existing versions are unaffected.INTERNALDUAL_WRITEEXTERNALexternalStorageReadMode(store-level only) —int, default0Added to:
StorePropertiesinStoreMetaValue v44— applies to the store as a whole, not per-version.UpdateStoreinAdminOperation v99— the admin-op setter.VENICE_ONLYDUAL_MODE_CONSISTENCY_CHECKDUAL_MODE_EARLY_RETURNEXTERNAL_ONLYbuild.gradlekeeps the existingversionOverridespinning toStoreMetaValue v43/AdminOperation v98, so v44/v99 remain staged-but-inactive — this change is schema-only and has no runtime effect until follow-up PRs land the Java wiring.Code changes
build.gradleversionOverrides until the Java/server/VPJ/client wiring lands.)Concurrency-Specific Checks
N/A — schema-only change, no Java logic touched.
How was this PR tested?
0(INTERNAL/VENICE_ONLY), which preserves today's behavior; Avro forward-compat is preserved by appending at the end offields[]../gradlew :internal:venice-common:compileJava— Avro codegen + compile pass../gradlew :services:venice-controller:compileJava— Avro codegen + compile pass.Does this PR introduce any user-facing or breaking changes?
build.gradle, so v44/v99 (and the new fields) are not yet in the active protocol. No behavior change in any code path.