[protocol] Stage StoreMetaValue v44 and AdminOperation v99 with targetRegionPromoted#2806
Merged
Merged
Conversation
targetRegionPromoted Adds a targetRegionPromoted: boolean (default false) field to: - StoreVersion record in StoreMetaValue/v44 (extends v43) - UpdateStore record in AdminOperation/v99 (extends v98) This field will be written by the parent controller when the target region promotes a version during a target-region push with deferred swap. Non-target DaVinci clients observe this field via parent-to-child store-state replication to know when to resume paused ingestion (paused-SIT design, Phase 1). Pins the Avro code generator to the current active versions (v43 / v98) so the new schemas are staged but not yet active. The Java wiring, AvroProtocolDefinition bump, and pin removal follow in a separate PR.
There was a problem hiding this comment.
Pull request overview
Stages new Avro schema versions to introduce a targetRegionPromoted boolean signal used by paused sequential ingestion (SIT) flows, while pinning codegen to the currently-active schema versions so runtime behavior does not change yet.
Changes:
- Added
targetRegionPromoted(defaultfalse) toStoreMetaValueStoreVersioninv44. - Added
targetRegionPromoted(defaultfalse) toAdminOperationUpdateStoreinv99. - Updated Gradle Avro codegen pin/override list to keep generation on
StoreMetaValue/v43andAdminOperation/v98.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| services/venice-controller/src/main/resources/avro/AdminOperation/v99/AdminOperation.avsc | Stages AdminOperation v99 adding targetRegionPromoted to UpdateStore. |
| internal/venice-common/src/main/resources/avro/StoreMetaValue/v44/StoreMetaValue.avsc | Stages StoreMetaValue v44 adding targetRegionPromoted to StoreVersion. |
| build.gradle | Pins Avro codegen to StoreMetaValue/v43 and AdminOperation/v98 while schemas are staged. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
sixpluszero
approved these changes
May 20, 2026
misyel
changed the title
5 tasks
7 tasks
14 tasks
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, and `externalStorageReadMode` on `StoreProperties`, along with `targetRegionPromoted`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98, so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-store: which external storage database to bind to. The external client is typically constructed against a single database per store at construction time, so this is a store-level knob. * Per-version: which table within that database to query for a given Venice version. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44: * Add `externalTableName` (string, default "") on `StoreVersion`. * Add `externalDbName` (string, default "") on `StoreProperties`. - AdminOperation v99 `UpdateStore`: * Add `externalDbName` (string, default "") — direct mirror. * Add `externalTableName` (string, default "") — per-version field also surfaced on `UpdateStore` so operators can override it via admin op for recovery / migration scenarios (e.g. point a store at a pre-existing external table outside the normal push-creates-table lifecycle). ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: yes — the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) will round-trip cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not-pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing AvroCompatibility test suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
11 tasks
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, and `externalStorageReadMode` on `StoreProperties`, along with `targetRegionPromoted`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98, so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-store: which external storage database to bind to. The external client is typically constructed against a single database per store at construction time, so this is a store-level knob. * Per-version: which table within that database to query for a given Venice version. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44: * Add `externalTableName` (string, default "") on `StoreVersion`. * Add `externalDbName` (string, default "") on `StoreProperties`. - AdminOperation v99 `UpdateStore`: * Add `externalDbName` (string, default "") — direct mirror. * Add `externalTableName` (string, default "") — per-version field also surfaced on `UpdateStore` so operators can override it via admin op for recovery / migration scenarios (e.g. point a store at a pre-existing external table outside the normal push-creates-table lifecycle). ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: yes — the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) will round-trip cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not-pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing AvroCompatibility test suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…n v99 + Java accessors ## Problem Statement Predecessor PRs (linkedin#2806, linkedin#2814, linkedin#2820) staged StoreMetaValue v44 and AdminOperation v99 with five external-storage-related fields (`targetRegionPromoted` and `storageMode` on `StoreVersion`; `externalStorageReadMode`, `externalDbName` on `StoreProperties`; plus `externalTableName` on `StoreVersion` and `externalDbName` / `externalTableName` on `UpdateStore`). build.gradle's `versionOverrides` list still forces the Avro compiler to use v43 / v98 instead of letting it pick the highest-numeric directory, so v44 / v99 are not yet wire-real. Downstream consumers (Fast Client metadata refresh, controller, server, VPJ producers) need typed Java accessors to start integrating against. ## Solution Drop the `versionOverrides` entries so the Avro compiler picks the latest schema directories (v44 / v99) naturally — that's exactly the remove-the-override step the comment above `versionOverrides` describes ("remove the override in a follow-up PR when actually using the new protocol"). Bump `AvroProtocolDefinition` constants in lockstep so the in-process protocol version matches the compiled schema. Add Java accessors that read/write directly from the now-Avro-backed `StoreProperties` / `StoreVersion` records (no transient POJO mirroring), so updates round-trip through ZK / metadata-system-store serialization. - `build.gradle`: remove the `versionOverrides` entries for StoreMetaValue and AdminOperation. The Avro generator now selects v44 / v99 by the default max-numeric rule. - `AvroProtocolDefinition`: bump constants so the in-process protocol version matches the compiled schema (`METADATA_SYSTEM_SCHEMA_STORE` 43 -> 44; `ADMIN_OPERATION` 98 -> 99). Required for the runtime guard in `UtilsTest.testGetAllSchemasFromResources` and the system schema initializer to operate consistently. - New enums (`VeniceEnumValue`, `EnumUtils`-backed `valueOf`): * `StorageMode { INTERNAL, DUAL_WRITE, EXTERNAL }` — mirrors `StoreVersion.storageMode`. * `ExternalStorageReadMode { VENICE_ONLY, DUAL_MODE_CONSISTENCY_CHECK, DUAL_MODE_EARLY_RETURN, EXTERNAL_ONLY }` — mirrors `StoreProperties.externalStorageReadMode`. - `Store` interface gains `get/setExternalStorageReadMode`. `ZKStore` reads/writes directly from `storeProperties.externalStorageReadMode` (int <-> enum). `ReadOnlyStore` delegates the getter and throws UOE on the setter; `SystemStore` does the same via its existing `throwUnsupportedOperationException` helper. `StoreInfo.fromStore` mirrors the field for JSON. - `Version` interface gains `get/setStorageMode`. `VersionImpl` reads/writes directly from `storeVersion.storageMode` (int <-> enum); the existing `cloneVersion` propagates it through the Avro-record copy. `ReadOnlyVersion` delegates the getter and throws UOE on the setter. ### Code changes - [ ] Added new code behind a config. (no.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: new fields are plain Avro-record fields on the existing storeProperties / storeVersion containers, accessed under the same synchronization invariants as the fields they sit alongside. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections; only enum singletons and a primitive-or-enum field per type. - [x] No new threading. ## How was this PR tested? - [x] New unit tests added: `ExternalStorageReadModeTest`, `StorageModeTest` (`VeniceEnumValueTest`-based int-mapping coverage); new test methods in `TestStoreInfo`, `TestZKStore`, `TestVersion` for defaults / round-trip / clone preservation / null coercion; new "persists through Avro data model" tests on `TestZKStore` and `TestVersion` that guard against a regression to transient POJO fields by reading the Avro record directly via `dataModel()`. - [x] Modified or extended existing tests: yes (`TestStoreInfo` / `TestZKStore` / `TestVersion`). - [x] Verified backward compatibility: the v44 schema is forward-compatible with v43 because the new fields have defaults; downgrading a process that writes v44 records to one that reads v43 records works as long as both sides treat unknown fields as ignorable, which Avro does for record fields with defaults. ## Rollout note Activating the v44 / v99 schemas changes what the controller writes to ZK and what controllers exchange via the admin topic. All Venice services must be redeployed at this version before any caller starts setting `storageMode` / `externalStorageReadMode` / `externalDbName` / `externalTableName` to non-default values, otherwise a v43-only service performing a read-modify-write could silently drop the new fields. ## Does this PR introduce any user-facing or breaking changes? - [x] No (defaults preserve existing behavior; new fields are inert until callers opt in). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
12 tasks
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…n v99 + Java accessors ## Problem Statement Predecessor PRs (linkedin#2806, linkedin#2814, linkedin#2820) staged StoreMetaValue v44 and AdminOperation v99 with five external-storage-related fields: `targetRegionPromoted`, `storageMode`, `externalTableName` on `StoreVersion` and `externalStorageReadMode`, `externalDbName` on `StoreProperties`; plus `storageMode`, `externalStorageReadMode`, `externalDbName`, `externalTableName` on `UpdateStore` (admin op). build.gradle's `versionOverrides` list still forces the Avro compiler to use v43 / v98 instead of letting it pick the highest-numeric directory, so v44 / v99 are not yet wire-real. Downstream consumers (Fast Client metadata refresh, controller, server, VPJ producers) need typed Java accessors to start integrating against. ## Solution Drop the `versionOverrides` entries so the Avro compiler picks the latest schema directories (v44 / v99) naturally — that's exactly the remove-the-override step the comment above `versionOverrides` describes ("remove the override in a follow-up PR when actually using the new protocol"). Bump `AvroProtocolDefinition` constants in lockstep so the in-process protocol version matches the compiled schema. Add Java accessors that read/write directly from the now-Avro-backed `StoreProperties` / `StoreVersion` records (no transient POJO mirroring), so updates round-trip through ZK / metadata-system-store serialization. - `build.gradle`: remove the `versionOverrides` entries for StoreMetaValue and AdminOperation. The Avro generator now selects v44 / v99 by the default max-numeric rule. - `AvroProtocolDefinition`: bump constants (`METADATA_SYSTEM_SCHEMA_STORE` 43 -> 44; `ADMIN_OPERATION` 98 -> 99) so the in-process protocol version matches the compiled schema. Required for the runtime guard in `UtilsTest.testGetAllSchemasFromResources` and the system schema initializer to operate consistently. - New enums (`VeniceEnumValue`, `EnumUtils`-backed `valueOf`): * `StorageMode { INTERNAL, DUAL_WRITE, EXTERNAL }` — mirrors `StoreVersion.storageMode`. * `ExternalStorageReadMode { VENICE_ONLY, DUAL_MODE_CONSISTENCY_CHECK, DUAL_MODE_EARLY_RETURN, EXTERNAL_ONLY }` — mirrors `StoreProperties.externalStorageReadMode`. - Store-level Java accessors backed by `StoreProperties` Avro record: * `Store.get/setExternalStorageReadMode` → `storeProperties.externalStorageReadMode`. * `Store.get/setExternalDbName` → `storeProperties.externalDbName`. Implemented on `ZKStore` (int <-> enum where applicable, with null-coercion on the string). `ReadOnlyStore` delegates the getter and throws UOE on the setter; `SystemStore` does the same via its existing `throwUnsupportedOperationException` helper. `StoreInfo.fromStore` mirrors both fields for JSON. - Version-level Java accessors backed by `StoreVersion` Avro record: * `Version.get/setStorageMode` → `storeVersion.storageMode`. * `Version.get/setExternalTableName` → `storeVersion.externalTableName`. Implemented on `VersionImpl`; `cloneVersion` propagates both through the Avro-record copy. `ReadOnlyVersion` delegates getters and throws UOE on setters. ### Code changes - [ ] Added new code behind a config. (no.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: new fields are plain Avro-record fields on the existing storeProperties / storeVersion containers, accessed under the same synchronization invariants as the fields they sit alongside. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections; only enum singletons and a primitive-or-enum-or-String field per type. - [x] No new threading. ## How was this PR tested? - [x] New unit tests added: `ExternalStorageReadModeTest`, `StorageModeTest` (`VeniceEnumValueTest`-based int-mapping coverage); new test methods in `TestStoreInfo`, `TestZKStore`, `TestVersion` for defaults / round-trip / clone preservation / null coercion for both the enum fields (storageMode, externalStorageReadMode) and the string fields (externalDbName, externalTableName); plus "persists through Avro data model" tests on `TestZKStore` and `TestVersion` for all four fields that read the Avro record directly via `dataModel()` to guard against a regression to transient POJO mirroring. - [x] Modified or extended existing tests: yes (`TestStoreInfo` / `TestZKStore` / `TestVersion`). - [x] Verified backward compatibility: the v44 schema is forward-compatible with v43 because the new fields have defaults; downgrading a process that writes v44 records to one that reads v43 records works as long as both sides treat unknown fields as ignorable, which Avro does for record fields with defaults. ## Rollout note Activating the v44 / v99 schemas changes what the controller writes to ZK and what controllers exchange via the admin topic. All Venice services must be redeployed at this version before any caller starts setting `storageMode` / `externalStorageReadMode` / `externalDbName` / `externalTableName` to non-default values, otherwise a v43-only service performing a read-modify-write could silently drop the new fields. ## Does this PR introduce any user-facing or breaking changes? - [x] No (defaults preserve existing behavior; new fields are inert until callers opt in). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…n v99 + Java accessors ## Problem Statement Predecessor PRs (linkedin#2806, linkedin#2814, linkedin#2820) staged StoreMetaValue v44 and AdminOperation v99 with five external-storage-related fields: `targetRegionPromoted`, `storageMode`, `externalTableName` on `StoreVersion` and `externalStorageReadMode`, `externalDbName` on `StoreProperties`; plus `storageMode`, `externalStorageReadMode`, `externalDbName`, `externalTableName` on `UpdateStore` (admin op). build.gradle's `versionOverrides` list still forces the Avro compiler to use v43 / v98 instead of letting it pick the highest-numeric directory, so v44 / v99 are not yet wire-real. Downstream consumers (Fast Client metadata refresh, controller, server, VPJ producers) need typed Java accessors to start integrating against. ## Solution Drop the `versionOverrides` entries so the Avro compiler picks the latest schema directories (v44 / v99) naturally — that's exactly the remove-the-override step the comment above `versionOverrides` describes ("remove the override in a follow-up PR when actually using the new protocol"). Bump `AvroProtocolDefinition` constants in lockstep so the in-process protocol version matches the compiled schema. Add Java accessors that read/write directly from the now-Avro-backed `StoreProperties` / `StoreVersion` records (no transient POJO mirroring), so updates round-trip through ZK / metadata-system-store serialization. - `build.gradle`: remove the `versionOverrides` entries for StoreMetaValue and AdminOperation. The Avro generator now selects v44 / v99 by the default max-numeric rule. - `AvroProtocolDefinition`: bump constants (`METADATA_SYSTEM_SCHEMA_STORE` 43 -> 44; `ADMIN_OPERATION` 98 -> 99) so the in-process protocol version matches the compiled schema. Required for the runtime guard in `UtilsTest.testGetAllSchemasFromResources` and the system schema initializer to operate consistently. - New enums (`VeniceEnumValue`, `EnumUtils`-backed `valueOf`): * `StorageMode { INTERNAL, DUAL_WRITE, EXTERNAL }` — mirrors `StoreVersion.storageMode`. * `ExternalStorageReadMode { VENICE_ONLY, DUAL_MODE_CONSISTENCY_CHECK, DUAL_MODE_EARLY_RETURN, EXTERNAL_ONLY }` — mirrors `StoreProperties.externalStorageReadMode`. - Store-level Java accessors backed by `StoreProperties` Avro record: * `Store.get/setExternalStorageReadMode` → `storeProperties.externalStorageReadMode`. * `Store.get/setExternalDbName` → `storeProperties.externalDbName`. Implemented on `ZKStore` (int <-> enum where applicable, with null-coercion on the string). `ReadOnlyStore` delegates the getter and throws UOE on the setter; `SystemStore` does the same via its existing `throwUnsupportedOperationException` helper. `StoreInfo.fromStore` mirrors both fields for JSON. - Version-level Java accessors backed by `StoreVersion` Avro record: * `Version.get/setStorageMode` → `storeVersion.storageMode`. * `Version.get/setExternalTableName` → `storeVersion.externalTableName`. Implemented on `VersionImpl`; `cloneVersion` propagates both through the Avro-record copy. `ReadOnlyVersion` delegates getters and throws UOE on setters. ### Code changes - [ ] Added new code behind a config. (no.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: new fields are plain Avro-record fields on the existing storeProperties / storeVersion containers, accessed under the same synchronization invariants as the fields they sit alongside. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections; only enum singletons and a primitive-or-enum-or-String field per type. - [x] No new threading. ## How was this PR tested? - [x] New unit tests added: `ExternalStorageReadModeTest`, `StorageModeTest` (`VeniceEnumValueTest`-based int-mapping coverage); new test methods in `TestStoreInfo`, `TestZKStore`, `TestVersion` for defaults / round-trip / clone preservation / null coercion for both the enum fields (storageMode, externalStorageReadMode) and the string fields (externalDbName, externalTableName); plus "persists through Avro data model" tests on `TestZKStore` and `TestVersion` for all four fields that read the Avro record directly via `dataModel()` to guard against a regression to transient POJO mirroring. - [x] Modified or extended existing tests: yes (`TestStoreInfo` / `TestZKStore` / `TestVersion`). - [x] Verified backward compatibility: the v44 schema is forward-compatible with v43 because the new fields have defaults; downgrading a process that writes v44 records to one that reads v43 records works as long as both sides treat unknown fields as ignorable, which Avro does for record fields with defaults. ## Rollout note Activating the v44 / v99 schemas changes what the controller writes to ZK and what controllers exchange via the admin topic. All Venice services must be redeployed at this version before any caller starts setting `storageMode` / `externalStorageReadMode` / `externalDbName` / `externalTableName` to non-default values, otherwise a v43-only service performing a read-modify-write could silently drop the new fields. ## Does this PR introduce any user-facing or breaking changes? - [x] No (defaults preserve existing behavior; new fields are inert until callers opt in). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, `externalStorageReadMode` on `StoreProperties`, and `targetRegionPromoted` on `StoreVersion`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98 so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-version external database — which external storage database holds this version's data. Captured per-version (alongside externalTableName) so that successive pushes can address distinct external databases when needed (e.g. cross-database migration without store migration). * Per-version external table — which table within that database holds this version's data. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44 `StoreVersion` (both fields per-version, adjacent to the existing `storageMode`): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). - AdminOperation v99 `UpdateStore` (operator-override surface for recovery/migration; successful application copies into the corresponding StoreVersion fields): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). The "NOT_SPECIFIED" sentinel matches the convention established by the `blobTransferInServerEnabled` and `blobDbEnabled` fields already present on UpdateStore — non-null default that makes the field safe to serialize even when not explicitly initialized at construction time. ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) round-trips cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not- pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing `AvroCompatibility` suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version (all four external-storage-identity fields now per-version on Version). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, `externalStorageReadMode` on `StoreProperties`, and `targetRegionPromoted` on `StoreVersion`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98 so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-version external database — which external storage database holds this version's data. Captured per-version (alongside externalTableName) so that successive pushes can address distinct external databases when needed (e.g. cross-database migration without store migration). * Per-version external table — which table within that database holds this version's data. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44 `StoreVersion` (both fields per-version, adjacent to the existing `storageMode`): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). - AdminOperation v99 `UpdateStore` (operator-override surface for recovery/migration; successful application copies into the corresponding StoreVersion fields): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). The "NOT_SPECIFIED" sentinel matches the convention established by the `blobTransferInServerEnabled` and `blobDbEnabled` fields already present on UpdateStore — non-null default that makes the field safe to serialize even when not explicitly initialized at construction time. ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) round-trips cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not- pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing `AvroCompatibility` suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version (all four external-storage-identity fields now per-version on Version). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, `externalStorageReadMode` on `StoreProperties`, and `targetRegionPromoted` on `StoreVersion`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98 so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-version external database — which external storage database holds this version's data. Captured per-version (alongside externalTableName) so that successive pushes can address distinct external databases when needed (e.g. cross-database migration without store migration). * Per-version external table — which table within that database holds this version's data. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44 `StoreVersion` (both fields per-version, adjacent to the existing `storageMode`): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). - AdminOperation v99 `UpdateStore` (operator-override surface for recovery/migration; successful application copies into the corresponding StoreVersion fields): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). The "NOT_SPECIFIED" sentinel matches the convention established by the `blobTransferInServerEnabled` and `blobDbEnabled` fields already present on UpdateStore — non-null default that makes the field safe to serialize even when not explicitly initialized at construction time. ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) round-trips cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not- pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing `AvroCompatibility` suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version (all four external-storage-identity fields now per-version on Version). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, `externalStorageReadMode` on `StoreProperties`, and `targetRegionPromoted` on `StoreVersion`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98 so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-version external database — which external storage database holds this version's data. Captured per-version (alongside externalTableName) so that successive pushes can address distinct external databases when needed (e.g. cross-database migration without store migration). * Per-version external table — which table within that database holds this version's data. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44 `StoreVersion` (both fields per-version, adjacent to the existing `storageMode`): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). - AdminOperation v99 `UpdateStore` (operator-override surface for recovery/migration; successful application copies into the corresponding StoreVersion fields): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). The "NOT_SPECIFIED" sentinel matches the convention established by the `blobTransferInServerEnabled` and `blobDbEnabled` fields already present on UpdateStore — non-null default that makes the field safe to serialize even when not explicitly initialized at construction time. ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) round-trips cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not- pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing `AvroCompatibility` suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version (all four external-storage-identity fields now per-version on Version). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ymuppala
added a commit
to ymuppala/venice
that referenced
this pull request
May 22, 2026
…Value v44 and AdminOperation v99 ## Problem Statement StoreMetaValue v44 (staged by PR linkedin#2806 and extended by PR linkedin#2814) already carries `storageMode` on `StoreVersion`, `externalStorageReadMode` on `StoreProperties`, and `targetRegionPromoted` on `StoreVersion`. AdminOperation v99 carries the corresponding store-level fields on `UpdateStore`. Both versions are staged but `build.gradle` still pins to v43/v98 so they have not yet become wire-real. The external-storage-backed read path needs two more identity fields that v44/v99 do not yet describe: * Per-version external database — which external storage database holds this version's data. Captured per-version (alongside externalTableName) so that successive pushes can address distinct external databases when needed (e.g. cross-database migration without store migration). * Per-version external table — which table within that database holds this version's data. Each Venice push creates a distinct external table (e.g. storeFoo_v17, storeFoo_v18) so rollback to an earlier version naturally points reads at the older table without rewriting it. Since v44/v99 are still staged-but-not-pinned, adding these new fields to the same versions in place is cleaner than introducing v45/v100 — it groups the related external-storage staging into one schema bump and avoids a second pin-advance cycle later. ## Solution Schema-only PR — strictly additive. No Java code is touched. The `build.gradle` pin stays at v43/v98, so v44/v99 remain staged but inactive (the Avro generator keeps producing Java for v43/v98). A follow-up PR will advance the pin and add the Java accessors. - StoreMetaValue v44 `StoreVersion` (both fields per-version, adjacent to the existing `storageMode`): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). - AdminOperation v99 `UpdateStore` (operator-override surface for recovery/migration; successful application copies into the corresponding StoreVersion fields): * Add `externalDbName` (string, default "NOT_SPECIFIED"). * Add `externalTableName` (string, default "NOT_SPECIFIED"). The "NOT_SPECIFIED" sentinel matches the convention established by the `blobTransferInServerEnabled` and `blobDbEnabled` fields already present on UpdateStore — non-null default that makes the field safe to serialize even when not explicitly initialized at construction time. ### Code changes - [ ] Added new code behind a config. (no — schema-only.) - [ ] Introduced new log lines. (no.) ### Concurrency-Specific Checks - [x] No race conditions: schema staging only. No new shared state. - [x] No new synchronization primitives required. - [x] No blocking calls introduced. - [x] No new collections. - [x] No new threading. ## How was this PR tested? - [x] Verified backward compatibility: the new fields have defaults, so any reader-writer pair across v43/v44 (and v98/v99) round-trips cleanly under Avro forward+backward compat rules. v44/v99 stay staged-but-not- pinned, so on-the-wire format is unchanged. - [x] Modified or extended existing tests: no — the existing `AvroCompatibility` suite covers the structural compat check. - [x] Local `:internal:venice-common:test` is green. ## Does this PR introduce any user-facing or breaking changes? - [x] No. Follow-up PRs will: - Advance the StoreMetaValue / AdminOperation pin from v43/v98 to v44/v99, bumping the corresponding AvroProtocolDefinition constants in lockstep. - Add Java accessors that surface storageMode, externalStorageReadMode, externalDbName, and externalTableName on Store / Version (all four external-storage-identity fields now per-version on Version). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Statement
Phase 1 of the paused-SIT design (DVC sequential ingestion for target-region push + deferred swap) requires a signal that tells non-target DaVinci clients when to resume paused ingestion. Today there is no cross-region field on the
Versionobject that a DVC client can observe locally to know the target region has promoted.Solution
Adds
targetRegionPromoted: boolean(defaultfalse) to:StoreVersionrecord inStoreMetaValue/v44(extends v43)UpdateStorerecord inAdminOperation/v99(extends v98)The field will be written by
DeferredVersionSwapServiceon the parent controller once the target region's push completes, then propagated to each child controller via the existingupdateStoreadmin-message path. DVC clients observe the field viahandleStoreChanged(fired by parent-to-child store-state replication) — no cross-region polling, no new controller dependency.This PR is schema-only. The build pin is updated to v43/v98 (the current active versions) so the new schemas are staged but not yet active. The Java wiring (
Versiongetters/setters,UpdateStoreQueryParams,VeniceParentHelixAdmin,AdminExecutionTask,VeniceHelixAdmin,DeferredVersionSwapService,AvroProtocolDefinitionbump, and pin removal) follows in a stacked PR.How was this PR tested?
TestDeferredVersionSwapWithSequentialRolloutWithDvcverifyingtargetRegionPromoted=truepropagates to all child controllers after target-region push completes)"default": false; existing blobs without the field deserialize cleanly tofalsevia Avro schema evolution)Does this PR introduce any user-facing or breaking changes?