[controller] Remove V1 admin topic metadata and standardize on V2 format by sushantmane · Pull Request #2465 · linkedin/venice

sushantmane · 2026-02-08T05:13:01Z

Problem Statement

The Venice controller currently supports both V1 (numeric offsets) and V2 (PubSubPosition-based) admin topic metadata formats. This dual support adds complexity and is no longer needed since V2 has been stable.

Solution

Remove all V1 admin topic metadata code and standardize exclusively on V2 format.

Code changes

ConfigKeys.java: Removed USE_V2_ADMIN_TOPIC_METADATA config key (no longer needed)
VeniceZkPaths.java: Removed ADMIN_TOPIC_METADATA constant for V1 path
VeniceControllerClusterConfig.java: Removed isUseV2AdminTopicMetadata() method
ZkAdminTopicMetadataAccessor.java: Removed all V1 methods, now only uses V2 format with AdminMetadata
AdminConsumptionTask.java: Added code to synchronize numeric offsets with positions when persisting metadata (backward compatibility during migration)
AdminMetadata.java: Removed V1-related methods
AdminConsumptionStats.java: Removed failed_admin_message_offset metric that exposed numeric offset
AdminTopicMetadataAccessor.java: Added generateAdminMetadata() helper for V2 format
VeniceParentHelixAdmin.java: Added @VisibleForTesting setter for adminTopicMetadataAccessor

Test changes

Updated test base class to use mock AdminTopicMetadataAccessor instead of mocking zkClient.readData()
Removed obsolete zkClient mock setups and verification calls
Tests now focus on behavior rather than ZK implementation details

Code changes

Added new code behind a config. If so list the config names and their default values in the PR description.
Introduced new log lines.
- Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Code has no race conditions or thread safety issues.
Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

Modified or extended existing tests.
Verified backward compatibility (if applicable).

All 160+ controller unit tests pass.

Does this PR introduce any user-facing or breaking changes?

No. You can skip the rest of this section.

This is a cleanup PR that removes deprecated V1 admin topic metadata support. Systems should already be using V2 format.

Based on PR #2297 by @haoxu07, rebased and fixed test failures

Copilot

Pull request overview

Removes legacy V1 (numeric-offset-based) admin topic metadata handling from the Venice controller and standardizes persistence and reads on the V2 AdminMetadata (PubSubPosition-based) format, with corresponding unit test updates and ZK path/documentation adjustments.

Changes:

Removed V1 admin topic metadata config/path support and updated controller/ZK accessor code to use only V2 AdminMetadata.
Updated controller admin-consumption persistence logic to keep numeric offsets aligned with PubSub positions during migration.
Refactored/updated unit tests and admin-tool ZK path expectations to align with V2-only metadata.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
services/venice-controller/src/test/java/com/linkedin/venice/controller/kafka/consumer/AdminConsumptionTaskTest.java	Updates tests to use `AdminMetadata` instead of legacy metadata maps.
services/venice-controller/src/test/java/com/linkedin/venice/controller/TestZkAdminTopicMetadataAccessor.java	Updates ZK accessor tests to V2-only `AdminMetadata` reads/writes.
services/venice-controller/src/test/java/com/linkedin/venice/controller/TestVeniceParentHelixAdminWithAcl.java	Removes legacy ZK metadata mocking tied to V1.
services/venice-controller/src/test/java/com/linkedin/venice/controller/TestVeniceParentHelixAdmin.java	Removes legacy ZK metadata mocking tied to V1.
services/venice-controller/src/test/java/com/linkedin/venice/controller/AbstractTestVeniceParentHelixAdmin.java	Switches to V2 ZK metadata path and injects a mock metadata accessor for tests.
services/venice-controller/src/main/java/com/linkedin/venice/controller/stats/AdminConsumptionStats.java	Removes numeric-offset-based failure metric.
services/venice-controller/src/main/java/com/linkedin/venice/controller/kafka/consumer/AdminMetadata.java	Removes legacy map conversion and simplifies position/offset handling.
services/venice-controller/src/main/java/com/linkedin/venice/controller/kafka/consumer/AdminConsumptionTask.java	Persists V2 metadata and adds logic intended to sync numeric offsets with positions.
services/venice-controller/src/main/java/com/linkedin/venice/controller/ZkAdminTopicMetadataAccessor.java	Removes V1 ZK path/serialization and reads/writes only V2 `AdminMetadata`.
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java	Makes metadata accessor settable for tests via `@VisibleForTesting` setter.
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerClusterConfig.java	Removes the V2-toggle config field/method (now always V2).
services/venice-controller/src/main/java/com/linkedin/venice/controller/AdminTopicMetadataAccessor.java	Adds a V2 helper method for generating `AdminMetadata`.
internal/venice-common/src/main/java/com/linkedin/venice/zk/VeniceZkPaths.java	Removes V1 admin topic metadata constant and updates cluster managed-path set.
internal/venice-common/src/main/java/com/linkedin/venice/ConfigKeys.java	Removes obsolete `USE_V2_ADMIN_TOPIC_METADATA` config key.
docs/venice-zk-helix-layout.md	Adds ZK/Helix layout documentation including admin topic metadata sections.
clients/venice-admin-tool/src/test/java/com/linkedin/venice/TestZkCopier.java	Updates admin-tool tests toward V2 constant usage.
clients/venice-admin-tool/src/test/java/com/linkedin/venice/TestTreeNode.java	Updates admin-tool tree tests toward V2 constant usage.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mynameborat

Changes looks fine.
One question about renaming occurrences of AdminTopicMetadataV2 to just AdminTopicMetadata as some of the accessors and stuff point to just metadata.

Do you think its okay to rename everything to AdminTopicMetadata give we have cleaned up and fully ramped?

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mynameborat

Thank you! LGTM

sushantmane · 2026-02-17T21:05:43Z

Thanks a lot for the review, @bharathkk!

- Remove offset/upstreamOffset fields from AdminMetadata and serializer - Add AdminMetadataOld and AdminMetadataJSONSerializerOld for forward compatibility testing - Add testForwardCompatibilityNewDataWithoutOffsetFields to verify rollback scenario - Make adminTopicMetadataAccessor volatile in VeniceParentHelixAdmin - Rename ADMIN_TOPIC_METADATA_SECONDARY_NODE to ADMIN_TOPIC_METADATA_NODE - Remove unused getExecutionId(Map) method from AdminTopicMetadataAccessor

Copilot AI review requested due to automatic review settings February 8, 2026 05:13

Copilot started reviewing on behalf of sushantmane February 8, 2026 05:13 View session

Copilot AI reviewed Feb 8, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings February 8, 2026 06:19

Copilot started reviewing on behalf of sushantmane February 8, 2026 06:19 View session

Copilot AI reviewed Feb 8, 2026

View reviewed changes

sushantmane commented Feb 8, 2026

View reviewed changes

Comment thread clients/venice-admin-tool/src/test/resources/zk_paths.txt

mynameborat reviewed Feb 9, 2026

View reviewed changes

Comment thread ...ce-controller/src/main/java/com/linkedin/venice/controller/ZkAdminTopicMetadataAccessor.java

Comment thread ...nice-controller/src/test/java/com/linkedin/venice/controller/TestVeniceParentHelixAdmin.java

sushantmane added the requested-changes Reviewer requested changes label Feb 13, 2026

Copilot AI review requested due to automatic review settings February 16, 2026 07:13

sushantmane force-pushed the removeNumericOffsetFromAdmin branch from 3d42243 to 5e0099f Compare February 16, 2026 07:13

Copilot started reviewing on behalf of sushantmane February 16, 2026 07:13 View session

Copilot AI reviewed Feb 16, 2026

View reviewed changes

sushantmane force-pushed the removeNumericOffsetFromAdmin branch from 5e0099f to 7bac18a Compare February 16, 2026 07:59

Copilot AI review requested due to automatic review settings February 16, 2026 11:02

sushantmane force-pushed the removeNumericOffsetFromAdmin branch from 7bac18a to 5bdbf22 Compare February 16, 2026 11:02

Copilot started reviewing on behalf of sushantmane February 16, 2026 11:02 View session

Copilot AI reviewed Feb 16, 2026

View reviewed changes

Comment thread ...-test-common/src/main/java/com/linkedin/venice/admin/InMemoryAdminTopicMetadataAccessor.java

Comment thread ...src/main/java/com/linkedin/venice/controller/kafka/consumer/AdminMetadataJSONSerializer.java

sushantmane added review-addressed Author has addressed review comments and removed requested-changes Reviewer requested changes labels Feb 16, 2026

mynameborat approved these changes Feb 17, 2026

View reviewed changes

mynameborat added approved PR has been approved and removed review-addressed Author has addressed review comments labels Feb 17, 2026

sushantmane enabled auto-merge (squash) February 17, 2026 21:09

sushantmane force-pushed the removeNumericOffsetFromAdmin branch from 5bdbf22 to 136ce8f Compare February 17, 2026 23:38

sushantmane merged commit c7c55d9 into linkedin:main Feb 18, 2026
51 checks passed

Uh oh!

Conversation

sushantmane commented Feb 8, 2026

Problem Statement

Solution

Code changes

Test changes

Code changes

Concurrency-Specific Checks

How was this PR tested?

Does this PR introduce any user-facing or breaking changes?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mynameborat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

mynameborat left a comment

Choose a reason for hiding this comment

Uh oh!

sushantmane commented Feb 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants