Skip to content

KAFKA-15038: Remove topicIdByPartitionMap from RemoteLogManager#22345

Open
JiayaoS wants to merge 1 commit into
apache:trunkfrom
JiayaoS:feature/KAFKA-15038
Open

KAFKA-15038: Remove topicIdByPartitionMap from RemoteLogManager#22345
JiayaoS wants to merge 1 commit into
apache:trunkfrom
JiayaoS:feature/KAFKA-15038

Conversation

@JiayaoS
Copy link
Copy Markdown
Contributor

@JiayaoS JiayaoS commented May 22, 2026

Remove topicIdByPartitionMap from RemoteLogManager and resolve topic IDs from the metadata cache for normal remote log operations. For stop-partition paths, pass TopicIdPartition through StopPartition so remote log cleanup can use the topic ID from the metadata image even when the current metadata cache no longer contains the topic.

@github-actions github-actions Bot added triage PRs from the community core Kafka Broker storage Pull requests that target the storage module tiered-storage Related to the Tiered Storage feature labels May 22, 2026
@JiayaoS
Copy link
Copy Markdown
Contributor Author

JiayaoS commented May 22, 2026

This revives and updates #20256 for KAFKA-15038. RemoteLogManager previously maintained its own topic ID cache, populated from leadership changes and cleared during stop-partition handling. That duplicated metadata state and could be missing during topic deletion or partition stop paths.

With this change, RemoteLogManager no longer owns a topic ID cache. Normal operations resolve topic IDs from the broker metadata cache. Stop-partition handling uses the TopicIdPartition supplied by the caller. This is important for KRaft topic deletion, where the metadata cache may already reflect the new image before ReplicaManager.applyDelta processes the delete delta. ReplicaManager now uses the previous metadata image topic ID when constructing StopPartition, allowing RemoteLogManager to clean up remote log state with the correct topic ID.

Tests:

  • Added ReplicaManager coverage for topic deletion where the current metadata cache no longer contains the deleted topic ID.
  • Added RemoteLogManager coverage to verify stopPartitions uses the TopicIdPartition supplied by StopPartition.
  • Removed obsolete RemoteLogManager tests that validated topicIdByPartitionMap cache population/eviction and cache-based partition readiness, since RemoteLogManager no longer maintains that cache.
  • Updated existing tests for the new StopPartition topic ID flow.

@JiayaoS JiayaoS force-pushed the feature/KAFKA-15038 branch from c9921e5 to 3f25030 Compare May 22, 2026 00:52
@JiayaoS JiayaoS marked this pull request as ready for review May 22, 2026 00:56
@JiayaoS JiayaoS force-pushed the feature/KAFKA-15038 branch from 3f25030 to befa268 Compare May 22, 2026 01:01
@JiayaoS JiayaoS force-pushed the feature/KAFKA-15038 branch from befa268 to 1ec501c Compare May 22, 2026 01:16
@github-actions
Copy link
Copy Markdown

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-approved core Kafka Broker needs-attention storage Pull requests that target the storage module tiered-storage Related to the Tiered Storage feature triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants