feat: fetch partition routing and on-demand metadata refresh#126
Merged
novatechflow merged 2 commits intoKafScale:mainfrom Mar 2, 2026
Merged
feat: fetch partition routing and on-demand metadata refresh#126novatechflow merged 2 commits intoKafScale:mainfrom
novatechflow merged 2 commits intoKafScale:mainfrom
Conversation
klaudworks
changed the title
novatechflow
previously approved these changes
Mar 2, 2026
Collaborator
|
@klaudworks - can you please resolve the conflicts? |
Split fetch requests by owning broker, forward concurrently, and merge responses. Retry partitions rejected with NOT_LEADER_OR_FOLLOWER up to 3 times. For v12+ requests that use topic IDs instead of names, resolve IDs via a metadata-refreshed cache and use a collision-safe key to prevent unresolved topics from merging silently. Adds EncodeFetchRequest and ParseFetchResponse codecs with round-trip and kmsg validation tests.
Remove the 3-second polling ticker and refresh broker/topic caches on demand when a lookup misses. Concurrent misses are coalesced via singleflight to avoid thundering herd metadata fetches. Readiness probe now checks cached state first (fast path), falling back to a live metadata fetch only when the cache TTL expires. Static backends are always ready. Clean up comments per AGENTS.md: remove low-value comments, condense verbose doc comments.
klaudworks
force-pushed
the
feat/fetch-partition-routing
branch
from
4acb9c5 to
fa6938f
Compare
Collaborator
Author
|
@novatechflow Unfortunately github has no support for stacked PR so I have to rebase each PR after the prior one is merged. Once this is approved + merged, I'll rebase the next one. |
novatechflow
approved these changes
Mar 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge #125 first (group coordination routing). This PR is stacked on top.
Closes #115
Summary
Outlook
Consumers will now directly fetch data from brokers that already have it in their segment cache. Slow S3 requests are only required to reprocess older data after partition reassignments or similar.