[KIP-848] Added new error code handling to OffsetCommit and OffsetFetch with Test cases #4672
+7,696
−1,547
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pranavrth
changed the title
emasab
force-pushed
the
dev_kip848_mock_handler_and_integration_tests
branch
13 times, most recently
from
600989c
to
2de4368
Compare
pranavrth
force-pushed
the
dev_kip848_offset_fetch_commit_error_codes
branch
from
5a5657f
to
0e78f04
Compare
pranavrth
changed the title
emasab
force-pushed
the
dev_kip848_mock_handler_and_integration_tests
branch
4 times, most recently
from
0cc9ad2
to
06f34b1
Compare
and response with basic cgrp_consumer flow - Added topic id to topic_partition_t while reading from buffer - Added new methods and way to add topic_id to topic partition private - Added new configs group.protocol and group.remote.assignor - Added ConsumerGroupHeartbeat API Request Contract - Added ConsumerGroupApi Request without handling different cases - Working ConsumerGroupHeartbeat API with proper response - Properly receiving assigned partitions with topic_id from the Response - Added metadata request as well after parsing the response. Separate it out from to the topic partition branch - Added metadata response flow with rko - Updated OffsetFetch to v9 - Removed unrequred fields from ConsumerGroupHeartbeat API to make it work with AK > 3.6 - OffsetFetch working fine. Able to consume. - Changed subsribed list to use correct field - Fixed few memory leaks - Some more memory leak fixes. Added updation to subscribed topics list - Minor changes - [KIP-848] Added new configs group.protocol and group.remote.assignor (#4414) - Added new configs group.protocol and group.remote.assignor - Removed printfs and Updated hardcoded one topic for metadata request - Removed some changes related to removed changes in the protocols - [KIP-848] Added topic id to topic_partition_t while reading from buffer (#4416) - Updating topic name from metadata response for all the requested topic_ids instead of just 1. - Style fixes and fixed skip tag issue in buf_*_topic_partition - Changed variable for next assignment - Added topic name while reading topic partition buffer - Changed variable name from assignments to assignment
- Rebased with master - WIP: assignment, revocation, leave group flow - Remove print statements - Remove print statement left - Separate rd_kafka_cgrp_consumer_assignment_done - Allow changing subscription to empty - Expedite next heartbeat - Static group membership and max poll interval checks - Expedite next heartbeat - Fix existing protocol - Partial implementation of reconciliation and next assignment handling - Uniform tests handling across scripts and KRaft mode - Run tests with group.protocol=consumer and reusable condition to skip mock cluster - Test 0113 partial - Test 0018 - Test 0113 stickyness - Test 0113 complete except regex subscription and u_multiple_subscription_changes(true) - Skip some tests, fix subscription change - Test 0029 exclusion clarified - Debug statements - Introduce current assignment rename rkcg_current_target_assignments to rkcg_target_assignment rename rkcg_next_target_assignments to rkcg_next_target_assignment - change to ConsumerGroupHeartbeat in logs - Add remote assignor to debug log - Fix rd_kafka_buf_write_topic_partitions not using topic ids for comparison
…onse and various fixes (#4634) - Added error handling to ConsumerGroupHeartbeat API - Added type new errors - UNRELEASED_INSTANCE_ID and UNSUPPORTED_ASSIGNOR - Added partial acknowledgement flow - Upgraded OffsetCommit Request and response to v9 - Fixed metadata being called with duplicate topic id - Fixed next_target_assignment not getting reset to NULL - Fixed member stuck if fenced during rebalancing - Fixed segfault with current and target assignment while resetting consumer group - Fixed segfault due to deleted topic in metadata - Fixed leave not being called if the consumer without any assignment leaves
improve documentation of Metadata functions
Metadata cache was cleared on full metadata refresh, leading to unnecessary refreshes and occasional `UNKNOWN_TOPIC_OR_PART` errors. Solved by updating cache for existing or hinted entries instead of clearing them. Happening since 2.1.0
emasab
force-pushed
the
dev_kip848_mock_handler_and_integration_tests
branch
from
06f34b1
to
4d66505
Compare
…g a consumer group (#4678) A metadata call before member joins consumer group, could lead to an `UNKNOWN_TOPIC_OR_PART` error. Solved by updating the consumer group following a metadata refresh only in safe states. Happening since 2.1.0
emasab
force-pushed
the
dev_kip848_mock_handler_and_integration_tests
branch
2 times, most recently
from
59bc52c
to
504c49b
Compare
Metadata refreshes without partition leader change could lead to a loop of metadata calls at fixed intervals. Solved by stopping metadata refresh when all existing metadata is non-stale. Happening since 2.3.0
emasab
force-pushed
the
dev_kip848_mock_handler_and_integration_tests
branch
2 times, most recently
from
a047935
to
7662fc4
Compare
- rename 'generic' protocol to 'classic'
- consumer group serve timer to awake the loop earlier
- compare and find into topic partition list by topic id only
- fix memory leak when instance creation fails and app_conf is provided
- fix cases where HB response is received after unsubscription
- use topic name from current assignment if it's missing from metadata
- expedite heartbeat simplification and
fixes to next interval check and to the place where
number of retries is increased
- expedite HB after changing state back to INIT
- use the CONSUMER_F_WAIT_REJOIN to trigger
the rejoin instead of calling it from max poll interval timer
- schedule timer for next execution
reschedule it if expected earlier
expedite through scheduling cgrp serve timer
- treat unsupported feature error
as fatal
- avoid removing partitions not matched by a new subscription
immediately, for a possible desynchronization with the coordinator,
with said partitions not being consumed anymore
…ch (#4681) - Added new errors to manual commit. - improvements to OffsetCommit and OffsetFetch error code handling.
- Mock handler implementation - Rename current consumer protocol from generic to classic - Mock handler with automatic or manual assignment - More consumer group metadata getters - Test helpers - Expedite next HB after FindCoordinator doing it with an exponential backoff to avoid tight loops - Configurable session timeout and HB interval - Fix mock handler ListOffsets response LeaderEpoch instead of CurrentLeaderEpoch - Integration tests passing with AK trunk - Improve documentation and KIP 848 specific mock tests
in metadata request and partial reconciliation
emasab
force-pushed
the
dev_kip848_mock_handler_and_integration_tests
branch
from
7662fc4
to
e08e55d
Compare
pranavrth
force-pushed
the
dev_kip848_offset_fetch_commit_error_codes
branch
from
0e78f04
to
79690d6
Compare
emasab
force-pushed
the
dev_kip848_mock_handler_and_integration_tests
branch
5 times, most recently
from
8d9ed36
to
c95d2ee
Compare
emasab
force-pushed
the
dev_kip848_mock_handler_and_integration_tests
branch
from
d4d3f8f
to
24e2ef5
Compare
emasab
force-pushed
the
dev_kip848_mock_handler_and_integration_tests
branch
2 times, most recently
from
f8467ec
to
c3e5f66
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.