Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[consumer] Add rack-awareness (KIP-881) to assignors #4252

Merged
merged 15 commits into from
Jun 10, 2023
Merged

[consumer] Add rack-awareness (KIP-881) to assignors #4252

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
061b45f
[consumer] Add rack-awareness (KIP-881) to range assignor
milindl Apr 13, 2023
804f241
Fix some memory issues in unittests
milindl Apr 13, 2023
09d67ef
Include metadata refactor changes
milindl May 16, 2023
ddcedb4
Add rack-awareness (KIP-881) to sticky-assignor (#4274)
milindl May 18, 2023
ee0c0a4
Remove unused functions/macros
milindl May 18, 2023
1cb36e3
Fix MSVC build issue
milindl May 18, 2023
b4de1ee
4252 (KIP-881) Verify number of partitions with rack mismatch (#4289)
emasab May 25, 2023
cb809f7
Use precomputed toppar-rack mapping for assignors
milindl Jun 5, 2023
fb2c94f
Strip changes from cooperative rebalance fix
milindl Jun 5, 2023
4ed1db8
Fix style issues
milindl Jun 5, 2023
3d8d9f9
Remove TODOs and address internal comments
milindl Jun 7, 2023
8a31959
Address review comments
milindl Jun 8, 2023
3ce509d
Remove partition from currentPartitionConsumer in case of extra consumer
milindl Jun 9, 2023
e07e4be
Avoid recomputation of partition racks per topic
milindl Jun 9, 2023
30a26dc
Revert accidental commit which changed null checks
milindl Jun 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ librdkafka v2.2.0 is a feature release:
* Store offset commit metadata in `rd_kafka_offsets_store` (@mathispesch, #4084).
* Fix a bug that happens when skipping tags, causing buffer underflow in
MetadataResponse (#4278).
* [KIP-881](https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers):
Add support for rack-aware partition assignment for consumers
(#4184, #4291, #4252).
milindl marked this conversation as resolved.
Show resolved Hide resolved
* Fix several bugs with sticky assignor in case of partition ownership
changing between members of the consumer group (#4252).


## Fixes

Expand All @@ -19,6 +25,19 @@ librdkafka v2.2.0 is a feature release:
Fixed by skipping the correct amount of bytes when tags are received.


### Consumer fixes

* In case of multiple owners of a partition with different generations, the
sticky assignor would pick the earliest (lowest generation) member as the
current owner, which would lead to stickiness violations. Fixed by
choosing the latest (highest generation) member.
* In case where the same partition is owned by two members with the same
generation, it indicates an issue. The sticky assignor had some code to
handle this, but it was non-functional, and did not have parity with the
Java assignor. Fixed by invalidating any such partition from the current
assignment completely.


# librdkafka v2.1.1

librdkafka v2.1.1 is a maintenance release:
Expand Down
1 change: 1 addition & 0 deletions INTRODUCTION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1957,6 +1957,7 @@ The [Apache Kafka Implementation Proposals (KIPs)](https://cwiki.apache.org/conf
| KIP-654 - Aborted txns with non-flushed msgs should not be fatal | 2.7.0 | Supported |
| KIP-735 - Increase default consumer session timeout | 3.0.0 | Supported |
| KIP-768 - SASL/OAUTHBEARER OIDC support | 3.0 | Supported |
| KIP-881 - Rack-aware Partition Assignment for Kafka Consumers | 3.5.0 (WIP) | Supported |



Expand Down