Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incremental alter configs implementation [KIP-339] #4110

Merged
merged 34 commits into from
Jul 10, 2023
Merged

Incremental alter configs implementation [KIP-339] #4110

Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
5451d69
incremental alter configs implementation
PrasanthV454 Dec 16, 2022
1fb808a
added simple test to test.c
PrasanthV454 Dec 20, 2022
2c6b6b5
incremental alter enum fix
PrasanthV454 Dec 20, 2022
af9b012
Api Version revert max to zero
PrasanthV454 Jan 2, 2023
c164e25
latest changes update
PrasanthV454 Feb 5, 2023
a9f24f9
removed temp files
PrasanthV454 Feb 5, 2023
de30be6
removed temp files
PrasanthV454 Feb 5, 2023
3db3b75
exported append,subtract and delete
PrasanthV454 Feb 5, 2023
da3d4f1
style errors fixed
PrasanthV454 Feb 5, 2023
74f053d
Merge branch 'master' into feature/incremental-alter-KIP339
emasab May 30, 2023
0abdca2
Upgrade IncrementalAlterConfigs version to 1
emasab May 30, 2023
98816ed
Address comments
emasab May 30, 2023
96f13bb
Documentation fixes
emasab May 30, 2023
e295a28
Add example
emasab May 30, 2023
3c746e5
CMake fix
emasab May 30, 2023
fddd40e
Fail early on duplicate ConfigResource or
emasab Jun 1, 2023
c33e72b
Return rd_kafka_error_t * instead of
emasab Jun 6, 2023
fd23815
Doxygen fixes
emasab Jun 6, 2023
9991b3c
Changelog entry
emasab Jun 8, 2023
e80b257
Use single function for all the
emasab Jun 21, 2023
a63ca7f
Rename rd_kafka_ConfigResource_incremental_alter_config
emasab Jun 21, 2023
d81c161
Documentation for rd_kafka_AlterConfigOpType_t
emasab Jun 29, 2023
fd209ae
Address comments
emasab Jun 30, 2023
c462e01
rd_kafka_AlterOperation_t doc
emasab Jun 30, 2023
b233fea
Remove previous never really used
emasab Jun 30, 2023
82c8278
Merge branch 'master' into feature/incremental-alter-KIP339
emasab Jun 30, 2023
17c0a7c
Revert broker option to default if
emasab Jul 3, 2023
eb0d852
Documentation about broker option.
emasab Jul 3, 2023
6976690
Test healper with different resource type
emasab Jul 3, 2023
0370ad0
Fix parameter type and documentation.
emasab Jul 3, 2023
ee26cd5
Incremental alter configs integration
emasab Jul 4, 2023
9cbc19d
Rename set_incremental_config
emasab Jul 5, 2023
f88940c
Merge branch 'master' into feature/incremental-alter-KIP339
emasab Jul 10, 2023
df55c44
Changelog contribution
emasab Jul 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ librdkafka v2.2.0 is a feature release:
* Store offset commit metadata in `rd_kafka_offsets_store` (@mathispesch, #4084).
* Fix a bug that happens when skipping tags, causing buffer underflow in
MetadataResponse (#4278).
* [KIP-339](https://cwiki.apache.org/confluence/display/KAFKA/KIP-339%3A+Create+a+new+IncrementalAlterConfigs+API):
Create a new IncrementalAlterConfigs API (#4110).
emasab marked this conversation as resolved.
Show resolved Hide resolved

## Fixes

Expand Down
67 changes: 34 additions & 33 deletions INTRODUCTION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1909,7 +1909,7 @@ The [Apache Kafka Implementation Proposals (KIPs)](https://cwiki.apache.org/conf
| KIP-302 - Use all addresses for resolved broker hostname | 2.1.0 | Supported |
| KIP-320 - Consumer: handle log truncation | 2.1.0, 2.2.0 | Supported |
| KIP-322 - DeleteTopics disabled error code | 2.1.0 | Supported |
| KIP-339 - AdminAPI: incrementalAlterConfigs | 2.3.0 | Not supported |
| KIP-339 - AdminAPI: incrementalAlterConfigs | 2.3.0 | Supported |
| KIP-341 - Update Sticky partition assignment data | 2.3.0 | Not supported (superceeded by KIP-429) |
| KIP-342 - Custom SASL OAUTHBEARER extensions | 2.1.0 | Supported |
| KIP-345 - Consumer: Static membership | 2.4.0 | Supported |
Expand Down Expand Up @@ -1963,42 +1963,43 @@ The [Apache Kafka Implementation Proposals (KIPs)](https://cwiki.apache.org/conf

### Supported protocol versions

"Kafka max" is the maximum ApiVersion supported in Apache Kafka 3.3.1, while
"Kafka max" is the maximum ApiVersion supported in Apache Kafka 3.4.0, while
"librdkafka max" is the maximum ApiVersion supported in the latest
release of librdkafka.


| ApiKey | Request name | Kafka max | librdkafka max |
| ------- | ------------------- | ----------- | ----------------------- |
| 0 | Produce | 9 | 7 |
| 1 | Fetch | 13 | 11 |
| 2 | ListOffsets | 7 | 2 |
| 3 | Metadata | 12 | 9 |
| 8 | OffsetCommit | 8 | 7 |
| 9 | OffsetFetch | 8 | 7 |
| 10 | FindCoordinator | 4 | 2 |
| 11 | JoinGroup | 9 | 5 |
| 12 | Heartbeat | 4 | 3 |
| 13 | LeaveGroup | 5 | 1 |
| 14 | SyncGroup | 5 | 3 |
| 15 | DescribeGroups | 5 | 4 |
| 16 | ListGroups | 4 | 4 |
| 17 | SaslHandshake | 1 | 1 |
| 18 | ApiVersions | 3 | 3 |
| 19 | CreateTopics | 7 | 4 |
| 20 | DeleteTopics | 6 | 1 |
| 21 | DeleteRecords | 2 | 1 |
| 22 | InitProducerId | 4 | 4 |
| 24 | AddPartitionsToTxn | 3 | 0 |
| 25 | AddOffsetsToTxn | 3 | 0 |
| 26 | EndTxn | 3 | 1 |
| 28 | TxnOffsetCommit | 3 | 3 |
| 32 | DescribeConfigs | 4 | 1 |
| 33 | AlterConfigs | 2 | 1 |
| 36 | SaslAuthenticate | 2 | 0 |
| 37 | CreatePartitions | 3 | 0 |
| 42 | DeleteGroups | 2 | 1 |
| 47 | OffsetDelete | 0 | 0 |
| ApiKey | Request name | Kafka max | librdkafka max |
| ------- | ------------------------| ----------- | ----------------------- |
| 0 | Produce | 9 | 7 |
| 1 | Fetch | 13 | 11 |
| 2 | ListOffsets | 7 | 2 |
| 3 | Metadata | 12 | 9 |
| 8 | OffsetCommit | 8 | 7 |
| 9 | OffsetFetch | 8 | 7 |
| 10 | FindCoordinator | 4 | 2 |
| 11 | JoinGroup | 9 | 5 |
| 12 | Heartbeat | 4 | 3 |
| 13 | LeaveGroup | 5 | 1 |
| 14 | SyncGroup | 5 | 3 |
| 15 | DescribeGroups | 5 | 4 |
| 16 | ListGroups | 4 | 4 |
| 17 | SaslHandshake | 1 | 1 |
| 18 | ApiVersions | 3 | 3 |
| 19 | CreateTopics | 7 | 4 |
| 20 | DeleteTopics | 6 | 1 |
| 21 | DeleteRecords | 2 | 1 |
| 22 | InitProducerId | 4 | 4 |
| 24 | AddPartitionsToTxn | 3 | 0 |
| 25 | AddOffsetsToTxn | 3 | 0 |
| 26 | EndTxn | 3 | 1 |
| 28 | TxnOffsetCommit | 3 | 3 |
| 32 | DescribeConfigs | 4 | 1 |
| 33 | AlterConfigs | 2 | 2 |
| 36 | SaslAuthenticate | 2 | 0 |
| 37 | CreatePartitions | 3 | 0 |
| 42 | DeleteGroups | 2 | 1 |
| 44 | IncrementalAlterConfigs | 1 | 1 |
| 47 | OffsetDelete | 0 | 0 |



Expand Down
1 change: 1 addition & 0 deletions examples/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,4 @@ list_consumer_groups
describe_consumer_groups
list_consumer_group_offsets
alter_consumer_group_offsets
incremental_alter_configs
23 changes: 23 additions & 0 deletions examples/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,29 @@ target_link_libraries(openssl_engine_example_cpp PUBLIC rdkafka++)
add_executable(misc misc.c ${win32_sources})
target_link_libraries(misc PUBLIC rdkafka)

add_executable(idempotent_producer idempotent_producer.c ${win32_sources})
target_link_libraries(idempotent_producer PUBLIC rdkafka)

add_executable(transactions transactions.c ${win32_sources})
target_link_libraries(transactions PUBLIC rdkafka)

add_executable(delete_records delete_records.c ${win32_sources})
target_link_libraries(delete_records PUBLIC rdkafka)

add_executable(list_consumer_groups list_consumer_groups.c ${win32_sources})
target_link_libraries(list_consumer_groups PUBLIC rdkafka)

add_executable(describe_consumer_groups describe_consumer_groups.c ${win32_sources})
target_link_libraries(describe_consumer_groups PUBLIC rdkafka)

add_executable(list_consumer_group_offsets list_consumer_group_offsets.c ${win32_sources})
target_link_libraries(list_consumer_group_offsets PUBLIC rdkafka)

add_executable(alter_consumer_group_offsets alter_consumer_group_offsets.c ${win32_sources})
target_link_libraries(alter_consumer_group_offsets PUBLIC rdkafka)

add_executable(incremental_alter_configs incremental_alter_configs.c ${win32_sources})
target_link_libraries(incremental_alter_configs PUBLIC rdkafka)

# The targets below has Unix include dirs and do not compile on Windows.
if(NOT WIN32)
Expand Down
5 changes: 5 additions & 0 deletions examples/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ EXAMPLES ?= rdkafka_example rdkafka_performance rdkafka_example_cpp \
describe_consumer_groups \
list_consumer_group_offsets \
alter_consumer_group_offsets \
incremental_alter_configs \
misc

all: $(EXAMPLES)
Expand Down Expand Up @@ -80,6 +81,10 @@ alter_consumer_group_offsets: ../src/librdkafka.a alter_consumer_group_offsets.c
$(CC) $(CPPFLAGS) $(CFLAGS) $@.c -o $@ $(LDFLAGS) \
../src/librdkafka.a $(LIBS)

incremental_alter_configs: ../src/librdkafka.a incremental_alter_configs.c
$(CC) $(CPPFLAGS) $(CFLAGS) $@.c -o $@ $(LDFLAGS) \
../src/librdkafka.a $(LIBS)

rdkafka_complex_consumer_example: ../src/librdkafka.a rdkafka_complex_consumer_example.c
$(CC) $(CPPFLAGS) $(CFLAGS) rdkafka_complex_consumer_example.c -o $@ $(LDFLAGS) \
../src/librdkafka.a $(LIBS)
Expand Down
1 change: 1 addition & 0 deletions examples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,4 @@ For more complex uses, see:
* [describe_consumer_groups.c](describe_consumer_groups.c) - Describe consumer groups.
* [list_consumer_group_offsets.c](list_consumer_group_offsets.c) - List offsets of a consumer group.
* [alter_consumer_group_offsets.c](alter_consumer_group_offsets.c) - Alter offsets of a consumer group.
* [incremental_alter_configs.c](incremental_alter_configs.c) - Incrementally alter resource configurations.