Merge master to async #2068

k-raina · 2025-09-26T11:56:51Z

Summary

Merge master branch to async branch
Resolve ALL conflicts why picking up async branch changes

Conflict details:

On branch async
Your branch is up to date with 'origin/async'.

You have unmerged paths.
  (fix conflicts and run "git commit")
  (use "git merge --abort" to abort the merge)

Changes to be committed:
        new file:   .ducktape/metadata/session_id
        modified:   .semaphore/semaphore.yml
        new file:   CONTRIBUTOR.md
        modified:   DEVELOPER.md
        modified:   README.md
        modified:   examples/README.md
        modified:   examples/docker/README.md
        modified:   requirements/requirements-schemaregistry.txt
        modified:   requirements/requirements-tests.txt
        modified:   src/confluent_kafka/schema_registry/__init__.py
        modified:   src/confluent_kafka/schema_registry/_async/schema_registry_client.py
        modified:   src/confluent_kafka/schema_registry/_sync/schema_registry_client.py
        modified:   src/confluent_kafka/schema_registry/common/json_schema.py
        modified:   src/confluent_kafka/schema_registry/common/schema_registry_client.py
        modified:   src/confluent_kafka/schema_registry/rules/encryption/encrypt_executor.py
        modified:   tests/README.md
        new file:   tests/ducktape/test_producer_with_schema_registry.py
        modified:   tests/schema_registry/_async/test_api_client.py
        modified:   tests/schema_registry/_async/test_avro_serdes.py
        modified:   tests/schema_registry/_async/test_config.py
        modified:   tests/schema_registry/_async/test_json_serdes.py
        modified:   tests/schema_registry/_sync/test_api_client.py
        modified:   tests/schema_registry/_sync/test_avro_serdes.py
        modified:   tests/schema_registry/_sync/test_config.py
        modified:   tests/schema_registry/_sync/test_json_serdes.py
        modified:   tests/schema_registry/conftest.py
        modified:   tests/schema_registry/test_schema_id.py
        modified:   tox.ini

Unmerged paths:
  (use "git add <file>..." to mark resolution)
        both modified:   .gitignore
        both modified:   tests/ducktape/README.md
        both added:      tests/ducktape/benchmark_bounds.json
        both added:      tests/ducktape/benchmark_metrics.py
        both modified:   tests/ducktape/run_ducktape_test.py
        both modified:   tests/ducktape/test_producer.py

…2017) * updaet * update * tests * fix integration tests * regenerate sync code + fix it * fix some unit tests broken from unasync * missing params and address feedback * lint * revert breaking chamges * update * update * address feedback * remove unused import

* DGS-21988 Fix transformation of nullable JSON props * Minor cleanup

* Support key password for schema registry client. * Remove unused import.

…#2024) * updaet * update * tests * fix integration tests * regenerate sync code + fix it * fix some unit tests broken from unasync * missing params and address feedback * lint * revert breaking chamges * update * update * update * mode * mode * updaste * linter * fix typos * update * add more * update * fix merge conflict errors * update

…CURITY] (master) (#1998) * WARNING: MAJOR (BREAKING) CHANGE: Update dependency urllib3 to v2 [SECURITY] * Fixed security update to ignore 3.6 --------- Co-authored-by: renovatebot-confluentinc[bot] <169726756+renovatebot-confluentinc[bot]@users.noreply.github.com> Co-authored-by: Matthew Seal <mseal@confluent.io>

…ub markdown patterns (#1997) Applied PR feedback Reapply pr feedback Applied PR feedback Addressed more python3 references + review request

* update * lint * address feedback

* Replace all occurrences of 's1-prod-macos-13-5-amd64' with 's1-macos-15-amd64-6' * Replace all occurrences of 's1-prod-macos-13-5-arm64' with 's1-macos-15-arm64-8' --------- Co-authored-by: service-bot-app[bot] <service-bot-app[bot]@users.noreply.github.com> Co-authored-by: service-bot-app[bot] <189278048+service-bot-app[bot]@users.noreply.github.com>

* basic sr test * more tests * update * update * lint

🤖 Generated with [Claude Code](https://claude.ai/code) Co-authored-by: Claude <noreply@anthropic.com>

confluent-cla-assistant · 2025-09-26T11:57:11Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ insightfuls
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

MSeal · 2025-09-26T16:12:44Z

.ducktape/metadata/session_id

@@ -0,0 +1 @@
+2025-08-21--020


Do we need to add .ducktape to the gitignore?

https://github.com/confluentinc/confluent-kafka-python/blob/master/.gitignore#L37C1-L37C11
Seems like we have added.. i will remove file manually

sonarqube-confluent · 2025-09-26T16:34:32Z

77.30% Coverage on New Code (is less than 80.00%)

Analysis Details

47 Issues

3 Bugs
2 Vulnerabilities
42 Code Smells

Coverage and Duplications

77.30% Coverage (66.80% Estimated after merge)
No duplication information (5.00% Estimated after merge)

Project ID: confluent-kafka-python

View in SonarQube

* Asyncio Producer and Consumer implementation with same API as the sync ones * Add unit tests for asyncio producer + consumer (#2036) * update * update * fix producer and add consumer uts * refactor * rename * Add Benchmark Framework for ducktape (#2030) * Integrate Schema Registry with ducktape load tests (#2027) * basic sr test * more tests * update * update * lint * Add ducktape benchmark tests for consumer (sync + async) (#2045) * draft * update and cleanup * add perf comments, add batch_size param to consume test, lint fix * linter fix * Fix linter issues in consumer testing code (#2051) * Add remaining functions missing in async producer & consumer (#2050) * Add semaphore block for ducktape tests (#2037) * Add semaphore block for ducktape tests * Increase kafka start timeout * Increase kafka start timeout * Increase kafka start timeout * Add logs to debug pipeline * Start kafka in kraft mode * Fix directory failures * Fix directory failures * Fix directory failures * templatise path * Fix ductape run * Fix kafka broker listner * Fix ducktape version error * Cleanup * Fix bound voilation should fail tests * Now expand bounds for success * Add schema registry instance * Update Schema Registry hostname * Update Schema Registry hostname * Update Schema Registry hostname * Fix for linux CI environment * Address minor feedback * Fix semaphore * Minor fix after rebase * Add more async consumer unit & integration tests (#2052) * basic rebalance test * rebalance tests * refactor and linter fix * feebdack * refactor and cleanup * update * remove jit imports * Add produce batch api to producer (#2047) * Add integration tests for transactions (#2056) * add tests * cleanup and linter fix * remove jit import * refactor * cleanup * minot rlinter * Update AsyncIO producer architecture to improve performance (#2044) * Fix helper function name to avoid ducktape test discovery * Integrate schema registry with producer sync/async performance test + clean up the old SR test (#2063) * Add comprehensive producer benchmark tests with Schema Registry support - Updated message serialization to use comprehensive structure with all protobuf fields - Implemented proper strategy pattern for sync/async serializers - Added Schema Registry authentication configuration - Fixed JSON serialization issues (schema title, async serializer initialization) - Added performance validation with configurable JSON validation - Enhanced producer strategies with comprehensive Avro, JSON, and Protobuf support * remove * remove confusing msg * Minor: Producer close calls flush() (#2066) * Integrate schema registry with consumer sync/async performance test (#2059) * update * remove auth * cleanup and ensure same msg size * more cleanup * Add comprehensive producer benchmark tests with Schema Registry support - Updated message serialization to use comprehensive structure with all protobuf fields - Implemented proper strategy pattern for sync/async serializers - Added Schema Registry authentication configuration - Fixed JSON serialization issues (schema title, async serializer initialization) - Added performance validation with configurable JSON validation - Enhanced producer strategies with comprehensive Avro, JSON, and Protobuf support * update * Group messages by topic partition before passing to produce_batch API (#2069) * Merge master to async (#2068) * Pre release (#2067) * Attempting to add python versioning to read from project toml and setting beta flag * Updated docs to read project toml version as well * Updated to read from c file for now. Updaed docs and fixed bad AI code * NPI-7572: Add content for AsyncIO Python client (#2070) * Updates for AsyncIO and other improvements * Add updates based on asyncio blog * Add SR updates relatd to AsyncIO * Reorganize content, remove redundancy, and improve content * Edits to diagram and other content * Add why to use this client in both readme files * Improve CHANGELOG title * Add release dates to versions in CHANGELOG * Add release dates back to v2.4.0 * Edits based on feedback * AsyncIO: Only clear messages from buffer if executor passed (#2071) * Fix async producer transaction behavior + add transactional produce benchmark test (#2072) * update * linter fix * Fix the async transaction behavior related to flush() (#2073) * fix * linter * more linter fix * linter and add link * Removed very old librdkafka version checks * Resolved admin import conflict issue * Fix test_version unit test (#2079) * Fix broken tests (#2077) * fix tests * fix linter * Removed set operation from test --------- Co-authored-by: Matthew Seal <mseal@confluent.io> * Async fix buffer cleanup (#2078) * Fix buffer cleanup logic * Add tests * fix linter * Remove SR key * Removed incorrect assert * Change ducktape tests to install more dependencies * Fix semaphore for producer ducktape tests + clean up files that should've been removed (#2081) * update * use warning for producer validate * remove unnecessary assert --------- Co-authored-by: Emanuele Sabellico <esabellico@confluent.io> Co-authored-by: Matthew Seal <mseal@confluent.io> Co-authored-by: Kaushik Raina <103954755+k-raina@users.noreply.github.com> Co-authored-by: Matthew Seal <mseal007@gmail.com> Co-authored-by: Steve Bang <sbang@confluent.io>

rayokota and others added 14 commits August 19, 2025 11:20

First cut (#2018)

c15fdf8

DGS-21988 Fix transformation of nullable JSON props (#2032)

8724197

* DGS-21988 Fix transformation of nullable JSON props * Minor cleanup

Support key password for schema registry client. (#2033)

58e6bee

* Support key password for schema registry client. * Remove unused import.

Add Benchmark Framework for ducktape (#2030)

858e77c

Added contributor.md and updated code block styles to better fix gith…

3e30757

…ub markdown patterns (#1997) Applied PR feedback Reapply pr feedback Applied PR feedback Addressed more python3 references + review request

Handle empty context properly for Avro deserializer (#2035)

97bded0

* update * lint * address feedback

Integrate Schema Registry with ducktape load tests (#2027)

5b54aa6

* basic sr test * more tests * update * update * lint

Add Confluent Cloud promotional banner to README (#2038)

55b5555

🤖 Generated with [Claude Code](https://claude.ai/code) Co-authored-by: Claude <noreply@anthropic.com>

Renamed semaphore per recommendation (#2043)

9ad4e3a

Resolve merge conflicts by accepting async branch changes

778db76

k-raina requested review from a team and MSeal as code owners September 26, 2025 11:56

This comment has been minimized.

Sign in to view

MSeal approved these changes Sep 26, 2025

View reviewed changes

Remove .ducktape

df70e7c

k-raina merged commit 9674882 into async Sep 26, 2025
1 of 3 checks passed

k-raina deleted the merge-master-to-async branch September 26, 2025 16:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Merge master to async #2068

Merge master to async #2068

Uh oh!

k-raina commented Sep 26, 2025

Uh oh!

confluent-cla-assistant bot commented Sep 26, 2025

Uh oh!

This comment has been minimized.

MSeal Sep 26, 2025

Uh oh!

k-raina Sep 26, 2025

Uh oh!

sonarqube-confluent bot commented Sep 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

		@@ -0,0 +1 @@
		2025-08-21--020 No newline at end of file

Merge master to async #2068

Merge master to async #2068

Uh oh!

Conversation

k-raina commented Sep 26, 2025

Uh oh!

confluent-cla-assistant bot commented Sep 26, 2025

Uh oh!

This comment has been minimized.

MSeal Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

k-raina Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

sonarqube-confluent bot commented Sep 26, 2025

Analysis Details

47 Issues

Coverage and Duplications

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants