Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add config param to pass additional parameters to confluent-kafka-python #1505

Merged
merged 10 commits into from
Jun 11, 2024

Conversation

kumaranvpl
Copy link
Contributor

@kumaranvpl kumaranvpl commented Jun 6, 2024

Description

  • Add config param which needs to be passed to confluent-kafka-python
  • Remove ssl_context
  • Raise SetupError if ssl_context is passed
  • Use TypedDict for config
  • Remove unused params from Async wrapper which are added to maintain feature parity with AIOKafka implementation. This is not needed anymore.

Fixes #1405

Type of change

Please delete options that are not relevant.

  • Documentation (typos, code examples, or any documentation updates)
  • Bug fix (a non-breaking change that resolves an issue)
  • New feature (a non-breaking change that adds functionality)
  • Breaking change (a fix or feature that would disrupt existing functionality)
  • This change requires a documentation update

Checklist

  • My code adheres to the style guidelines of this project (scripts/lint.sh shows no errors)
  • I have conducted a self-review of my own code
  • I have made the necessary changes to the documentation
  • My changes do not generate any new warnings
  • I have added tests to validate the effectiveness of my fix or the functionality of my new feature
  • Both new and existing unit tests pass successfully on my local environment by running scripts/test-cov.sh
  • I have ensured that static analysis tests are passing by running scripts/static-analysis.sh
  • I have included code examples to illustrate the modifications

@kumaranvpl kumaranvpl requested a review from Lancetnik June 6, 2024 11:59
@kumaranvpl kumaranvpl marked this pull request as draft June 6, 2024 11:59
@kumaranvpl
Copy link
Contributor Author

@Lancetnik Points to discuss

  1. I am removing ssl_context because it is causing confusion and confluent-kafka-python can't use ssl_context directly. It uses separate individual security params. Shall I raise error if ssl_context is passed ? - https://github.com/airtai/faststream/blob/pass-config-to-confluent/faststream/confluent/security.py#L16

  2. Confluent kafka python accepts huge list of config mentioned here - https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md. To make users' life easier we have added some of those config as producer/consumer params, rest can be sent as part of config dict. When we are merging params and config dict which should take precedence? params or config items? - https://github.com/airtai/faststream/blob/pass-config-to-confluent/faststream/confluent/client.py#L151

@Lancetnik
Copy link
Collaborator

Lancetnik commented Jun 8, 2024

@kumaranvpl your decision looks good to me, but I have some notices:

  1. We should raise RuntimeWarning if user setted up ssl_context option in parse_security method instead of suppressing it

  2. We should create a TypedDict for config option to make it auto-compliteable and much explicit. Also, it should solves problems with another init parameters conflict (but the current merging priority looks good too)

@@ -14,6 +15,9 @@


def parse_security(security: Optional[BaseSecurity]) -> "AnyDict":
if security and isinstance(security.ssl_context, ssl.SSLContext):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, we should use SetupError instead of basic ValueError in wrong broker parameters case

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted. Thanks.

@kumaranvpl kumaranvpl marked this pull request as ready for review June 10, 2024 12:30
Lancetnik
Lancetnik previously approved these changes Jun 10, 2024
Lancetnik
Lancetnik previously approved these changes Jun 10, 2024
Copy link
Collaborator

@Lancetnik Lancetnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm, I think, we can release it

@Lancetnik Lancetnik added this pull request to the merge queue Jun 11, 2024
Merged via the queue into main with commit df2a0a1 Jun 11, 2024
29 checks passed
@Lancetnik Lancetnik deleted the pass-config-to-confluent branch June 11, 2024 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Problems configuring faststream for SASL_SSL security
2 participants