Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(dlq): Policy closure added + Producer closed #68

Merged
merged 3 commits into from
May 18, 2022

Conversation

rahul-kumar-saini
Copy link
Contributor

@rahul-kumar-saini rahul-kumar-saini commented May 17, 2022

Overview

  • The produce policy takes in a producer which is never closed, this leaves the consumer hanging on shutdown
    • Added self.__producer.close() to the produce policy join method, this fixes the hanging issue
  • The produce policy takes in a producer which is not recreated upon consumer rebalancing
    • Introduced the policy closure to the Consumer Strategy Factory in order to recreate the policy entirely every time the create() method is called (eg when rebalancing)
  • Introduced close() and terminate() methods to Policies to be used when DLQ is closed/terminated

@rahul-kumar-saini rahul-kumar-saini marked this pull request as ready for review May 17, 2022 19:45
@rahul-kumar-saini rahul-kumar-saini requested a review from a team as a code owner May 17, 2022 19:45
Copy link
Contributor

@fpacifici fpacifici left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see the comment inline

@@ -61,9 +64,17 @@ def _produce(self, payload: KafkaPayload) -> None:
)

def join(self, timeout: Optional[float] = None) -> None:
self.__producer.close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please Close this after you have waited on all futures.
Now the producer may still be able to send you the callbacks after it was closed, but this is an implementation detail that may change.
It is generally a safer option to close a resource when you are done using it.

@rahul-kumar-saini rahul-kumar-saini merged commit 637e54d into main May 18, 2022
@rahul-kumar-saini rahul-kumar-saini deleted the fix/dlq/close-producer branch May 18, 2022 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants