Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question: what would happen if produce a mount of messages by an async Writer to a Kafka cluster without response? #831

Closed
3AceShowHand opened this issue Jan 26, 2022 · 3 comments

Comments

@3AceShowHand
Copy link
Contributor

3AceShowHand commented Jan 26, 2022

Describe the solution you'd like
A clear and concise description of what you want to happen.

Consider the following steps

  • Use an async Writer to write messages to a one node kafka cluster,this will buffer messages in the Writer I think.
  • Before the Writer flush all messages to the kafka,kill -s STOP the kafka broker,this mocks that the broker can not send response
  • The Writer should receive errors quickly
  • User application code try to close the Writer by receiving the error

**Expected Behavior

Before the Writer return from Close(), it may try to flush all messages to the kafka,but now it will cost a lot of time, and finally failed, just as what we meet by using another kafka client library, which blocked more than one hour without returned.

What we really expected is, provide the ability that when the Close() called, drop all buffered messages, and return as soon as possible, to prevent block the application.

Hope get response from the community, thanks.

@3AceShowHand
Copy link
Contributor Author

For the current implementation,is there any way to prevent close blocked a long time?

@3AceShowHand 3AceShowHand changed the title Question: what would happen if produce amount of messages by an async Writer to a Kafka cluster without response? Question: what would happen if produce a mount of messages by an async Writer to a Kafka cluster without response? Jan 26, 2022
@rhansen2 rhansen2 assigned rhansen2 and dominicbarnes and unassigned rhansen2 Jan 28, 2022
@dominicbarnes
Copy link
Contributor

Looking over the recent commit history, it looks like #805 included a fix to a problem with closing async writers that sounds like it could be related to what you are describing here. This fix was deployed with v0.4.26 only a few weeks ago, so if you are not already on this version, would you mind trying to upgrade to see if that fixes?

If you were already using at least v0.4.26 or even after upgrading you are still experiencing this problem, we can troubleshoot further. (it may be possible to create a reproducible test case out of this if it's needed)

@dominicbarnes
Copy link
Contributor

I'm closing this issue as we haven't received a response for some time, feel free to re-open if you have more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants