-
Notifications
You must be signed in to change notification settings - Fork 555
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: allow configure message.timeout.ms and max.in.flight for kafka sink #12574
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
exposing batch.size
is nice to have option
1MB looks preferable to me because we prefer high throughput over low latency. We used to set it to 20MB in production so 1MB is reasonable in our case.
The |
|
Codecov Report
@@ Coverage Diff @@
## main #12574 +/- ##
=======================================
Coverage 69.41% 69.41%
=======================================
Files 1469 1469
Lines 240680 240684 +4
=======================================
+ Hits 167064 167067 +3
- Misses 73616 73617 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 4 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
message.timeout.ms
andmax.in.flight.requests.per.connection
for kafka sink.max.in.flight.requests.per.connection
to 5, which is recommended in the confluent doc. Note that the previous default set by librdkafka is1000000
, which is not reasonable since it will create excessive produce requests that can potentially overload kafka broker and may cause a large number of duplicates on network blips.timeout
anduse_transaction
Checklist
./risedev check
(or alias,./risedev c
)Documentation
Release note
message.timeout.ms
andmax.in.flight.requests.per.connection
for kafka sink.max.in.flight.requests.per.connection
to 5.See here for the meaning of these kafka client configs.