dynamicstream: support batch bytes and make it configurable at the changefeed level #3992
Conversation
ti-chi-bot
bot
added
do-not-merge/needs-linked-issue
release-note
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
ti-chi-bot
bot
added
the
size/M
Summary of ChangesHello @3AceShowHand, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a mechanism to enhance the dynamic stream's batching performance by allowing individual sinks to report their optimal batch sizes, both in terms of event count and byte volume. This change provides greater flexibility and efficiency in how data is processed and delivered to various downstream targets, especially benefiting cloud storage operations with byte-aware batching. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces BatchCount() and BatchBytes() methods to the Sink interface, aiming to allow dynamicstream to configure batching. The implementations for various sinks are provided. While the goal is to make batching configurable, most implementations currently use hardcoded values. I assume this is a first step and configurability will be added in a subsequent change. My review includes suggestions to improve maintainability by replacing magic numbers with constants, fixing a typo, and addressing inconsistencies in batch size definitions.
| func (s *sink) BatchCount() int { | ||
| return 4096 | ||
| } |
There was a problem hiding this comment.
This method returns a hardcoded value of 4096, but the existing batchSize constant in this file is 2048. This inconsistency can be confusing and may lead to suboptimal batching behavior. It would be better to use the existing batchSize constant. If 4096 is intentional, a comment explaining the difference is needed.
| func (s *sink) BatchCount() int { | |
| return 4096 | |
| } | |
| func (s *sink) BatchCount() int { | |
| return batchSize | |
| } |
| func (s *sink) BatchCount() int { | ||
| return 4096 | ||
| } |
There was a problem hiding this comment.
This method returns a hardcoded value of 4096, while the batchSize constant in this file is 2048. This inconsistency can be confusing and might affect batching efficiency. Please consider using the batchSize constant here for consistency. If 4096 is the correct value, please add a comment explaining why it differs from batchSize.
| func (s *sink) BatchCount() int { | |
| return 4096 | |
| } | |
| func (s *sink) BatchCount() int { | |
| return batchSize | |
| } |
| func (s *sink) BatchCount() int { | ||
| return 4096 | ||
| } |
There was a problem hiding this comment.
The magic number 4096 is used here and also for the size of eventCh on line 33. To improve readability and maintainability, it would be better to define this value as a named constant, for example: const defaultBatchCount = 4096.
| func (s *sink) BatchCount() int { | ||
| return 4096 | ||
| } |
There was a problem hiding this comment.
The magic number 4096 is hardcoded here. It's recommended to define it as a named constant to clarify its purpose and make it easier to manage in the future. For example: const defaultBatchCount = 4096.
| Close(removeChangefeed bool) | ||
| Run(ctx context.Context) error | ||
|
|
||
| // BatchCount and BatchBytes are used by the dynamic stream to achive better batch performance. |
There was a problem hiding this comment.
There's a typo in the comment. achive should be achieve.
| // BatchCount and BatchBytes are used by the dynamic stream to achive better batch performance. | |
| // BatchCount and BatchBytes are used by the dynamic stream to achieve better batch performance. |
|
[FORMAT CHECKER NOTIFICATION] Notice: To remove the 📖 For more info, you can check the "Contribute Code" section in the development guide. |
ti-chi-bot
bot
added
size/L
|
@3AceShowHand: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
What problem does this PR solve?
Issue Number: close #xxx
What is changed and how it works?
Check List
Tests
Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note