sink: support batch dml with virtual column (#3787)#4299
Conversation
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot
added
do-not-merge/hold
|
This cherry pick PR is for a release branch and has not yet been approved by triage owners. To merge this cherry pick:
DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@wk989898 This PR has conflicts, I have hold it. |
|
@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the sink component by enabling batch DML operations for tables that include virtual columns. It refines the logic for generating batch SQL, ensuring that virtual generated columns are correctly handled and excluded from critical DML components like WHERE clauses and index key constructions. These changes streamline data manipulation for tables with generated columns, improving efficiency and correctness across the system. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request aims to enable batch DML operations for tables with virtual columns, refining the logic in shouldGenBatchSQL and updating SQL generation and data modeling functions to correctly skip virtual columns. While the core logic appears sound, a critical security concern was identified in pkg/sink/sqlmodel/row_change.go where raw database values are logged without redaction during a panic, potentially leaking sensitive information. Additionally, the PR contains unresolved merge conflict markers in pkg/sink/mysql/mysql_writer_dml.go, which will prevent compilation and must be resolved.
| <<<<<<< HEAD | ||
| queryList, argsList = w.generateBatchSQL(eventsInGroup) | ||
| ======= | ||
| if !w.shouldGenBatchSQL(tableInfo, eventsInGroup) { | ||
| queryList, argsList = w.generateNormalSQLs(eventsInGroup) | ||
| } else { | ||
| queryList, argsList = w.generateBatchSQL(eventsInGroup) | ||
| } | ||
| >>>>>>> f420f1aed (sink: support batch dml with virtual column (#3787)) |
There was a problem hiding this comment.
Unresolved merge markers (<<<<<<< HEAD, =======, >>>>>>>) are detected in this block, which will cause compilation errors and indicates an incomplete merge process. The logic for deciding whether to generate batch SQL is duplicated and incorrectly structured due to this conflict. Please resolve this by replacing the entire conflicting block with the intended logic.
if !w.shouldGenBatchSQL(tableInfo, eventsInGroup) {
queryList, argsList = w.generateNormalSQLs(eventsInGroup)
} else {
queryList, argsList = w.generateBatchSQL(eventsInGroup)
}| <<<<<<< HEAD | ||
| ======= | ||
| func (w *Writer) genActiveActiveSQL(tableInfo *common.TableInfo, eventsInGroup []*commonEvent.DMLEvent) ([]string, [][]interface{}) { | ||
| if !w.shouldGenBatchSQL(tableInfo, eventsInGroup) { | ||
| return w.generateActiveActiveNormalSQLs(eventsInGroup) | ||
| } | ||
| return w.generateActiveActiveBatchSQL(eventsInGroup) | ||
| } | ||
|
|
||
| >>>>>>> f420f1aed (sink: support batch dml with virtual column (#3787)) |
There was a problem hiding this comment.
Unresolved merge markers (<<<<<<< HEAD, >>>>>>>) are detected in this block, which will cause compilation errors. This appears to be a merge conflict artifact, and the introduced genActiveActiveSQL function seems unrelated to the PR's scope. This block should be removed to resolve the conflict and maintain the focus of the PR.
func (w *Writer) genActiveActiveSQL(tableInfo *common.TableInfo, eventsInGroup []*commonEvent.DMLEvent) ([]string, [][]interface{}) {
if !w.shouldGenBatchSQL(tableInfo, eventsInGroup) {
return w.generateActiveActiveNormalSQLs(eventsInGroup)
}
return w.generateActiveActiveBatchSQL(eventsInGroup)
}| log.Panic("columnNames are not equal columnValues", zap.Int("len(columnNames)", len(columnNames)), zap.Int("len(values)", len(columnValues)), | ||
| zap.Any("columnNames", columnNames), zap.Any("columnValues", columnValues), zap.Any("table", r.targetTable)) |
There was a problem hiding this comment.
The log.Panic call logs raw column values (columnValues) which may contain sensitive information (PII). In production environments, logging raw database values can lead to data leakage in log files. Use util.RedactAny to mask sensitive data before logging.
| log.Panic("columnNames are not equal columnValues", zap.Int("len(columnNames)", len(columnNames)), zap.Int("len(values)", len(columnValues)), | |
| zap.Any("columnNames", columnNames), zap.Any("columnValues", columnValues), zap.Any("table", r.targetTable)) | |
| log.Panic("columnNames are not equal columnValues", zap.Int("len(columnNames)", len(columnNames)), zap.Int("len(columnValues)", len(columnValues)), | |
| zap.Any("columnNames", columnNames), zap.Any("columnValues", util.RedactAny(columnValues)), zap.Any("table", r.targetTable)) |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: wk989898 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
wk989898
removed
the
do-not-merge/hold
|
@ti-chi-bot: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
This is an automated cherry-pick of #3787
What problem does this PR solve?
Issue Number: close #3294 close #4184
What is changed and how it works?
Check List
Tests
Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note
Summary by CodeRabbit
Bug Fixes
New Features
Refactor
Tests