planner/core: update missing virtual columns in update and insert (#58401)#66878
Conversation
ti-chi-bot
added
do-not-merge/hold
|
@joechenrh This PR has conflicts, I have hold it. |
|
@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
📝 WalkthroughWalkthroughThis pull request fixes a bug where UPDATE and INSERT ON DUPLICATE KEY UPDATE statements fail to properly update indexes containing generated columns. The fix modifies the planner's column dependency tracking to detect when dependent columns with OnUpdateNow flags may be modified and propagates this information during ON DUPLICATE KEY UPDATE processing. Tests are added to validate the corrections. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.11.3)Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot
force-pushed
the
cherry-pick-58401-to-release-7.5
branch
from
179b81f to
89b54b6
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
pkg/executor/update_test.go (1)
695-697: Avoid wall-clock sleep in this regression test.
SELECT sleep(1)makes the test slower and timing-dependent. You can force distinctCURRENT_TIMESTAMPvalues deterministically with the sessiontimestampvariable and keep the same coverage.♻️ Suggested change
- tk.MustExec("INSERT INTO ttime SET id = 1") - tk.MustExec("SELECT sleep(1)") - tk.MustExec("UPDATE ttime SET id = 2") + tk.MustExec("SET @@timestamp = 1700000000") + tk.MustExec("INSERT INTO ttime SET id = 1") + tk.MustExec("SET @@timestamp = 1700000001") + tk.MustExec("UPDATE ttime SET id = 2") + tk.MustExec("SET @@timestamp = 0")As per coding guidelines,
**/{*_test.go,testdata/**}: Keep test changes minimal and deterministic; avoid broad golden/testdata churn unless required.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/executor/update_test.go` around lines 695 - 697, Replace the wall-clock sleep between the two statements: instead of tk.MustExec("SELECT sleep(1)"), set the session timestamp variable to a different deterministic value so CURRENT_TIMESTAMP advances (e.g. call tk.MustExec("SET @@session.timestamp = <new_unix_ts>") or similar) between tk.MustExec("INSERT INTO ttime SET id = 1") and tk.MustExec("UPDATE ttime SET id = 2"); this keeps the test deterministic and fast while preserving coverage of CURRENT_TIMESTAMP changes (refer to tk.MustExec and the session 'timestamp' variable).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@pkg/planner/core/planbuilder.go`:
- Around line 3883-3892: Remove the merge conflict markers and adopt the HEAD
signature for resolveGeneratedColumns: use the local type LogicalPlan (no
package prefix) for the mockPlan parameter and keep the original comment
wording; ensure the function declaration matches "func (b *PlanBuilder)
resolveGeneratedColumns(ctx context.Context, columns []*table.Column, onDups
map[string]struct{}, mockPlan LogicalPlan) (igc InsertGeneratedColumns, err
error)" and delete the alternate "base.LogicalPlan" variant so the file
compiles.
---
Nitpick comments:
In `@pkg/executor/update_test.go`:
- Around line 695-697: Replace the wall-clock sleep between the two statements:
instead of tk.MustExec("SELECT sleep(1)"), set the session timestamp variable to
a different deterministic value so CURRENT_TIMESTAMP advances (e.g. call
tk.MustExec("SET @@session.timestamp = <new_unix_ts>") or similar) between
tk.MustExec("INSERT INTO ttime SET id = 1") and tk.MustExec("UPDATE ttime SET id
= 2"); this keeps the test deterministic and fast while preserving coverage of
CURRENT_TIMESTAMP changes (refer to tk.MustExec and the session 'timestamp'
variable).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 498286bd-37de-45ff-9ae8-784a50947129
📒 Files selected for processing (4)
pkg/executor/insert_test.gopkg/executor/update_test.gopkg/planner/core/logical_plan_builder.gopkg/planner/core/planbuilder.go
Signed-off-by: Ruihao Chen <joechenrh@gmail.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@tests/realtikvtest/importintotest/import_into_test.go`:
- Around line 253-267: The current TestOnUpdateColumn doesn't exercise the
planner/core generated-column transitive OnUpdateNow bug because it uses IMPORT
INTO; rewrite TestOnUpdateColumn to create a table with a primary key (or unique
index), a column with ON UPDATE CURRENT_TIMESTAMP, and a dependent VIRTUAL
generated column that is indexed (to trigger the dependency path), then perform
an UPDATE or INSERT ... ON DUPLICATE KEY UPDATE that changes the ON UPDATE
column (use s.tk.MustExec to run the statement), assert the row values
(s.tk.MustQuery) and run ADMIN CHECK TABLE to validate integrity; ensure the
test uses the same setup helpers (s.prepareAndUseDB, s.server.CreateObject if
needed) and that the sequence fails on the pre-fix behavior and passes after the
fix.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: b0d97fb6-1468-405a-b555-fbd44c46b136
📒 Files selected for processing (3)
pkg/planner/core/logical_plan_builder.gopkg/planner/core/planbuilder.gotests/realtikvtest/importintotest/import_into_test.go
|
/retest |
|
/retest |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## release-7.5 #66878 +/- ##
================================================
Coverage ? 72.2022%
================================================
Files ? 1422
Lines ? 415289
Branches ? 0
================================================
Hits ? 299848
Misses ? 95391
Partials ? 20050
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
joechenrh
removed
the
do-not-merge/hold
ti-chi-bot
Bot
added
cherry-pick-approved
Signed-off-by: Ruihao Chen <joechenrh@gmail.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
pkg/executor/test/issuetest/executor_issue_test.go (1)
776-810: LGTM! Test properly validates generated column index consistency fix.The test covers both scenarios from issue
#58400:
tmvtable: Tests INSERT ... ON DUPLICATE KEY UPDATE where a virtual columnj2depends on updated columnj1.ttimetable: Tests chained virtual column dependencies (t2→t1,t3→t2,t4→t2) withON UPDATE CURRENT_TIMESTAMPsemantics.Both are validated with
admin check tableunder both fast and slow table check modes, ensuring index consistency.💡 Optional: Add DROP TABLE statements for consistency with other tests
Other tests in this file include
DROP TABLE IF EXISTSbefore creating tables. While not strictly necessary with a fresh mock store, adding them would improve consistency:func TestInsertDuplicateToGeneratedColumns(t *testing.T) { store := testkit.CreateMockStore(t) tk := testkit.NewTestKit(t, store) tk.MustExec("use test") + tk.MustExec("drop table if exists tmv, ttime") tk.MustExec(` CREATE TABLE tmv (🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/executor/test/issuetest/executor_issue_test.go` around lines 776 - 810, Add DROP TABLE IF EXISTS statements for tables used in TestInsertDuplicateToGeneratedColumns to match the surrounding tests' pattern; specifically, before the CREATE TABLE tmv and CREATE TABLE ttime statements in the TestInsertDuplicateToGeneratedColumns function, insert DROP TABLE IF EXISTS tmv and DROP TABLE IF EXISTS ttime respectively so the test is consistent and resilient even if run in non-fresh environments.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@pkg/executor/test/issuetest/executor_issue_test.go`:
- Around line 776-810: Add DROP TABLE IF EXISTS statements for tables used in
TestInsertDuplicateToGeneratedColumns to match the surrounding tests' pattern;
specifically, before the CREATE TABLE tmv and CREATE TABLE ttime statements in
the TestInsertDuplicateToGeneratedColumns function, insert DROP TABLE IF EXISTS
tmv and DROP TABLE IF EXISTS ttime respectively so the test is consistent and
resilient even if run in non-fresh environments.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: e70a4ef2-7880-417e-afff-aa77dc3b8b57
📒 Files selected for processing (2)
pkg/executor/test/issuetest/BUILD.bazelpkg/executor/test/issuetest/executor_issue_test.go
✅ Files skipped from review due to trivial changes (1)
- pkg/executor/test/issuetest/BUILD.bazel
ti-chi-bot
Bot
added
lgtm
and removed
needs-1-more-lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Benjamin2037, qw4990 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
|
/retest |
1 similar comment
|
/retest |
|
/test? |
|
@joechenrh: The following commands are available to trigger required jobs: The following commands are available to trigger optional jobs: Use DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@joechenrh: No presubmit jobs available for pingcap/tidb@release-7.5 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test check-dev2 |
|
@joechenrh: No presubmit jobs available for pingcap/tidb@release-7.5 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/retest |
1 similar comment
|
/retest |
4 participants
This is an automated cherry-pick of #58401
What problem does this PR solve?
Issue Number: close #58400
Problem Summary:
In #58494, we fixed a problem related to virtual columns. But we haven't completely fixed it yet. There still have problems with virtual columns.
For Update:
In
updatestatements, virtual columns will be added intoUpdateListsif their dependent columns have changed. #55829 has fixed nested generated column.tidb/pkg/planner/core/logical_plan_builder.go
Lines 5826 to 5837 in f2db9c4
However, we still ignore the columns with OnUpdateNow flag when checking dependency. Since these columns may be updated in the execution stage too.
For Insert:
For SQL like
INSERT INTO ... ON DUPLICATE KEY UPDATE ..., only virtual columns that directly depend on the assignments afterUPDATEwill be updated. Take the table in #58400 as example:The dependencies of each column are listed below. Only
j2and_V$_i1_0are updated since they depends onj1. We should handle it the same way as what we do in #55829.What changed and how does it work?
In summary, when extracting generated columns, we have to consider both chain dependencies and on-update-now flag too.
Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.
Summary by CodeRabbit
Bug Fixes
Tests