Gen4: Add UPDATE planning #9871

harshit-gangal · 2022-03-11T12:08:45Z

Description

Uses the gen4 planner infrastructure to plan UPDATE statements. This makes it possible to plan more queries - UPDATE with subqueries are now possible.

Unfortunately we still have the limitation that only a minority of correlated subqueries are possible - only when we can merge the UPDATE with the subquery into a single route can we perform these queries.

Related Issue(s)

#9640

Checklist

Tests were added or are not required
Documentation was added or is not required

Deployment Notes

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1202058565177941

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

… skip] Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

…skip] Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

…p) [ci skip] Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Signed-off-by: Andres Taylor <andres@planetscale.com>

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Signed-off-by: Andres Taylor <andres@planetscale.com>

frouioui

Looks good to me! 👍🏻

go/vt/vtgate/planbuilder/physical/dml_planning.go

frouioui · 2022-04-10T09:22:32Z

go/vt/vtgate/planbuilder/physical/route_planning.go

+		}
+
+		if r.RouteOpCode == engine.Scatter && op.AST.Limit != nil {
+			// TODO systay: we should probably check for other op code types - IN could also hit multiple shards (2022-04-07)


let's create an issue for this for tracking.

Signed-off-by: Andres Taylor <andres@planetscale.com>

…ressions Signed-off-by: Andres Taylor <andres@planetscale.com>

… UNION Signed-off-by: Andres Taylor <andres@planetscale.com>

Signed-off-by: Andres Taylor <andres@planetscale.com>

harshit-gangal

looks good overall.

go/test/endtoend/vtgate/queries/subquery/subquery_test.go

go/vt/vtgate/executor_dml_test.go

go/vt/vtgate/planbuilder/physical/route.go

harshit-gangal · 2022-04-12T16:58:58Z

go/vt/vtgate/planbuilder/physical/route_planning.go

+		}
+
+		if r.RouteOpCode == engine.Scatter && op.AST.Limit != nil {
+			// TODO systay: we should probably check for other op code types - IN could also hit multiple shards (2022-04-07)


let's create an issue for this for tracking.

go/vt/vtgate/planbuilder/primitive_builder.go

go/vt/vtgate/planbuilder/testdata/dml_cases.txt

go/vt/vtgate/planbuilder/testdata/unsupported_cases.txt

Signed-off-by: Andres Taylor <andres@planetscale.com>

harshit-gangal added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Query Serving release notes labels Mar 11, 2022

harshit-gangal mentioned this pull request Mar 29, 2022

add subquery support in update query #9848

Closed

3 tasks

gen4: analyze simple update query

c3d45ab

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

systay force-pushed the gen4-update branch from 066b6af to c3d45ab Compare April 5, 2022 08:10

frouioui force-pushed the gen4-update branch from 9ac2bed to f393075 Compare April 5, 2022 11:05

frouioui added 2 commits April 5, 2022 13:07

feat: addition of gen4UpdateStmtPlanner in the gen4_planner (wip) [ci…

0cd46fc

… skip] Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

feat: support sqlparser.Statement in the semantic analyzer (wip) [ci …

977a1df

…skip] Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

frouioui force-pushed the gen4-update branch from f393075 to 977a1df Compare April 5, 2022 11:08

frouioui and others added 18 commits April 5, 2022 13:46

feat: support unsharded update queries in gen4 without subqueries (wi…

3c40a1e

…p) [ci skip] Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

feat: support sharded update queries in gen4 (wip) [ci skip]

946085f

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

feat: support routed tables for update queries in gen4 (wip) [ci skip]

e956085

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

feat: analyzer nows support subquery in update [ci skip]

30215c3

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

feat: rewrite routed tables everywhere, not just on the updated table

13673f6

Signed-off-by: Andres Taylor <andres@planetscale.com>

test: updated test expectations to match the new error message

c97f166

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: add support for update in the logical operator structure

765f155

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: add update to the physical operator list

29d8854

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: produce a fuller physical operator for updates

148c3e3

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: represent UPDATE as a physical operator

4504582

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: handle single sharded update queries

63b1db2

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: consider all vindexes for routing

1bba4d6

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: support Update's MultiShardAutocommit and QueryTimeout in Gen4

50a81b9

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

feat: begin supporting subqueries in update

84d9e78

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

feat: fix subquery support in gen4's update for sharded subquery

a0213ec

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

feat: fix expected output for multicol vindex update

cfad564

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

feat: merge subqueries with update with literal values

cd7b2ff

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: make it possible to merge operators other than SELECT

ee7d40a

Signed-off-by: Andres Taylor <andres@planetscale.com>

systay added the Skip Upgrade Downgrade label Apr 10, 2022

systay marked this pull request as ready for review April 10, 2022 08:56

systay requested review from systay, frouioui and GuptaManan100 as code owners April 10, 2022 08:56

systay changed the title ~~Gen4: Update statement planner~~ Gen4: Add UPDATE planning Apr 10, 2022

frouioui approved these changes Apr 10, 2022

View reviewed changes

Merge branch main into gen4-update

9931b09

Signed-off-by: Andres Taylor <andres@planetscale.com>

systay force-pushed the gen4-update branch from b43b92c to 9931b09 Compare April 11, 2022 05:08

systay added 7 commits April 11, 2022 07:18

chore: small fixes

d80a4c7

Signed-off-by: Andres Taylor <andres@planetscale.com>

refactor: clean up file

088bab6

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: add locking for the read part of updates with subqueries

92c693f

Signed-off-by: Andres Taylor <andres@planetscale.com>

refactor: clean up operator transformers

66ce4b3

Signed-off-by: Andres Taylor <andres@planetscale.com>

test: update tests now that update goes through semantic analysis

ce3d73c

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: handle extracted subqueries when planning update evalengine exp…

8d863b7

…ressions Signed-off-by: Andres Taylor <andres@planetscale.com>

test: turn off Gen4 vs V3 comparisons on everything except SELECT and…

ae8a27f

… UNION Signed-off-by: Andres Taylor <andres@planetscale.com>

systay force-pushed the gen4-update branch from 47c59c1 to ae8a27f Compare April 12, 2022 09:55

systay added 3 commits April 12, 2022 11:58

Merge branch main into gen4-update

2c49426

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: allow explitly routed UPDATE queries

d9ae23c

Signed-off-by: Andres Taylor <andres@planetscale.com>

chore: update proto definitions

4b42831

Signed-off-by: Andres Taylor <andres@planetscale.com>

harshit-gangal requested review from ajm188, doeg and notfelineit as code owners April 12, 2022 11:34

harshit-gangal commented Apr 12, 2022

View reviewed changes

systay added 3 commits April 13, 2022 10:30

chore: addressed review comments

498ea25

Signed-off-by: Andres Taylor <andres@planetscale.com>

feat: fail on UPDATE with derived table with the proper error

42e2f59

Signed-off-by: Andres Taylor <andres@planetscale.com>

Merge branch main into gen4-update

9c7c8ed

Signed-off-by: Andres Taylor <andres@planetscale.com>

systay approved these changes Apr 13, 2022

View reviewed changes

systay merged commit 085db75 into vitessio:main Apr 13, 2022

systay deleted the gen4-update branch April 13, 2022 13:12

harshit-gangal mentioned this pull request Apr 30, 2022

Single shard update with select returns: Unsupported: subqueries in sharded DML #9640

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gen4: Add UPDATE planning #9871

Gen4: Add UPDATE planning #9871

harshit-gangal commented Mar 11, 2022 •

edited by hallaroo

frouioui left a comment

frouioui Apr 10, 2022

harshit-gangal Apr 12, 2022

harshit-gangal left a comment

harshit-gangal Apr 12, 2022

Gen4: Add UPDATE planning #9871

Gen4: Add UPDATE planning #9871

Conversation

harshit-gangal commented Mar 11, 2022 • edited by hallaroo

Description

Related Issue(s)

Checklist

Deployment Notes

frouioui left a comment

Choose a reason for hiding this comment

frouioui Apr 10, 2022

Choose a reason for hiding this comment

harshit-gangal Apr 12, 2022

Choose a reason for hiding this comment

harshit-gangal left a comment

Choose a reason for hiding this comment

harshit-gangal Apr 12, 2022

Choose a reason for hiding this comment

harshit-gangal commented Mar 11, 2022 •

edited by hallaroo