[reverted]planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. #24516

LittleFall · 2021-05-10T07:04:18Z

What problem does this PR solve?

Issue Number: partially pingcap/tiflash#1766

Problem Summary:

#24157 import a workaround to enforce mpp, and we still need a normalized way.

What is changed and how it works?

Proposal: enforce mpp

What's Changed:

modify tidb_allow_mpp to type enum and expanded tidb_allow_mpp's value range:

0 or "OFF" means never use MPP.
1 or "ON" means selected by optimizer.
2 or "ENFORCE" means use MPP whenever possible.

when tidb_allow_mpp is set 2, change the cost of mppTask to 0.

Related changes

PR to update pingcap/docs/pingcap/docs-cn: need modify
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Manual test (for compatibility)

-- old tidb
set @@global.tidb_allow_mpp = 1;
mysql> select @@tidb_allow_mpp;
+------------------+
| @@tidb_allow_mpp |
+------------------+
|                1 |
+------------------+


use tiup cluster to upgrade to new tidb, no error.
-- new tidb
mysql> select @@tidb_allow_mpp;
+------------------+
| @@tidb_allow_mpp |
+------------------+
| ON               |
+------------------+
mysql> set @@global.tidb_allow_mpp=2;
Query OK, 0 rows affected (0.01 sec)

mysql> select @@global.tidb_allow_mpp;
+-------------------------+
| @@global.tidb_allow_mpp |
+-------------------------+
| ENFORCE                 |
+-------------------------+
1 row in set (0.00 sec)

use tiup cluster to downgrade to old tidb:
...
Upgrading component tidb
        Restarting instance 172.16.5.81

Error: failed to restart: tidb 172.16.5.81:23600, please check the instance's log(/data2/qizhi/qz-500/deploy/tidb-23600/log) for more detail.: timed out waiting for port 23600 to be started after 2m0s

Verbose debug logs has been written to /root/.tiup/logs/tiup-cluster-debug-2021-05-10-16-00-28.log.
Error: run `/root/.tiup/components/cluster/v1.4.2/tiup-cluster` (wd:/root/.tiup/data/SWzczQV) failed: exit status 1

[2021/05/10 16:05:53.149 +08:00] [WARN] [cache.go:299] ["load mysql.user fail"] [error="[variable:1231]Variable 'tidb_allow_mpp' can't be set to the value of 'ENFORCE'"]
[2021/05/10 16:05:53.149 +08:00] [FATAL] [terror.go:276] ["unexpected error"] [error="[privilege:8049]mysql.user"] [stack="github.com/pingcap/parser/terror.MustNil\n\t/root/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20210325072920-0d17053a8a69/terror/terror.go:276\nmain.createStoreAndDomain\n\t/data2/qizhi/tidb/tidb-server/main.go:270\nmain.main\n\t/data2/qizhi/tidb/tidb-server/main.go:183\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"] [stack="github.com/pingcap/parser/terror.MustNil\n\t/root/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20210325072920-0d17053a8a69/terror/terror.go:276\nmain.createStoreAndDomain\n\t/data2/qizhi/tidb/tidb-server/main.go:270\nmain.main\n\t/data2/qizhi/tidb/tidb-server/main.go:183\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"]

Side effects

Breaking backward compatibility
The variable value is expanded, and the test result shows that it can be upgraded, but downgraded will fail after this issue.

If you want to downgrade, you should first set tidb_allow_mpp to 0 or 1.

Release note

planner: support set tidb_allow_mpp to 2 or ENFORCE to enforce use mpp mode.

ti-srebot · 2021-05-10T07:16:08Z

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

LittleFall · 2021-05-10T07:30:10Z

/run-all-tests

c4pt0r · 2021-05-10T07:55:56Z

not sure tidb_allow_mpp accecpts string value, like:

set tidb_allow_mpp='enforce'

set tidb_allow_mapp=2 is not very user-friendly.

LittleFall · 2021-05-10T08:16:19Z

not sure tidb_allow_mpp accecpts string value, like:

set tidb_allow_mpp='enforce'

set tidb_allow_mapp=2 is not very user-friendly.

It can accept both string value and index number, I'll add this to the unit test and docs.

LittleFall · 2021-05-10T08:37:37Z

/run-all-tests

LittleFall · 2021-05-10T08:37:54Z

/cc @bb7133

LittleFall · 2021-05-10T08:39:07Z

/cc @hanfei1991

LittleFall · 2021-05-10T08:39:31Z

/cc @winoros

LittleFall · 2021-05-10T09:35:46Z

/uncc @winoros

LittleFall · 2021-05-10T09:36:11Z

/uncc @planner-reviewers

LittleFall · 2021-05-10T09:38:17Z

/run-all-tests

LittleFall · 2021-05-12T08:52:25Z

/run-all-tests

ti-chi-bot · 2021-05-12T09:26:35Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

hanfei1991
lzmhhh123

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

LittleFall · 2021-05-12T09:40:54Z

/run-check_dev_2

lzmhhh123 · 2021-05-12T09:46:14Z

/merge

ti-chi-bot · 2021-05-12T09:46:19Z

This pull request has been accepted and is ready to merge.

Commit hash: 7d26779

…ingcap#24052) * *: fix revoke statement for CURRENT_USER() and refine error message planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. (pingcap#24516) store/tikv: remove use of SchemaAmender option in store/tikv (pingcap#24408) *: the value of tikv-client.store-liveness-timeout should not less than 0 (pingcap#24244) store/tikv: remove use of EnableAsyncCommit option in store/tikv (pingcap#24462) txn: Add txn state's view (pingcap#22908) planner: ignore lock for temporary table of PointGet and BatchPointGet (pingcap#24540) store/tikv: remove use of ReplicaRead transaction option in store/tikv (pingcap#24409) store/driver: move error to single package (pingcap#24549) ddl: add check table compatibility for temporary table (pingcap#24501) store/tikv: remove use of IsStatenessReadOnly option in store/tikv (pingcap#24464) store/tikv: change backoff type for missed tiflash peer. (pingcap#24577) store/tikv: remove use of MatchStoreLabels transaction option in store/tikv (pingcap#24465) executor, meta: Allocate auto id for global temporary tables (pingcap#24506) store/tikv: remove use of SampleStep option in store/tikv (pingcap#24461) executor: add partition pruning tests for adding and dropping partition operations (pingcap#24573) ddl: forbid partition on temporary mode before put into queue (pingcap#24565) ddl: speedup test case TestIndexOnMultipleGeneratedColumn (pingcap#24487) execution: Fix issue 24439 Inconsistent error with MySQL for GRANT CREATE USER ON <specific db>.* (pingcap#24485) *: fix errcheck (pingcap#24463) test: make TestExtractStartTs stable (pingcap#24585) ddl: forbid recover/flashback temporary tables (pingcap#24518) executor: fix point_get result on clustered index when new-row-format disabled but new-collation enabled (pingcap#24544) executor: Improve the performance of appending not fixed columns (pingcap#20969) *: typo fix (pingcap#24564) planner/core: refresh stale regions in cache for batch cop response (pingcap#24457) binlog: DML on temporary tables do not write binlog (pingcap#24570) store/tikv: remove use of GuaranteeLinearizability option in store/tikv (pingcap#24605) store/tikv: remove use of CollectRuntimeStats option in store/tikv (pingcap#24604) store/tikv: move Backoffer into a single package (pingcap#24525) variables: init cte max recursive deeps in a new session (pingcap#24609) store/tikv: move transaction options out to /kv (pingcap#24619) store/driver: move backoff driver into single package so we can use i… (pingcap#24624) server: close the temporary session in HTTP API to avoid memory leak (pingcap#24339) store/tikv: use latest PD TS plus one as min commit ts (pingcap#24579) planner: fix incorrect TableDual plan built from nulleq (pingcap#24596) ranger: fix the case which could have duplicate ranges (pingcap#24590) executor, store: Pass the SQL digest down to pessimistic lock request (pingcap#24380) kv: remove UnionStore interface (pingcap#24625) *: enable gosimple linter (pingcap#24617) txn: avoid the gc resolving pessimistic locks of ongoing transactions (pingcap#24601) util: fix wrong enum building for index range (pingcap#24632) sessionctx: change innodb large prefix default (pingcap#24555) store: fix data race about KVStore.tikvClient (pingcap#24655) executor, privileges: Add dynamic privileges to SHOW PRIVILEGES (pingcap#24646) ddl: refactor rule [4/6] (pingcap#24007) cmd: ddl_test modify retryCnt from 5 to 20 (pingcap#24662) executor: add correctness tests about direct reading with ORDER BY and LIMIT (pingcap#24455) store/tikv: remove options from unionstore (pingcap#24629) planner: fix wrongly check for update statement (pingcap#24614) store/tikv: remove CompareTS (pingcap#24657) planner, privilege: Add security enhanced mode part 4 (pingcap#24416) executor: add some test cases about partition table dynamic-mode with split-region (pingcap#24665) planner: fix wrong column offsets when processing dynamic pruning for IndexJoin (pingcap#24659) *: Add security enhanced mode part 3 (pingcap#24412) store/tikv: resolve ReplicaReadType dependencies (pingcap#24653) executor: add test cases about partition table with `expression` (pingcap#24628) tablecodec: fix write wrong prefix index value when collation is ascii_bin/latin1_bin (pingcap#24578) *: compatibility with staleread (pingcap#24285) session: test that temporary tables will also be retried (pingcap#24505) domain, session: Add new sysvarcache to replace global values cache (pingcap#24359) ddl, transaction: DDL on temporary tables won't affect transactions (pingcap#24534) *: implement tidb_bounded_staleness built-in function (pingcap#24328) executor: add correctness tests for partition table with different joins (pingcap#24673) expression: fix the spelling of word arithmetical (pingcap#24713) store/copr: balance region for batch cop task (pingcap#24521) store, metrics: Add metrics for safetTS updating (pingcap#24687) sem: add tidbredact log to restricted variables (pingcap#24701) session: fix dml_batch_size doesn't load the global variable (pingcap#24710) store/tikv: retry TSO RPC (pingcap#24682) expression, planner: push cast down to control function with enum type. (pingcap#24542) executor: add correctness tests about IndexMerge (pingcap#24674) variable: change default for DefDMLBatchSize tidbOptInt64 call (pingcap#24697) planner: add partitioning pruning tests for range partitioning (pingcap#24554) *: add option for enum push down (pingcap#24685) txn: break dependency from store/tikv to tidb/kv cause by TransactionOption (pingcap#24656) executor: enhancement for ListInDisk(support writing after reading) (pingcap#24379) kv: move TxnScope into kv (pingcap#24715) execution: fix the incorrect use of cached plan for point get (pingcap#24749) executor: add correctness tests about direct reading with indexJoin (pingcap#24497) variable: fix sysvar datarace with deep copy (pingcap#24732) *: Implementing RENAME USER (pingcap#24413) infoschema, executor: Add the deadlock table (pingcap#24524) docs: Some proposal for renaming and configurations for Lock View (pingcap#24718) planner: add range partition boundaries tests with BETWEEN expression (pingcap#24598) oracle: simplify timestamp utilities (pingcap#24688) executor: fix wrong enum key in point get (pingcap#24618) ranger: fix incorrect enum range for xxx_ci collation (pingcap#24661) executor: add some test cases about dynamic-mode and apply operator (pingcap#24683) store/tikv: remove Variables.Hook (pingcap#24758) ddl: speed up the execution time of `TestBackwardCompatibility`. (pingcap#24704) *: prepare errors for CTE (pingcap#24763) expression: support cast real/int as real (pingcap#24670) executor: add table name in log (pingcap#24666) expression: add builtin function ``json_pretty`` (pingcap#24675) ddl: make `TestDropLastVisibleColumns` stable (pingcap#24790) * ddl: make `TestDropLastVisibleColumns` stable *: support AS OF TIMESTAMP read-only begin statement (pingcap#24740) executor: Fix unstable TestTiDBLastTxnInfoCommitMode (pingcap#24779) planner: add tests for partition range boundaries for LT/GT (pingcap#24574) test: record random seed in TestIssue20658 (pingcap#24782) store/tikv/retry: define Config instead of BackoffType (pingcap#24692) config: ignore tiflash when show config (pingcap#24770) privileges: improve dynamic privs registration and tests (pingcap#24773) README: remove the link of TiDB Monthly Update (pingcap#24791) region_cache: filter peers on tombstone or dropped stores (pingcap#24726) util/stmtsummary: remove import package tikv (pingcap#24776) ddl: grammar check for create unsupported temporary table (pingcap#24723) *: update go.etcd.io/bbolt (pingcap#24799) ddl: speed up the execution time of `ddl test` and `Test Chunk pingcap#7 ddl-other` (pingcap#24780) executor: remove the unnecessary use of fmt.Sprintf (pingcap#24815) executor: fix index join panic on prefix index on some cases (pingcap#24568)

…e mpp mode. (pingcap#24516)

* planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. (#24516) * solve conflicts * fix tests * fix vet * fix test

…e mpp mode. (pingcap#24516)

ti-chi-bot · 2021-08-09T07:13:34Z

@LittleFall: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

LittleFall added 3 commits May 8, 2021 16:45

expand var tidb_allow_mpp values from {true, false} to {0,1,2}

8965030

use typeEnum instead of typeInt to keep compatible with old versions.

746785a

add test

6ed580a

LittleFall requested review from a team as code owners May 10, 2021 07:04

LittleFall requested review from qw4990 and removed request for a team May 10, 2021 07:04

ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 10, 2021

github-actions bot added the sig/execution SIG execution label May 10, 2021

Merge branch 'master' into enforce-mpp

2faaad0

LittleFall changed the title ~~planner: Set tidb_allow_mpp=2 to enforce use mpp mode.~~ planner: support set tidb_allow_mpp=2 to enforce use mpp mode. May 10, 2021

LittleFall changed the title ~~planner: support set tidb_allow_mpp=2 to enforce use mpp mode.~~ planner: support set tidb_allow_mpp=2 or ENFORCE to enforce use mpp mode. May 10, 2021

LittleFall changed the title ~~planner: support set tidb_allow_mpp=2 or ENFORCE to enforce use mpp mode.~~ planner: support set tidb_allow_mpp to 2 or ENFORCE to enforce use mpp mode. May 10, 2021

LittleFall added 2 commits May 10, 2021 16:30

add test for set variable

cca0686

Merge remote-tracking branch 'qizhi/enforce-mpp' into enforce-mpp

1173292

ti-chi-bot requested a review from bb7133 May 10, 2021 08:37

ti-chi-bot requested a review from hanfei1991 May 10, 2021 08:39

ti-chi-bot requested a review from winoros May 10, 2021 08:39

ti-chi-bot removed the request for review from winoros May 10, 2021 09:35

LittleFall and others added 2 commits May 12, 2021 16:50

fix lint

930de0b

Merge branch 'master' into enforce-mpp

7d26779

hanfei1991 approved these changes May 12, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 12, 2021

LittleFall mentioned this pull request May 12, 2021

test random failed in TestExtractStartTs #24583

Closed

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 12, 2021

LittleFall mentioned this pull request May 12, 2021

test failed because of DATA RACE #24586

Closed

ti-chi-bot merged commit ec618f1 into pingcap:master May 12, 2021

LittleFall deleted the enforce-mpp branch May 12, 2021 11:07

LittleFall mentioned this pull request May 12, 2021

add docs for enforce mpp pingcap/docs-cn#6250

Merged

5 tasks

LittleFall mentioned this pull request May 23, 2021

planner: enforce push mpp down #24849

Merged

LittleFall added a commit to LittleFall/tidb that referenced this pull request May 23, 2021

planner: support set tidb_allow_mpp to 2 or ENFORCE to enforce us…

dad1538

…e mpp mode. (pingcap#24516)

zanmato1984 pushed a commit that referenced this pull request May 23, 2021

planner: enforce push mpp down (#24849)

7204906

* planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. (#24516) * solve conflicts * fix tests * fix vet * fix test

LittleFall mentioned this pull request Jun 9, 2021

planner: Revert tidb_allow_mpp modification for downgrade compatibility and add warnings for enforce mpp. #25302

Merged

This was referenced Jun 11, 2021

planner: Revert tidb_allow_mpp modification for downgrade compatibility and add warnings for enforce mpp. (#25302) #25353

Merged

planner: Revert tidb_allow_mpp modification for downgrade compatibility and add warnings for enforce mpp. (#25302) #25358

Closed

LittleFall added a commit to LittleFall/tidb that referenced this pull request Jul 20, 2021

planner: support set tidb_allow_mpp to 2 or ENFORCE to enforce us…

c255971

…e mpp mode. (pingcap#24516)

This was referenced Jul 20, 2021

DNM: planner: cherry-pick enforce-mpp to v5.0 #26378

Closed

planner: cherry-pick enforce-mpp to v5.0 #26382

Merged

LittleFall changed the title ~~planner: support set tidb_allow_mpp to 2 or ENFORCE to enforce use mpp mode.~~ [reverted]planner: support set tidb_allow_mpp to 2 or ENFORCE to enforce use mpp mode. Aug 9, 2021

ti-chi-bot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Aug 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[reverted]planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. #24516

[reverted]planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. #24516

LittleFall commented May 10, 2021 •

edited

ti-srebot commented May 10, 2021

LittleFall commented May 10, 2021

c4pt0r commented May 10, 2021 •

edited

LittleFall commented May 10, 2021 •

edited

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 12, 2021

ti-chi-bot commented May 12, 2021

LittleFall commented May 12, 2021

lzmhhh123 commented May 12, 2021

ti-chi-bot commented May 12, 2021

ti-chi-bot commented Aug 9, 2021

[reverted]planner: support set tidb_allow_mpp to 2 or ENFORCE to enforce use mpp mode. #24516

[reverted]planner: support set tidb_allow_mpp to 2 or ENFORCE to enforce use mpp mode. #24516

Conversation

LittleFall commented May 10, 2021 • edited

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

ti-srebot commented May 10, 2021

LittleFall commented May 10, 2021

c4pt0r commented May 10, 2021 • edited

LittleFall commented May 10, 2021 • edited

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 10, 2021

LittleFall commented May 12, 2021

ti-chi-bot commented May 12, 2021

LittleFall commented May 12, 2021

lzmhhh123 commented May 12, 2021

ti-chi-bot commented May 12, 2021

ti-chi-bot commented Aug 9, 2021

[reverted]planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. #24516

[reverted]planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. #24516

LittleFall commented May 10, 2021 •

edited

c4pt0r commented May 10, 2021 •

edited

LittleFall commented May 10, 2021 •

edited