expression, planner: push cast down to control function with enum type. #24542

wshwsh12 · 2021-05-11T05:52:25Z

What problem does this PR solve?

Issue Number: close #23114

Problem Summary: Control function with enum type gets wrong result.

What is changed and how it works?

Proposal: xxx

What's Changed:

Try to push down cast funtion into control function(if/case..when/elt)
(I have a test in mysql, ifnull(e,e) result is always string, and when compare with int, it will convert string to float)
Add cast to double for where condition if the eval type is ETString and the expression is control function with enum.

How it Works:

Related changes

Check List

Tests

Unit test
Integration test

Side effects

Release note

Fix wrong result for control function with enum type.

expression/builtin_cast.go

planner/core/logical_plan_builder.go

expression/integration_test.go

expression/builtin_cast.go

expression/integration_test.go

XuHuaiyu · 2021-05-17T13:14:13Z

expression/builtin_cast.go

+		if tp.EvalType() != types.ETJson {
+			res = FoldConstant(res)
+		}
+		return res


BuildCastFunc will also be used for explicit cast, if we return here, we may build a wrong scalar function.
e.g. select cast(if ()) from t ;

XuHuaiyu · 2021-05-17T13:24:51Z

expression/builtin_cast.go

+
+	args := sf.GetArgs()
+	switch sf.FuncName.L {
+	case ast.If:


what about coalesce?

I have a test. In mysql, coalesce(e) output is always string, and when compare with int, it will convert string to float

expression/builtin_cast.go

XuHuaiyu · 2021-05-18T06:58:26Z

expression/builtin_cast.go

+
+// TryPushCastDownToControlFunctionForHybridType try to push cast down to control function for Hybrid Type.
+// If necessary, it will rebuild control function using changed args.
+// When a hybrid type is the output of a control function, the result may be as a numeric type to subsequent calculation


This comment is hard to understand. We can use an example here to explain.

XuHuaiyu · 2021-05-18T07:10:53Z

planner/core/logical_plan_builder.go

+				Decimal: types.UnspecifiedLength,
+			}
+			types.SetBinChsClnFlag(tp)
+			if res, ok := expression.TryPushCastDownToControlFunctionForHybridType(b.ctx, expr, tp); ok {


What if the expr is cast(if as string) here?

It will push cast as string into if function.

But the expr.FuncName is Cast.

I got it wrong...
If the expr is cast(if as string), the res will be cast(if as string).

XuHuaiyu · 2021-05-18T07:13:12Z

expression/builtin_cast.go

+			if err != nil {
+				return expr, false
+			}
+			sf.RetType, sf.Function = f.getRetTp(), f


Why not return f, true ?

f is a builtinFunc, it is a field in ScalarFunction

XuHuaiyu · 2021-05-18T07:17:02Z

expression/builtin_cast.go

+		}
+		sf.RetType, sf.Function = f.getRetTp(), f
+		// Elt only supports ETString, so we need add extra cast to keep retType right.
+		return wrapCastFunc(ctx, sf), true


The invoker will wrap a cast, why do we need to do this here?

Ok, I remove this.

XuHuaiyu · 2021-05-18T09:39:14Z

planner/core/logical_plan_builder.go

+		if expr.GetType().EvalType() == types.ETString {
+			tp := &types.FieldType{
+				Tp:      mysql.TypeDouble,
+				Flag:    expr.GetType().Flag & mysql.UnsignedFlag,


Why add unsignedFlag?

XuHuaiyu · 2021-05-18T09:41:11Z

expression/builtin_cast.go

+// For example, the condition `if(1, e, 'a') = 1`, `if` function will output `e` and compare with `1`.
+// If the evaltype is ETString, it will get wrong result. So we can rewrite the condition to
+// `IfInt(1, cast(e as int), cast('a' as int)) = 1` to get the correct result.
+func TryPushCastIntoControlFunctionForHybridType(ctx sessionctx.Context, expr Expression, tp *types.FieldType) (res Expression, success bool) {


succes is useless

XuHuaiyu · 2021-05-18T09:43:11Z

planner/core/logical_plan_builder.go

+				Decimal: types.UnspecifiedLength,
+			}
+			types.SetBinChsClnFlag(tp)
+			if res, ok := expression.TryPushCastIntoControlFunctionForHybridType(b.ctx, expr, tp); ok {


Elt will not be wrapped a cast as string for now, will it cause an error.

XuHuaiyu · 2021-05-18T13:32:03Z

/lgtm

ti-chi-bot · 2021-05-18T13:32:06Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

XuHuaiyu
lzmhhh123

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

wshwsh12 · 2021-05-18T15:54:44Z

/merge

ti-chi-bot · 2021-05-18T15:54:49Z

This pull request has been accepted and is ready to merge.

Commit hash: 2a44f22

ti-chi-bot · 2021-05-18T17:46:52Z

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

ti-chi-bot · 2021-05-18T19:50:52Z

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

ti-chi-bot · 2021-05-18T22:25:52Z

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

…ingcap#24052) * *: fix revoke statement for CURRENT_USER() and refine error message planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. (pingcap#24516) store/tikv: remove use of SchemaAmender option in store/tikv (pingcap#24408) *: the value of tikv-client.store-liveness-timeout should not less than 0 (pingcap#24244) store/tikv: remove use of EnableAsyncCommit option in store/tikv (pingcap#24462) txn: Add txn state's view (pingcap#22908) planner: ignore lock for temporary table of PointGet and BatchPointGet (pingcap#24540) store/tikv: remove use of ReplicaRead transaction option in store/tikv (pingcap#24409) store/driver: move error to single package (pingcap#24549) ddl: add check table compatibility for temporary table (pingcap#24501) store/tikv: remove use of IsStatenessReadOnly option in store/tikv (pingcap#24464) store/tikv: change backoff type for missed tiflash peer. (pingcap#24577) store/tikv: remove use of MatchStoreLabels transaction option in store/tikv (pingcap#24465) executor, meta: Allocate auto id for global temporary tables (pingcap#24506) store/tikv: remove use of SampleStep option in store/tikv (pingcap#24461) executor: add partition pruning tests for adding and dropping partition operations (pingcap#24573) ddl: forbid partition on temporary mode before put into queue (pingcap#24565) ddl: speedup test case TestIndexOnMultipleGeneratedColumn (pingcap#24487) execution: Fix issue 24439 Inconsistent error with MySQL for GRANT CREATE USER ON <specific db>.* (pingcap#24485) *: fix errcheck (pingcap#24463) test: make TestExtractStartTs stable (pingcap#24585) ddl: forbid recover/flashback temporary tables (pingcap#24518) executor: fix point_get result on clustered index when new-row-format disabled but new-collation enabled (pingcap#24544) executor: Improve the performance of appending not fixed columns (pingcap#20969) *: typo fix (pingcap#24564) planner/core: refresh stale regions in cache for batch cop response (pingcap#24457) binlog: DML on temporary tables do not write binlog (pingcap#24570) store/tikv: remove use of GuaranteeLinearizability option in store/tikv (pingcap#24605) store/tikv: remove use of CollectRuntimeStats option in store/tikv (pingcap#24604) store/tikv: move Backoffer into a single package (pingcap#24525) variables: init cte max recursive deeps in a new session (pingcap#24609) store/tikv: move transaction options out to /kv (pingcap#24619) store/driver: move backoff driver into single package so we can use i… (pingcap#24624) server: close the temporary session in HTTP API to avoid memory leak (pingcap#24339) store/tikv: use latest PD TS plus one as min commit ts (pingcap#24579) planner: fix incorrect TableDual plan built from nulleq (pingcap#24596) ranger: fix the case which could have duplicate ranges (pingcap#24590) executor, store: Pass the SQL digest down to pessimistic lock request (pingcap#24380) kv: remove UnionStore interface (pingcap#24625) *: enable gosimple linter (pingcap#24617) txn: avoid the gc resolving pessimistic locks of ongoing transactions (pingcap#24601) util: fix wrong enum building for index range (pingcap#24632) sessionctx: change innodb large prefix default (pingcap#24555) store: fix data race about KVStore.tikvClient (pingcap#24655) executor, privileges: Add dynamic privileges to SHOW PRIVILEGES (pingcap#24646) ddl: refactor rule [4/6] (pingcap#24007) cmd: ddl_test modify retryCnt from 5 to 20 (pingcap#24662) executor: add correctness tests about direct reading with ORDER BY and LIMIT (pingcap#24455) store/tikv: remove options from unionstore (pingcap#24629) planner: fix wrongly check for update statement (pingcap#24614) store/tikv: remove CompareTS (pingcap#24657) planner, privilege: Add security enhanced mode part 4 (pingcap#24416) executor: add some test cases about partition table dynamic-mode with split-region (pingcap#24665) planner: fix wrong column offsets when processing dynamic pruning for IndexJoin (pingcap#24659) *: Add security enhanced mode part 3 (pingcap#24412) store/tikv: resolve ReplicaReadType dependencies (pingcap#24653) executor: add test cases about partition table with `expression` (pingcap#24628) tablecodec: fix write wrong prefix index value when collation is ascii_bin/latin1_bin (pingcap#24578) *: compatibility with staleread (pingcap#24285) session: test that temporary tables will also be retried (pingcap#24505) domain, session: Add new sysvarcache to replace global values cache (pingcap#24359) ddl, transaction: DDL on temporary tables won't affect transactions (pingcap#24534) *: implement tidb_bounded_staleness built-in function (pingcap#24328) executor: add correctness tests for partition table with different joins (pingcap#24673) expression: fix the spelling of word arithmetical (pingcap#24713) store/copr: balance region for batch cop task (pingcap#24521) store, metrics: Add metrics for safetTS updating (pingcap#24687) sem: add tidbredact log to restricted variables (pingcap#24701) session: fix dml_batch_size doesn't load the global variable (pingcap#24710) store/tikv: retry TSO RPC (pingcap#24682) expression, planner: push cast down to control function with enum type. (pingcap#24542) executor: add correctness tests about IndexMerge (pingcap#24674) variable: change default for DefDMLBatchSize tidbOptInt64 call (pingcap#24697) planner: add partitioning pruning tests for range partitioning (pingcap#24554) *: add option for enum push down (pingcap#24685) txn: break dependency from store/tikv to tidb/kv cause by TransactionOption (pingcap#24656) executor: enhancement for ListInDisk(support writing after reading) (pingcap#24379) kv: move TxnScope into kv (pingcap#24715) execution: fix the incorrect use of cached plan for point get (pingcap#24749) executor: add correctness tests about direct reading with indexJoin (pingcap#24497) variable: fix sysvar datarace with deep copy (pingcap#24732) *: Implementing RENAME USER (pingcap#24413) infoschema, executor: Add the deadlock table (pingcap#24524) docs: Some proposal for renaming and configurations for Lock View (pingcap#24718) planner: add range partition boundaries tests with BETWEEN expression (pingcap#24598) oracle: simplify timestamp utilities (pingcap#24688) executor: fix wrong enum key in point get (pingcap#24618) ranger: fix incorrect enum range for xxx_ci collation (pingcap#24661) executor: add some test cases about dynamic-mode and apply operator (pingcap#24683) store/tikv: remove Variables.Hook (pingcap#24758) ddl: speed up the execution time of `TestBackwardCompatibility`. (pingcap#24704) *: prepare errors for CTE (pingcap#24763) expression: support cast real/int as real (pingcap#24670) executor: add table name in log (pingcap#24666) expression: add builtin function ``json_pretty`` (pingcap#24675) ddl: make `TestDropLastVisibleColumns` stable (pingcap#24790) * ddl: make `TestDropLastVisibleColumns` stable *: support AS OF TIMESTAMP read-only begin statement (pingcap#24740) executor: Fix unstable TestTiDBLastTxnInfoCommitMode (pingcap#24779) planner: add tests for partition range boundaries for LT/GT (pingcap#24574) test: record random seed in TestIssue20658 (pingcap#24782) store/tikv/retry: define Config instead of BackoffType (pingcap#24692) config: ignore tiflash when show config (pingcap#24770) privileges: improve dynamic privs registration and tests (pingcap#24773) README: remove the link of TiDB Monthly Update (pingcap#24791) region_cache: filter peers on tombstone or dropped stores (pingcap#24726) util/stmtsummary: remove import package tikv (pingcap#24776) ddl: grammar check for create unsupported temporary table (pingcap#24723) *: update go.etcd.io/bbolt (pingcap#24799) ddl: speed up the execution time of `ddl test` and `Test Chunk pingcap#7 ddl-other` (pingcap#24780) executor: remove the unnecessary use of fmt.Sprintf (pingcap#24815) executor: fix index join panic on prefix index on some cases (pingcap#24568)

tiancaiamao · 2021-11-22T13:31:00Z

/run-cherry-pick

tiancaiamao · 2021-11-22T13:41:06Z

/run-cherry-picker

tiancaiamao · 2021-11-22T15:08:23Z

/cherry-pick release-4.0

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-11-22T15:15:02Z

cherry pick to release-4.0 in PR #30011

…e. (#24542) (#30011)

XuHuaiyu · 2021-12-20T02:21:43Z

/run-cherry-picker

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-12-20T02:22:20Z

cherry pick to release-5.0 in PR #30857

…e. (#24542) (#30857) close #23114

fix

ad10d0b

wshwsh12 requested review from a team as code owners May 11, 2021 05:52

wshwsh12 requested review from lzmhhh123 and removed request for a team May 11, 2021 05:52

ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 11, 2021

wshwsh12 added 2 commits May 11, 2021 14:08

fix

5b2db24

remove ast.ifnull

7531af5

github-actions bot added the component/expression label May 11, 2021

ichn-hu mentioned this pull request May 11, 2021

Welcome to contribute #20804

Closed

lzmhhh123 reviewed May 12, 2021

View reviewed changes

expression/builtin_cast.go Outdated Show resolved Hide resolved

expression/builtin_cast.go Outdated Show resolved Hide resolved

address comments

a37d37b

wshwsh12 requested a review from lzmhhh123 May 13, 2021 03:07

add function elt

d29f678

lzmhhh123 approved these changes May 13, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 13, 2021

XuHuaiyu reviewed May 17, 2021

View reviewed changes

wshwsh12 added 2 commits May 18, 2021 11:37

address comments

a19bd12

Merge branch 'master' into issue-23114

84e3acd

wshwsh12 requested a review from XuHuaiyu May 18, 2021 03:59

XuHuaiyu reviewed May 18, 2021

View reviewed changes

wshwsh12 added 2 commits May 18, 2021 17:21

address comments

471a890

Merge remote-tracking branch 'upstream/master' into issue-23114

42adbcc

XuHuaiyu reviewed May 18, 2021

View reviewed changes

wshwsh12 added 2 commits May 18, 2021 18:27

address comments

b276a15

address comments

2a44f22

XuHuaiyu added the type/bugfix This PR fixes a bug. label May 18, 2021

ti-chi-bot removed the status/LGT1 Indicates that a PR has LGTM 1. label May 18, 2021

ti-chi-bot added the status/LGT2 Indicates that a PR has LGTM 2. label May 18, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 18, 2021

ti-chi-bot added 2 commits May 18, 2021 23:54

Merge branch 'master' into issue-23114

4958693

Merge branch 'master' into issue-23114

3133cc3

Merge branch 'master' into issue-23114

66f9a47

ti-chi-bot merged commit ac7e6a4 into pingcap:master May 18, 2021

tiancaiamao added the needs-cherry-pick-4.0 label Nov 22, 2021

ti-srebot mentioned this pull request Nov 22, 2021

expression, planner: push cast down to control function with enum type. (#24542) #30011

Merged

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Nov 22, 2021

cherry pick pingcap#24542 to release-4.0

b995731

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-chi-bot pushed a commit that referenced this pull request Nov 24, 2021

expression, planner: push cast down to control function with enum typ…

8ec0b14

…e. (#24542) (#30011)

XuHuaiyu added the needs-cherry-pick-release-5.0 label Dec 20, 2021

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Dec 20, 2021

cherry pick pingcap#24542 to release-5.0

a42a0bf

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Dec 20, 2021

expression, planner: push cast down to control function with enum type. (#24542) #30857

Merged

ti-chi-bot pushed a commit that referenced this pull request Dec 20, 2021

expression, planner: push cast down to control function with enum typ…

ecdf57b

…e. (#24542) (#30857) close #23114

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

expression, planner: push cast down to control function with enum type. #24542

expression, planner: push cast down to control function with enum type. #24542

wshwsh12 commented May 11, 2021 •

edited

Loading

XuHuaiyu May 17, 2021

XuHuaiyu May 17, 2021

wshwsh12 May 18, 2021

XuHuaiyu May 18, 2021

XuHuaiyu May 18, 2021

wshwsh12 May 18, 2021

XuHuaiyu May 18, 2021

wshwsh12 May 18, 2021

XuHuaiyu May 18, 2021

wshwsh12 May 18, 2021

XuHuaiyu May 18, 2021

wshwsh12 May 18, 2021

XuHuaiyu May 18, 2021

XuHuaiyu May 18, 2021

XuHuaiyu May 18, 2021

XuHuaiyu commented May 18, 2021

ti-chi-bot commented May 18, 2021

wshwsh12 commented May 18, 2021

ti-chi-bot commented May 18, 2021

ti-chi-bot commented May 18, 2021

ti-chi-bot commented May 18, 2021

ti-chi-bot commented May 18, 2021

tiancaiamao commented Nov 22, 2021

tiancaiamao commented Nov 22, 2021

tiancaiamao commented Nov 22, 2021

ti-srebot commented Nov 22, 2021

XuHuaiyu commented Dec 20, 2021

ti-srebot commented Dec 20, 2021

expression, planner: push cast down to control function with enum type. #24542

expression, planner: push cast down to control function with enum type. #24542

Conversation

wshwsh12 commented May 11, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XuHuaiyu commented May 18, 2021

ti-chi-bot commented May 18, 2021

wshwsh12 commented May 18, 2021

ti-chi-bot commented May 18, 2021

ti-chi-bot commented May 18, 2021

ti-chi-bot commented May 18, 2021

ti-chi-bot commented May 18, 2021

tiancaiamao commented Nov 22, 2021

tiancaiamao commented Nov 22, 2021

tiancaiamao commented Nov 22, 2021

ti-srebot commented Nov 22, 2021

XuHuaiyu commented Dec 20, 2021

ti-srebot commented Dec 20, 2021

wshwsh12 commented May 11, 2021 •

edited

Loading