Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: refine the behavior of StrToInt and StrToFloat and support convert JSON to date, time and timestamp #17902

Merged
merged 14 commits into from
Jun 22, 2020

Conversation

wjhuang2016
Copy link
Member

@wjhuang2016 wjhuang2016 commented Jun 10, 2020

What problem does this PR solve?

Issue Number: close #17898

Problem Summary:

TiDB don't distinguish cast(explicit CAST function or implicit cast) from cast(when casting into a column type, CastValue()).
We try to distinguish them according to if the statement is a select statement or an insert statement.
However, it's not correct, especially when the virtual generated column comes out. We can't make sure the value inserted and read are the same.
So we can't depend on if the type of the statement. We need to keep a principle. If some values are converted to a column type, then use CastValue(). Otherwise, don't use it.

Here is an example:
We didn't support converted a JSON to date. When we had a virtual generated column and it needed to be converted from a JSON to date, we used CastJsonAsDuration to do that. This is wrong. So I supported convert JSON to date, time, and timestamp within CastValue(), though their logics are quite similar.

Finally, I correct some tests. If you test these tests in MySQL and previously TiDB, you will see that the warnings of TiDB are wrong.

What is changed and how it works?

  1. refine the behavior of StrToInt and StrToFloat. Handling them differently according to the function call them.
  2. implement convert JSON to date, time, and timestamp.
  3. correct some tests.

Related changes

  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test

Side effects

Release note

  • refine the behavior of StrToInt and StrToFloat and support convert JSON to date, time and timestamp

Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com>
Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com>
@codecov
Copy link

codecov bot commented Jun 10, 2020

Codecov Report

Merging #17902 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #17902   +/-   ##
===========================================
  Coverage   79.7056%   79.7056%           
===========================================
  Files           526        526           
  Lines        144306     144306           
===========================================
  Hits         115020     115020           
  Misses        20086      20086           
  Partials       9200       9200           

@wjhuang2016 wjhuang2016 added type/bugfix This PR fixes a bug. status/PTAL labels Jun 16, 2020
@wjhuang2016 wjhuang2016 requested a review from qw4990 June 18, 2020 09:48
@wjhuang2016
Copy link
Member Author

/run-all-tests

@wjhuang2016
Copy link
Member Author

/run-all-tests -tidb-test=pr/1067

@@ -1110,7 +1110,7 @@ func (b *builtinCastStringAsIntSig) handleOverflow(origRes int64, origStr string
}

sc := b.ctx.GetSessionVars().StmtCtx
if sc.InSelectStmt && types.ErrOverflow.Equal(origErr) {
if types.ErrOverflow.Equal(origErr) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like it would only happen in select statements before. Does the change make the insert statement different from mysql?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the test, this change won't break the existing test. The reason I delete sc.InSelectStmt is that we shouldn't use InSelectStmt to decide how to handle the err. We should use TruncateAsWarning , IgnoreTruncate to do that.

arg1Type = types.ETInt
}
bf, err := newBaseBuiltinFuncWithTp(ctx, c.funcName, args, types.ETDuration, arg0Type, arg1Type, types.ETReal)
bf, err := newBaseBuiltinFuncWithTp(ctx, c.funcName, args, types.ETDuration, types.ETInt, types.ETInt, types.ETReal)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For all types, arg is evaluated rounding to int64? Please add some comments.

Copy link
Contributor

@Reminiscent Reminiscent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM. Please fix the tests.

Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com>
@wjhuang2016 wjhuang2016 requested review from a team as code owners June 19, 2020 09:19
@wjhuang2016 wjhuang2016 requested review from hanfei1991 and removed request for a team June 19, 2020 09:19
Copy link
Contributor

@Reminiscent Reminiscent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qw4990
Copy link
Contributor

qw4990 commented Jun 19, 2020

/run-all-tests

@qw4990
Copy link
Contributor

qw4990 commented Jun 19, 2020

/run-common-test
/run-integration-ddl-test

@qw4990
Copy link
Contributor

qw4990 commented Jun 19, 2020

/run-integration-common-test

@wjhuang2016
Copy link
Member Author

/run-all-tests -tidb-test=pr/1067

@wjhuang2016
Copy link
Member Author

/run-all-tests -tidb-test=pr/1067

2 similar comments
@wjhuang2016
Copy link
Member Author

/run-all-tests -tidb-test=pr/1067

@wjhuang2016
Copy link
Member Author

/run-all-tests -tidb-test=pr/1067

@ti-srebot
Copy link
Contributor

@Reminiscent, @qw4990, @hanfei1991, PTAL.

@qw4990
Copy link
Contributor

qw4990 commented Jun 22, 2020

/run-all-tests

@wjhuang2016
Copy link
Member Author

/run-all-tests -tidb-test=pr/1067

1 similar comment
@wjhuang2016
Copy link
Member Author

/run-all-tests -tidb-test=pr/1067

@wjhuang2016
Copy link
Member Author

/run-all-tests -tidb-test=pr/1067

@wjhuang2016
Copy link
Member Author

/run-all-tests -tidb-test=pr/1067

Copy link
Contributor

@qw4990 qw4990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qw4990
Copy link
Contributor

qw4990 commented Jun 22, 2020

/merge

@ti-srebot ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 22, 2020
@ti-srebot
Copy link
Contributor

/run-all-tests

@ti-srebot
Copy link
Contributor

@wjhuang2016 merge failed.

@wjhuang2016
Copy link
Member Author

/merge

@ti-srebot
Copy link
Contributor

Sorry @wjhuang2016, you don't have permission to trigger auto merge event on this branch. You are not a committer for the related sigs:execution(slack).

@wjhuang2016 wjhuang2016 merged commit bca31ea into pingcap:master Jun 22, 2020
ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 22, 2020
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot
Copy link
Contributor

cherry pick to release-4.0 in PR #18159

ti-srebot added a commit that referenced this pull request Jul 28, 2020
… JSON to date, time and timestamp (#17902) (#18159)

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
zyguan added a commit to zyguan/tikv that referenced this pull request Jul 13, 2022
Signed-off-by: zyguan <zhongyangguan@gmail.com>
@wjhuang2016 wjhuang2016 deleted the fix_str_to_int branch November 17, 2022 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/expression status/can-merge Indicates a PR has been approved by a committer. type/bugfix This PR fixes a bug.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Wrong handling of strToInt and strToFloat
4 participants