[SPARK-38990][SQL] Avoid NullPointerException when evaluating date_trunc/trunc format as a bound reference
#36312
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HyukjinKwon
approved these changes
Apr 22, 2022
HyukjinKwon
pushed a commit
that referenced
this pull request
Apr 22, 2022
…trunc/trunc format as a bound reference
### What changes were proposed in this pull request?
Change `TruncInstant.evalHelper` to pass the input row to `format.eval` when `format` is a not a literal (and therefore might be a bound reference).
### Why are the changes needed?
This query fails with a `java.lang.NullPointerException`:
```
select date_trunc(col1, col2)
from values
('week', timestamp'2012-01-01')
as data(col1, col2);
```
This only happens if the data comes from an inline table. When the source is an inline table, `ConvertToLocalRelation` attempts to evaluate the function against the data in interpreted mode.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Update to unit tests.
Closes #36312 from bersprockets/date_trunc_issue.
Authored-by: Bruce Robbins <bersprockets@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 2e4f4ab)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon
pushed a commit
that referenced
this pull request
Apr 22, 2022
…trunc/trunc format as a bound reference
### What changes were proposed in this pull request?
Change `TruncInstant.evalHelper` to pass the input row to `format.eval` when `format` is a not a literal (and therefore might be a bound reference).
### Why are the changes needed?
This query fails with a `java.lang.NullPointerException`:
```
select date_trunc(col1, col2)
from values
('week', timestamp'2012-01-01')
as data(col1, col2);
```
This only happens if the data comes from an inline table. When the source is an inline table, `ConvertToLocalRelation` attempts to evaluate the function against the data in interpreted mode.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Update to unit tests.
Closes #36312 from bersprockets/date_trunc_issue.
Authored-by: Bruce Robbins <bersprockets@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 2e4f4ab)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon
pushed a commit
that referenced
this pull request
Apr 22, 2022
…trunc/trunc format as a bound reference
### What changes were proposed in this pull request?
Change `TruncInstant.evalHelper` to pass the input row to `format.eval` when `format` is a not a literal (and therefore might be a bound reference).
### Why are the changes needed?
This query fails with a `java.lang.NullPointerException`:
```
select date_trunc(col1, col2)
from values
('week', timestamp'2012-01-01')
as data(col1, col2);
```
This only happens if the data comes from an inline table. When the source is an inline table, `ConvertToLocalRelation` attempts to evaluate the function against the data in interpreted mode.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Update to unit tests.
Closes #36312 from bersprockets/date_trunc_issue.
Authored-by: Bruce Robbins <bersprockets@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 2e4f4ab)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon
pushed a commit
that referenced
this pull request
Apr 22, 2022
…trunc/trunc format as a bound reference
### What changes were proposed in this pull request?
Change `TruncInstant.evalHelper` to pass the input row to `format.eval` when `format` is a not a literal (and therefore might be a bound reference).
### Why are the changes needed?
This query fails with a `java.lang.NullPointerException`:
```
select date_trunc(col1, col2)
from values
('week', timestamp'2012-01-01')
as data(col1, col2);
```
This only happens if the data comes from an inline table. When the source is an inline table, `ConvertToLocalRelation` attempts to evaluate the function against the data in interpreted mode.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Update to unit tests.
Closes #36312 from bersprockets/date_trunc_issue.
Authored-by: Bruce Robbins <bersprockets@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 2e4f4ab)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
|
Merged to master, branch-3.3, branch-3.2, branch-3.1 and branch-3.0. |
lvshaokang
pushed a commit
to lvshaokang/spark
that referenced
this pull request
Apr 22, 2022
…trunc/trunc format as a bound reference
### What changes were proposed in this pull request?
Change `TruncInstant.evalHelper` to pass the input row to `format.eval` when `format` is a not a literal (and therefore might be a bound reference).
### Why are the changes needed?
This query fails with a `java.lang.NullPointerException`:
```
select date_trunc(col1, col2)
from values
('week', timestamp'2012-01-01')
as data(col1, col2);
```
This only happens if the data comes from an inline table. When the source is an inline table, `ConvertToLocalRelation` attempts to evaluate the function against the data in interpreted mode.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Update to unit tests.
Closes apache#36312 from bersprockets/date_trunc_issue.
Authored-by: Bruce Robbins <bersprockets@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
|
Thanks @HyukjinKwon ! |
kazuyukitanimura
pushed a commit
to kazuyukitanimura/spark
that referenced
this pull request
Aug 10, 2022
…trunc/trunc format as a bound reference
### What changes were proposed in this pull request?
Change `TruncInstant.evalHelper` to pass the input row to `format.eval` when `format` is a not a literal (and therefore might be a bound reference).
### Why are the changes needed?
This query fails with a `java.lang.NullPointerException`:
```
select date_trunc(col1, col2)
from values
('week', timestamp'2012-01-01')
as data(col1, col2);
```
This only happens if the data comes from an inline table. When the source is an inline table, `ConvertToLocalRelation` attempts to evaluate the function against the data in interpreted mode.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Update to unit tests.
Closes apache#36312 from bersprockets/date_trunc_issue.
Authored-by: Bruce Robbins <bersprockets@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 2e4f4ab)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Change
TruncInstant.evalHelperto pass the input row toformat.evalwhenformatis a not a literal (and therefore might be a bound reference).Why are the changes needed?
This query fails with a
java.lang.NullPointerException:This only happens if the data comes from an inline table. When the source is an inline table,
ConvertToLocalRelationattempts to evaluate the function against the data in interpreted mode.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Update to unit tests.