[SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates #28819

TJX2014 · 2020-06-13T03:53:15Z

What changes were proposed in this pull request?

Add judge equal as bigger condition in org.apache.spark.sql.catalyst.expressions.Sequence.TemporalSequenceImpl#eval
Unit test for interval day, month, year

Why are the changes needed?

Bug exists when sequence input get same equal start and end dates, which will occur while loop forever

Does this PR introduce any user-facing change?

Yes,
Before this PR, people will get a java.lang.ArrayIndexOutOfBoundsException, when eval as below:
sql("select sequence(cast('2011-03-01' as date), cast('2011-03-01' as date), interval 1 year)").show(false)

How was this patch tested?

Unit test.

…ve forever

DaveDeCaprio · 2020-06-13T12:54:29Z

I'm not an admin or committer, but this patch looks good to me.

HyukjinKwon · 2020-06-14T05:56:09Z

I think it matches the behaviour with Presto's (see #21155). Shall we check how it works in Presto?

TJX2014 · 2020-06-14T09:15:07Z

I think it matches the behaviour with Presto's (see #21155). Shall we check how it works in Presto?

Sure, we can get result from Presto(presto-server-0.236) as follows：
`
presto> select sequence(date('2011-03-01'),date('2011-03-01'),interval '1' year);
_col0
[2011-03-01]
(1 row)

Query 20200614_091013_00013_wnnjv, FINISHED, 1 node
Splits: 17 total, 17 done (100.00%)
0:00 [0 rows, 0B] [0 rows/s, 0B/s]

presto> select sequence(date('2011-03-01'),date('2012-03-01'),interval '1' year);
_col0
[2011-03-01, 2012-03-01]
(1 row)

Query 20200614_091017_00014_wnnjv, FINISHED, 1 node
Splits: 17 total, 17 done (100.00%)
0:00 [0 rows, 0B] [0 rows/s, 0B/s]

presto> select sequence(date('2011-03-01'),date('2010-03-01'),interval '1' year);
Query 20200614_091021_00015_wnnjv failed: sequence stop value should be greater than or equal to start value if step is greater than zero otherwise stop should be less than or equal to start
`

HyukjinKwon · 2020-06-14T12:07:42Z

ok to test

HyukjinKwon · 2020-06-14T12:07:55Z

cc @ueshin FYI

SparkQA · 2020-06-14T16:39:15Z

Test build #124007 has finished for PR 28819 at commit c1765e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin

LGTM.

dongjoon-hyun

+1, LGTM. Thank you all. Merged to master/3.0.

…e are equal dates ### What changes were proposed in this pull request? 1. Add judge equal as bigger condition in `org.apache.spark.sql.catalyst.expressions.Sequence.TemporalSequenceImpl#eval` 2. Unit test for interval `day`, `month`, `year` ### Why are the changes needed? Bug exists when sequence input get same equal start and end dates, which will occur `while loop` forever ### Does this PR introduce _any_ user-facing change? Yes, Before this PR, people will get a `java.lang.ArrayIndexOutOfBoundsException`, when eval as below: `sql("select sequence(cast('2011-03-01' as date), cast('2011-03-01' as date), interval 1 year)").show(false) ` ### How was this patch tested? Unit test. Closes #28819 from TJX2014/master-SPARK-31980. Authored-by: TJX2014 <xiaoxingstack@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 177a380) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2020-06-20T02:31:22Z

Hi, @TJX2014 .
In branch-2.4, stringToInterval doesn't exist. Could you make a PR for Apache Spark 2.4.7 please?

TJX2014 · 2020-06-20T03:54:30Z

Thanks, I will make a PR for branch-2.4.

TJX2014 · 2020-06-20T09:29:37Z

Thanks, I will make a PR for branch-2.4.

@dongjoon-hyun This is the backport.#28877

… range are equal dates ### What changes were proposed in this pull request? 1. Add judge equal as bigger condition in `org.apache.spark.sql.catalyst.expressions.Sequence.TemporalSequenceImpl#eval` 2. Unit test for interval `day`, `month`, `year` Former PR to master is [#28819 Current PR change `stringToInterval => CalendarInterval.fromString` in order to compatible with branch-2.4 ### Why are the changes needed? Bug exists when sequence input get same equal start and end dates, which will occur `while loop` forever ### Does this PR introduce _any_ user-facing change? Yes, Before this PR, people will get a `java.lang.ArrayIndexOutOfBoundsException`, when eval as below: `sql("select sequence(cast('2011-03-01' as date), cast('2011-03-01' as date), interval 1 year)").show(false) ` ### How was this patch tested? Unit test. Closes #28877 from TJX2014/branch-2.4-SPARK-31980-compatible. Authored-by: TJX2014 <xiaoxingstack@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

… ArrayIndexOutOfBoundsException if the arguments are under the condition of start == stop && step < 0 ### What changes were proposed in this pull request? This PR fixes an issue that `sequence` builtin function causes `ArrayIndexOutOfBoundsException` if the arguments are under the condition of `start == stop && step < 0`. This is an example. ``` SELECT sequence(timestamp'2021-08-31', timestamp'2021-08-31', -INTERVAL 1 month); 21/09/02 04:14:42 ERROR SparkSQLDriver: Failed in [SELECT sequence(timestamp'2021-08-31', timestamp'2021-08-31', -INTERVAL 1 month)] java.lang.ArrayIndexOutOfBoundsException: 1 ``` Actually, this example succeeded before SPARK-31980 (#28819) was merged. ### Why are the changes needed? Bug fix. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New tests. Closes #33895 from sarutak/fix-sequence-issue. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>

… ArrayIndexOutOfBoundsException if the arguments are under the condition of start == stop && step < 0 ### What changes were proposed in this pull request? This PR fixes an issue that `sequence` builtin function causes `ArrayIndexOutOfBoundsException` if the arguments are under the condition of `start == stop && step < 0`. This is an example. ``` SELECT sequence(timestamp'2021-08-31', timestamp'2021-08-31', -INTERVAL 1 month); 21/09/02 04:14:42 ERROR SparkSQLDriver: Failed in [SELECT sequence(timestamp'2021-08-31', timestamp'2021-08-31', -INTERVAL 1 month)] java.lang.ArrayIndexOutOfBoundsException: 1 ``` Actually, this example succeeded before SPARK-31980 (#28819) was merged. ### Why are the changes needed? Bug fix. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New tests. Closes #33895 from sarutak/fix-sequence-issue. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com> (cherry picked from commit cf3bc65) Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>

…auses ArrayIndexOutOfBoundsException if the arguments are under the condition of start == stop && step < 0 ### What changes were proposed in this pull request? This PR backports the change of SPARK-36639 (#33895) for `branch-3.0`. This PR fixes an issue that `sequence` builtin function causes `ArrayIndexOutOfBoundsException` if the arguments are under the condition of `start == stop && step < 0`. This is an example. ``` SELECT sequence(timestamp'2021-08-31', timestamp'2021-08-31', -INTERVAL 1 month); 21/09/02 04:14:42 ERROR SparkSQLDriver: Failed in [SELECT sequence(timestamp'2021-08-31', timestamp'2021-08-31', -INTERVAL 1 month)] java.lang.ArrayIndexOutOfBoundsException: 1 ``` Actually, this example succeeded before SPARK-31980 (#28819) was merged. ### Why are the changes needed? Bug fix. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New tests. ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Closes #33909 from sarutak/backport-SPARK-36639-branch-3.0. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

… ArrayIndexOutOfBoundsException if the arguments are under the condition of start == stop && step < 0 ### What changes were proposed in this pull request? This PR fixes an issue that `sequence` builtin function causes `ArrayIndexOutOfBoundsException` if the arguments are under the condition of `start == stop && step < 0`. This is an example. ``` SELECT sequence(timestamp'2021-08-31', timestamp'2021-08-31', -INTERVAL 1 month); 21/09/02 04:14:42 ERROR SparkSQLDriver: Failed in [SELECT sequence(timestamp'2021-08-31', timestamp'2021-08-31', -INTERVAL 1 month)] java.lang.ArrayIndexOutOfBoundsException: 1 ``` Actually, this example succeeded before SPARK-31980 (apache#28819) was merged. ### Why are the changes needed? Bug fix. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New tests. Closes apache#33895 from sarutak/fix-sequence-issue. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com> (cherry picked from commit cf3bc65) Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>

TJX2014 added 3 commits June 13, 2020 10:04

bug reproduce

4e9cdf7

support start and end time equal condition judge to fix while recursi…

f970add

…ve forever

day and year interval test cases are also support

c1765e1

probot-autolabeler bot added the SQL label Jun 13, 2020

ueshin approved these changes Jun 18, 2020

View reviewed changes

dongjoon-hyun approved these changes Jun 20, 2020

View reviewed changes

dongjoon-hyun closed this in 177a380 Jun 20, 2020

TJX2014 mentioned this pull request Jun 20, 2020

[SPARK-31980][SQL][2.4] Function sequence() fails if start and end of range are equal dates #28877

Closed

sarutak mentioned this pull request Sep 1, 2021

[SPARK-36639][SQL] Fix an issue that sequence builtin function causes ArrayIndexOutOfBoundsException if the arguments are under the condition of start == stop && step < 0 #33895

Closed

sarutak mentioned this pull request Sep 3, 2021

[SPARK-36639][SQL][3.0] Fix an issue that sequence builtin function causes ArrayIndexOutOfBoundsException if the arguments are under the condition of start == stop && step < 0 #33909

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates #28819

[SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates #28819

TJX2014 commented Jun 13, 2020 •

edited

Loading

DaveDeCaprio commented Jun 13, 2020

HyukjinKwon commented Jun 14, 2020

TJX2014 commented Jun 14, 2020 •

edited

Loading

HyukjinKwon commented Jun 14, 2020

HyukjinKwon commented Jun 14, 2020

SparkQA commented Jun 14, 2020

ueshin left a comment

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun commented Jun 20, 2020

TJX2014 commented Jun 20, 2020

TJX2014 commented Jun 20, 2020

[SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates #28819

[SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates #28819

Conversation

TJX2014 commented Jun 13, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

DaveDeCaprio commented Jun 13, 2020

HyukjinKwon commented Jun 14, 2020

TJX2014 commented Jun 14, 2020 • edited Loading

HyukjinKwon commented Jun 14, 2020

HyukjinKwon commented Jun 14, 2020

SparkQA commented Jun 14, 2020

ueshin left a comment

Choose a reason for hiding this comment

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun commented Jun 20, 2020

TJX2014 commented Jun 20, 2020

TJX2014 commented Jun 20, 2020

TJX2014 commented Jun 13, 2020 •

edited

Loading

TJX2014 commented Jun 14, 2020 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading