[SPARK-32133][SQL] Forbid time field steps for date start/end in Sequence #28926

TJX2014 · 2020-06-25T05:37:12Z

What changes were proposed in this pull request?

1.Add time field steps check for date start/end in Sequence at org.apache.spark.sql.catalyst.expressions.Sequence.TemporalSequenceImpl
2.Add a UT：SPARK-32133: Sequence step must be a day interval if start and end values are dates at org.apache.spark.sql.catalyst.expressions.CollectionExpressionsSuite

Why are the changes needed?

Sequence time field steps for date start/end looks strange in spark as follows:

scala> sql("select explode(sequence(cast('2011-03-01' as date), cast('2011-03-02' as date), interval 1 hour))").head(3)
res0: Array[org.apache.spark.sql.Row] = _Array([2011-03-01], [2011-03-01], [2011-03-01])_ **<- strange result.**

scala> sql("select explode(sequence(cast('2011-03-01' as date), cast('2011-03-02' as date), interval 1 day))").head(3)
res1: Array[org.apache.spark.sql.Row] = Array([2011-03-01], [2011-03-02])

While this behavior in Prosto make sense：

presto> select sequence(date('2011-03-01'),date('2011-03-02'),interval '1' hour);
Query 20200624_122744_00002_pehix failed: sequence step must be a day interval if start and end values are dates
presto> select sequence(date('2011-03-01'),date('2011-03-02'),interval '1' day);
_col0
[2011-03-01, 2011-03-02]

Does this PR introduce any user-facing change?

Yes, after this patch, users will get informed sequence step must be a day interval if start and end values are dates when
use time field steps for date start/end in Sequence.

How was this patch tested?

Unit test.

TJX2014 · 2020-06-25T05:38:54Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

@@ -2612,6 +2614,9 @@ object Sequence {
      val stepDays = step.days
      val stepMicros = step.microseconds

+      require(scale != MICROS_PER_DAY || stepMonths != 0 || stepDays != 0,


Add scale constraint here to DateType

TJX2014 · 2020-06-25T05:41:44Z

Follow: #28856
cc @cloud-fan @MaxGekk

cloud-fan · 2020-06-29T11:50:09Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

@@ -2612,6 +2614,9 @@ object Sequence {
      val stepDays = step.days
      val stepMicros = step.microseconds

+      require(scale != MICROS_PER_DAY || stepMonths != 0 || stepDays != 0,
+        "sequence step must be a day interval if start and end values are dates")
+
      if (stepMonths == 0 && stepMicros == 0 && scale == MICROS_PER_DAY) {


we can remove this branch now?

Seems we need the require check in eval, and I remove SPARK-32198 branch code from this PR, is it ok ?

@TJX2014 . What is SPARK-32198? It's not created yet.

I remove SPARK-32198 branch code from this PR

Also, I have the same question like @cloud-fan .

@dongjoon-hyun Sorry, the correct PR is SPARK-31982, this PR is forbid step and the other is cross the timezone.

cloud-fan · 2020-06-29T11:51:10Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

-        val startMicros: Long = num.toLong(start) * scale
-        val stopMicros: Long = num.toLong(stop) * scale
+
+        // Date to timestamp is not equal from GMT and Chicago timezones


can we exclude these changes and make this PR smaller to only focus on "forbid time field steps for date start/end"?

@cloud-fan Thanks, I have followed the suggestion and make a new jira ticket for this.

cloud-fan · 2020-06-30T07:21:35Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

@@ -2679,6 +2682,11 @@ object Sequence {
         |final int $stepDays = $step.days;
         |final long $stepMicros = $step.microseconds;
         |
+         |if (${scale}L == ${MICROS_PER_DAY}L && $stepMonths == 0 && $stepDays == 0) {


the scale is known before generating the code. We can do better

val check = if (scale == MICROS_PER_DAY) { s""" if ($stepMonths == 0 && $stepDays == 0) ... """ } else { "" }

cloud-fan · 2020-06-30T07:22:39Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

@@ -2612,6 +2612,9 @@ object Sequence {
      val stepDays = step.days
      val stepMicros = step.microseconds

+      require(scale != MICROS_PER_DAY || stepMonths != 0 || stepDays != 0,
+        "sequence step must be a day interval if start and end values are dates")


let's use

if (...) throw new IllegalArgumentException

to be more consistent with the codegen version.

cloud-fan · 2020-06-30T07:24:35Z

...st/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala

+  test("SPARK-32133: Sequence step must be a day interval " +
+    "if start and end values are dates") {
+    val e = intercept[Exception](
+      checkEvaluation(Sequence(


we should use checkExceptionInExpression for error tests.

TJX2014 · 2020-06-30T09:15:58Z

...st/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala

+    checkEvaluation(Sequence(
+      Cast(Literal("2011-03-01"), DateType),
+      Cast(Literal("2011-03-02"), DateType),
+      Option(Literal(stringToInterval("interval 1 day")))),
+      Seq(Date.valueOf("2011-03-01"), Date.valueOf("2011-03-02")))


@cloud-fan Thank you for your suggestion, I have done. Besides, I add a positive test case, hope this could be better.

cloud-fan · 2020-06-30T15:22:38Z

...st/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala

@@ -1854,4 +1854,18 @@ class CollectionExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper
      Literal(stringToInterval("interval 1 year"))),
      Seq(Date.valueOf("2018-01-01")))
  }
+
+  test("SPARK-32133: Sequence step must be a day interval " +


instead of adding a new test, shall we just put the new negative test in the existing test case Sequence of dates? We can point to the JIRA number in the code comment.

Thanks, I have moved the negative test to Sequence of dates and point to the JIRA number in the code comment.

cloud-fan · 2020-07-01T07:15:44Z

ok to test

cloud-fan · 2020-07-01T07:15:49Z

retest this please

SparkQA · 2020-07-01T07:48:50Z

Test build #124752 has finished for PR 28926 at commit 24384c9.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-01T08:20:04Z

retest this please

SparkQA · 2020-07-01T08:42:12Z

Test build #124759 has finished for PR 28926 at commit 24384c9.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-01T14:06:26Z

Test build #5049 has finished for PR 28926 at commit 24384c9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-01T14:58:16Z

retest this please

SparkQA · 2020-07-01T20:02:03Z

Test build #124791 has finished for PR 28926 at commit 24384c9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-07-02T01:14:57Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

@@ -2612,6 +2612,11 @@ object Sequence {
      val stepDays = step.days
      val stepMicros = step.microseconds

+      if(scale == MICROS_PER_DAY && stepMonths == 0 && stepDays == 0) {


nit. if( -> if (?

https://github.com/databricks/scala-style-guide/blame/master/README.md#L323

dongjoon-hyun · 2020-07-02T01:19:06Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+      val check = if (scale == MICROS_PER_DAY) {
+        s"""
+           if ($stepMonths == 0 && $stepDays == 0) {
+            throw new IllegalArgumentException(


indentation?

cloud-fan · 2020-07-02T12:19:21Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+      val check = if (scale == MICROS_PER_DAY) {
+        s"""
+           if ($stepMonths == 0 && $stepDays == 0) {
+              throw new IllegalArgumentException(


nit: for generated code, we shall use the multiline string syntax:

s""" |if ... |""".stripMargin

SparkQA · 2020-07-02T20:12:09Z

Test build #124896 has finished for PR 28926 at commit fe6a32d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-06T06:33:57Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+        throw new IllegalArgumentException(
+          "sequence step must be a day interval if start and end values are dates")
+      }
+
      if (stepMonths == 0 && stepMicros == 0 && scale == MICROS_PER_DAY) {


We can probably add comments for each branch. For example, this branch is for adding days to date start/end.

… github.com:TJX2014/spark into master-SPARK-31982-sequence-cross-dst-follow-presto

TJX2014 · 2020-07-06T10:05:08Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+        // Adding pure days to date start/end
        backedSequenceImpl.eval(start, stop, fromLong(stepDays))

      } else if (stepMonths == 0 && stepDays == 0 && scale == 1) {
+        // Adding pure microseconds to timestamp start/end


@cloud-fan Thanks, I add more comments for pure days and months branch, the former exception check seems the exception content has enough informs, and the last branch seems have detail content.

SparkQA · 2020-07-06T14:41:08Z

Test build #125059 has finished for PR 28926 at commit 8c4ffaa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-06T15:15:08Z

retest this please

SparkQA · 2020-07-06T15:45:18Z

Test build #125097 has finished for PR 28926 at commit 8c4ffaa.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-07-07T14:24:57Z

Retest this please.

SparkQA · 2020-07-07T15:01:18Z

Test build #125231 has finished for PR 28926 at commit 8c4ffaa.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-07-07T17:51:26Z

Retest this please.

SparkQA · 2020-07-08T00:32:45Z

Test build #125242 has finished for PR 28926 at commit 8c4ffaa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-08T08:05:00Z

retest this please

SparkQA · 2020-07-08T08:28:06Z

Test build #125333 has finished for PR 28926 at commit 8c4ffaa.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-08T12:21:21Z

retest this please

SparkQA · 2020-07-08T18:12:32Z

Test build #125358 has finished for PR 28926 at commit 8c4ffaa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

### What changes were proposed in this pull request? This PR aims to disable SBT `unidoc` generation testing in Jenkins environment because it's flaky in Jenkins environment and not used for the official documentation generation. Also, GitHub Action has the correct test coverage for the official documentation generation. - #28848 (comment) (amp-jenkins-worker-06) - #28926 (comment) (amp-jenkins-worker-06) - #28969 (comment) (amp-jenkins-worker-06) - #28975 (comment) (amp-jenkins-worker-05) - #28986 (comment) (amp-jenkins-worker-05) - #28992 (comment) (amp-jenkins-worker-06) - #28993 (comment) (amp-jenkins-worker-05) - #28999 (comment) (amp-jenkins-worker-04) - #29010 (comment) (amp-jenkins-worker-03) - #29013 (comment) (amp-jenkins-worker-04) - #29016 (comment) (amp-jenkins-worker-05) - #29025 (comment) (amp-jenkins-worker-04) - #29042 (comment) (amp-jenkins-worker-03) ### Why are the changes needed? Apache Spark `release-build.sh` generates the official document by using the following command. - https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh#L341 ```bash PRODUCTION=1 RELEASE_VERSION="$SPARK_VERSION" jekyll build ``` And, this is executed by the following `unidoc` command for Scala/Java API doc. - https://github.com/apache/spark/blob/master/docs/_plugins/copy_api_dirs.rb#L30 ```ruby system("build/sbt -Pkinesis-asl clean compile unidoc") || raise("Unidoc generation failed") ``` However, the PR builder disabled `Jekyll build` and instead has a different test coverage. ```python # determine if docs were changed and if we're inside the amplab environment # note - the below commented out until *all* Jenkins workers can get `jekyll` installed # if "DOCS" in changed_modules and test_env == "amplab_jenkins": # build_spark_documentation() ``` ``` Building Unidoc API Documentation ======================================================================== [info] Building Spark unidoc using SBT with these arguments: -Phadoop-3.2 -Phive-2.3 -Pspark-ganglia-lgpl -Pkubernetes -Pmesos -Phadoop-cloud -Phive -Phive-thriftserver -Pkinesis-asl -Pyarn unidoc ``` ### Does this PR introduce _any_ user-facing change? No. (This is used only for testing and not used in the official doc generation.) ### How was this patch tested? Pass the Jenkins without doc generation invocation. Closes #29017 from dongjoon-hyun/SPARK-DOC-GEN. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2020-07-08T23:53:44Z

Retest this please.

SparkQA · 2020-07-09T03:41:57Z

Test build #125419 has finished for PR 28926 at commit 8c4ffaa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-09T08:00:11Z

retest this please

SparkQA · 2020-07-09T14:22:35Z

Test build #125454 has finished for PR 28926 at commit 8c4ffaa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-09T15:04:15Z

retest this please

SparkQA · 2020-07-09T15:07:49Z

Test build #125487 has started for PR 28926 at commit 8c4ffaa.

dongjoon-hyun · 2020-07-09T18:00:44Z

Retest this please

SparkQA · 2020-07-09T19:20:01Z

Test build #125506 has finished for PR 28926 at commit 8c4ffaa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-10T07:24:47Z

retest this please

SparkQA · 2020-07-10T15:49:22Z

Test build #125577 has finished for PR 28926 at commit 8c4ffaa.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM. I tested PySpark/SparkR locally. Merged to master. Thank you, @TJX2014 and all!

Tests passed in 869 seconds

OK:       2306
Failed:   0
Warnings: 0
Skipped:  1

TJX2014 · 2020-07-11T00:27:24Z

Thank you very much.

probot-autolabeler bot added the SQL label Jun 25, 2020

TJX2014 commented Jun 25, 2020

View reviewed changes

cloud-fan reviewed Jun 29, 2020

View reviewed changes

TJX2014 force-pushed the master-SPARK-31982-sequence-cross-dst-follow-presto branch from e6ccfec to 7021539 Compare June 29, 2020 23:02

TJX2014 changed the title ~~[SPARK-31982][SQL][FOLLOWUP]Function sequence doesn't handle date increments that cross DST~~ [SPARK-32133][SQL] Forbid time field steps for date start/end in Sequence Jun 29, 2020

TJX2014 requested a review from cloud-fan June 30, 2020 00:21

cloud-fan reviewed Jun 30, 2020

View reviewed changes

TJX2014 commented Jun 30, 2020

View reviewed changes

cloud-fan reviewed Jun 30, 2020

View reviewed changes

cloud-fan approved these changes Jun 30, 2020

View reviewed changes

dongjoon-hyun reviewed Jul 2, 2020

View reviewed changes

TJX2014 requested a review from dongjoon-hyun July 2, 2020 12:00

cloud-fan reviewed Jul 2, 2020

View reviewed changes

TJX2014 added 3 commits July 3, 2020 06:50

forbid time field steps for date start/end

665ac46

SPARK-32198 -> SPARK-32133

7733184

improve code

74c683f

cloud-fan reviewed Jul 6, 2020

View reviewed changes

TJX2014 added 2 commits July 6, 2020 17:58

new code comment to date and timestamp

d860c0e

Merge branch 'master-SPARK-31982-sequence-cross-dst-follow-presto' of…

8c4ffaa

… github.com:TJX2014/spark into master-SPARK-31982-sequence-cross-dst-follow-presto

TJX2014 commented Jul 6, 2020

View reviewed changes

dongjoon-hyun mentioned this pull request Jul 7, 2020

[SPARK-32233][TESTS] Disable SBT unidoc generation testing in Jenkins #29017

Closed

dongjoon-hyun approved these changes Jul 10, 2020

View reviewed changes

dongjoon-hyun closed this in 500877e Jul 10, 2020

[SPARK-32133][SQL] Forbid time field steps for date start/end in Sequence #28926

[SPARK-32133][SQL] Forbid time field steps for date start/end in Sequence #28926

Conversation

TJX2014 commented Jun 25, 2020 • edited by dongjoon-hyun Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Choose a reason for hiding this comment

TJX2014 commented Jun 25, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TJX2014 Jul 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan commented Jul 1, 2020

cloud-fan commented Jul 1, 2020

SparkQA commented Jul 1, 2020

cloud-fan commented Jul 1, 2020

SparkQA commented Jul 1, 2020

SparkQA commented Jul 1, 2020

cloud-fan commented Jul 1, 2020

SparkQA commented Jul 1, 2020

dongjoon-hyun Jul 2, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan Jul 2, 2020 • edited Loading

Choose a reason for hiding this comment

SparkQA commented Jul 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TJX2014 Jul 6, 2020 • edited Loading

Choose a reason for hiding this comment

SparkQA commented Jul 6, 2020

cloud-fan commented Jul 6, 2020

SparkQA commented Jul 6, 2020

dongjoon-hyun commented Jul 7, 2020

SparkQA commented Jul 7, 2020

dongjoon-hyun commented Jul 7, 2020

SparkQA commented Jul 8, 2020

cloud-fan commented Jul 8, 2020

SparkQA commented Jul 8, 2020

cloud-fan commented Jul 8, 2020

SparkQA commented Jul 8, 2020

dongjoon-hyun commented Jul 8, 2020

SparkQA commented Jul 9, 2020

cloud-fan commented Jul 9, 2020

SparkQA commented Jul 9, 2020

cloud-fan commented Jul 9, 2020

SparkQA commented Jul 9, 2020

dongjoon-hyun commented Jul 9, 2020

SparkQA commented Jul 9, 2020

cloud-fan commented Jul 10, 2020

SparkQA commented Jul 10, 2020

dongjoon-hyun left a comment

Choose a reason for hiding this comment

TJX2014 commented Jul 11, 2020

TJX2014 commented Jun 25, 2020 •

edited by dongjoon-hyun

Loading

TJX2014 commented Jun 25, 2020 •

edited

Loading

TJX2014 Jul 6, 2020 •

edited

Loading

dongjoon-hyun Jul 2, 2020 •

edited

Loading

cloud-fan Jul 2, 2020 •

edited

Loading

TJX2014 Jul 6, 2020 •

edited

Loading