[SPARK-8192] [SPARK-8193] [SQL] udf current_date, current_timestamp #6985

adrian-wang · 2015-06-24T12:12:08Z

No description provided.

SparkQA · 2015-06-24T12:56:47Z

Test build #35674 has finished for PR 6985 at commit 9e4bca5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CurrentTimestamp() extends Expression
- case class CurrentDate() extends Expression

adrian-wang · 2015-06-30T04:21:38Z

Close it because #6782 (comment) for now

rxin · 2015-07-02T07:21:44Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala

+/**
+ * Adds a number of days to startdate: date_add('2008-12-31', 1) = '2009-01-01'.
+ */
+case class CurrentTimestamp() extends Expression {


can this one extend LeafExpression?

rxin · 2015-07-02T07:24:19Z

@adrian-wang can you do this one first? This one is simple enough that there is no debate. Update it to address my comment with test cases, and then we can merge it.

SparkQA · 2015-07-02T09:53:57Z

Test build #36384 has finished for PR 6985 at commit 427d9dc.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CurrentDate() extends LeafExpression
- case class CurrentTimestamp() extends LeafExpression

SparkQA · 2015-07-02T11:08:29Z

Test build #36385 has finished for PR 6985 at commit 98e8550.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CurrentDate() extends LeafExpression
- case class CurrentTimestamp() extends LeafExpression

adrian-wang · 2015-07-02T17:00:00Z

I think it again, and since System.currentTimeMillis is not so close to what's inside, I designed to put it back.

rxin · 2015-07-02T17:19:31Z

...talyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DatetimeFunctionsSuite.scala

+  test("datetime function current_date") {
+    checkEvaluation(
+      CurrentDate(),
+      new Date(System.currentTimeMillis), EmptyRow)


this test would be flaky -- I think you want to add some tolerance, e.g. checking System.currentTimeMillis +- 100ms

Yes, I thought that was rare to have. I'll modify the code here.

@adrian-wang just fyi this one is still flaky here ...

SparkQA · 2015-07-03T04:51:35Z

Test build #36472 has finished for PR 6985 at commit e11ae75.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CurrentDate() extends LeafExpression
- case class CurrentTimestamp() extends LeafExpression

rxin · 2015-07-03T06:21:05Z

sql/core/src/test/scala/org/apache/spark/sql/DatetimeExpressionsSuite.scala

+
+  import ctx.implicits._
+
+  val df1 = Seq((1, 2), (3, 1)).toDF("a", "b")


don't do this here. if there is an error with the df creation, it will make all the test cases disappear since it wasn't able to construct the object.

you can add a lazy in front of val here.

rxin · 2015-07-03T06:36:48Z

lgtm

SparkQA · 2015-07-03T07:16:08Z

Test build #36478 has finished for PR 6985 at commit 27c9f95.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CurrentDate() extends LeafExpression
- case class CurrentTimestamp() extends LeafExpression

SparkQA · 2015-07-03T08:11:04Z

Test build #36483 has finished for PR 6985 at commit 6a20b64.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CurrentDate() extends LeafExpression
- case class CurrentTimestamp() extends LeafExpression

rxin · 2015-07-04T05:19:55Z

Thanks - I've merged this.

…][SPARK-8179][SPARK-8177][SPARK-8178][SPARK-9115][SQL] date functions Jira: https://issues.apache.org/jira/browse/SPARK-8199 https://issues.apache.org/jira/browse/SPARK-8184 https://issues.apache.org/jira/browse/SPARK-8183 https://issues.apache.org/jira/browse/SPARK-8182 https://issues.apache.org/jira/browse/SPARK-8181 https://issues.apache.org/jira/browse/SPARK-8180 https://issues.apache.org/jira/browse/SPARK-8179 https://issues.apache.org/jira/browse/SPARK-8177 https://issues.apache.org/jira/browse/SPARK-8179 https://issues.apache.org/jira/browse/SPARK-9115 Regarding `day`and `dayofmonth` are both necessary? ~~I am going to add `Quarter` to this PR as well.~~ Done. ~~As soon as the Scala coding is reviewed and discussed, I'll add the python api.~~ Done Author: Tarek Auel <tarek.auel@googlemail.com> Author: Tarek Auel <tarek.auel@gmail.com> Closes #6981 from tarekauel/SPARK-8199 and squashes the following commits: f7b4c8c [Tarek Auel] [SPARK-8199] fixed bug in tests bb567b6 [Tarek Auel] [SPARK-8199] fixed test 3e095ba [Tarek Auel] [SPARK-8199] style and timezone fix 256c357 [Tarek Auel] [SPARK-8199] code cleanup 5983dcc [Tarek Auel] [SPARK-8199] whitespace fix 6e0c78f [Tarek Auel] [SPARK-8199] removed setTimeZone in tests, according to cloud-fans comment in #7488 4afc09c [Tarek Auel] [SPARK-8199] concise leap year handling ea6c110 [Tarek Auel] [SPARK-8199] fix after merging master 70238e0 [Tarek Auel] Merge branch 'master' into SPARK-8199 3c6ae2e [Tarek Auel] [SPARK-8199] removed binary search fb98ba0 [Tarek Auel] [SPARK-8199] python docstring fix cdfae27 [Tarek Auel] [SPARK-8199] cleanup & python docstring fix 746b80a [Tarek Auel] [SPARK-8199] build fix 0ad6db8 [Tarek Auel] [SPARK-8199] minor fix 523542d [Tarek Auel] [SPARK-8199] address comments 2259299 [Tarek Auel] [SPARK-8199] day_of_month alias d01b977 [Tarek Auel] [SPARK-8199] python underscore 56c4a92 [Tarek Auel] [SPARK-8199] update python docu e223bc0 [Tarek Auel] [SPARK-8199] refactoring d6aa14e [Tarek Auel] [SPARK-8199] fixed Hive compatibility b382267 [Tarek Auel] [SPARK-8199] fixed bug in day calculation; removed set TimeZone in HiveCompatibilitySuite for test purposes; removed Hive tests for second and minute, because we can cast '2015-03-18' to a timestamp and extract a minute/second from it 1b2e540 [Tarek Auel] [SPARK-8119] style fix 0852655 [Tarek Auel] [SPARK-8119] changed from ExpectsInputTypes to implicit casts ec87c69 [Tarek Auel] [SPARK-8119] bug fixing and refactoring 1358cdc [Tarek Auel] Merge remote-tracking branch 'origin/master' into SPARK-8199 740af0e [Tarek Auel] implement date function using a calculation based on days 4fb66da [Tarek Auel] WIP: date functions on calculation only 1a436c9 [Tarek Auel] wip f775f39 [Tarek Auel] fixed return type ad17e96 [Tarek Auel] improved implementation c42b444 [Tarek Auel] Removed merge conflict file ccb723c [Tarek Auel] [SPARK-8199] style and fixed merge issues 10e4ad1 [Tarek Auel] Merge branch 'master' into date-functions-fast 7d9f0eb [Tarek Auel] [SPARK-8199] git renaming issue f3e7a9f [Tarek Auel] [SPARK-8199] revert change in DataFrameFunctionsSuite 6f5d95c [Tarek Auel] [SPARK-8199] fixed year interval d9f8ac3 [Tarek Auel] [SPARK-8199] implement fast track 7bc9d93 [Tarek Auel] Merge branch 'master' into SPARK-8199 5a105d9 [Tarek Auel] [SPARK-8199] rebase after #6985 got merged eb6760d [Tarek Auel] Merge branch 'master' into SPARK-8199 f120415 [Tarek Auel] improved runtime a8edebd [Tarek Auel] use Calendar instead of SimpleDateFormat 5fe74e1 [Tarek Auel] fixed python style 3bfac90 [Tarek Auel] fixed style 356df78 [Tarek Auel] rely on cast mechanism of Spark. Simplified implementation 02efc5d [Tarek Auel] removed doubled code a5ea120 [Tarek Auel] added python api; changed test to be more meaningful b680db6 [Tarek Auel] added codegeneration to all functions c739788 [Tarek Auel] added support for quarter SPARK-8178 849fb41 [Tarek Auel] fixed stupid test 638596f [Tarek Auel] improved codegen 4d8049b [Tarek Auel] fixed tests and added type check 5ebb235 [Tarek Auel] resolved naming conflict d0e2f99 [Tarek Auel] date functions

yhuai · 2015-07-28T14:57:06Z

sql/core/src/test/scala/org/apache/spark/sql/DatetimeExpressionsSuite.scala

+  test("function current_timestamp") {
+    checkAnswer(df1.select(countDistinct(current_timestamp())), Row(1))
+    // Execution in one query should return the same value
+    checkAnswer(ctx.sql("""SELECT CURRENT_TIMESTAMP() = CURRENT_TIMESTAMP()"""),


Seems it is possible that CURRENT_TIMESTAMP() = CURRENT_TIMESTAMP() will return false?

I see it fails jenkins (https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3095/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.3,label=centos/testReport/junit/org.apache.spark.sql/DatetimeExpressionsSuite/function_current_timestamp/).

yes, this is a known issue as SPARK-9196 . we have to use a different way to evaluate this function. I think we'd better disable this function test for now.

adrian-wang closed this Jun 30, 2015

rxin reviewed Jul 2, 2015
View reviewed changes

udf current

0b69a1f

adrian-wang reopened this Jul 2, 2015

add tests and codegen

427d9dc

adrian-wang force-pushed the udfcurrent branch from 9e4bca5 to 427d9dc Compare July 2, 2015 09:46

fix sytle

98e8550

adrian-wang changed the title ~~[SPARK-8192] [SPARK-8193] [SQL] [WIP] udf current_date, current_timestamp~~ [SPARK-8192] [SPARK-8193] [SQL] udf current_date, current_timestamp Jul 2, 2015

add in functions

61ed3d5

rxin reviewed Jul 2, 2015
View reviewed changes

refine tests

e11ae75

refine tests..

27c9f95

rxin reviewed Jul 3, 2015
View reviewed changes

remove codegen and add lazy in testsuite

6a20b64

asfgit closed this in 9fb6b83 Jul 4, 2015

tarekbecker added a commit to tarekbecker/spark that referenced this pull request Jul 4, 2015

[SPARK-8199] rebase after apache#6985 got merged

5a105d9

adrian-wang deleted the udfcurrent branch July 21, 2015 00:37

yhuai reviewed Jul 28, 2015
View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8192] [SPARK-8193] [SQL] udf current_date, current_timestamp #6985

[SPARK-8192] [SPARK-8193] [SQL] udf current_date, current_timestamp #6985

adrian-wang commented Jun 24, 2015

SparkQA commented Jun 24, 2015

adrian-wang commented Jun 30, 2015

rxin Jul 2, 2015

rxin commented Jul 2, 2015

SparkQA commented Jul 2, 2015

SparkQA commented Jul 2, 2015

adrian-wang commented Jul 2, 2015

rxin Jul 2, 2015

adrian-wang Jul 2, 2015

rxin Jul 3, 2015

SparkQA commented Jul 3, 2015

rxin Jul 3, 2015

rxin commented Jul 3, 2015

SparkQA commented Jul 3, 2015

SparkQA commented Jul 3, 2015

rxin commented Jul 4, 2015

yhuai Jul 28, 2015

adrian-wang Jul 28, 2015


		import ctx.implicits._

		val df1 = Seq((1, 2), (3, 1)).toDF("a", "b")

[SPARK-8192] [SPARK-8193] [SQL] udf current_date, current_timestamp #6985

[SPARK-8192] [SPARK-8193] [SQL] udf current_date, current_timestamp #6985

Conversation

adrian-wang commented Jun 24, 2015

SparkQA commented Jun 24, 2015

adrian-wang commented Jun 30, 2015

rxin Jul 2, 2015

Choose a reason for hiding this comment

rxin commented Jul 2, 2015

SparkQA commented Jul 2, 2015

SparkQA commented Jul 2, 2015

adrian-wang commented Jul 2, 2015

rxin Jul 2, 2015

Choose a reason for hiding this comment

adrian-wang Jul 2, 2015

Choose a reason for hiding this comment

rxin Jul 3, 2015

Choose a reason for hiding this comment

SparkQA commented Jul 3, 2015

rxin Jul 3, 2015

Choose a reason for hiding this comment

rxin commented Jul 3, 2015

SparkQA commented Jul 3, 2015

SparkQA commented Jul 3, 2015

rxin commented Jul 4, 2015

yhuai Jul 28, 2015

Choose a reason for hiding this comment

adrian-wang Jul 28, 2015

Choose a reason for hiding this comment