[SPARK-8195] [SPARK-8196] [SQL] udf next_day last_day #6986

adrian-wang · 2015-06-24T12:48:01Z

next_day, returns next certain dayofweek.
last_day, returns the last day of the month which given date belongs to.

SparkQA · 2015-06-24T12:58:57Z

Test build #35677 has finished for PR 6986 at commit 5ef6a8c.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class LastDay(startDate: Expression) extends Expression
- case class NextDay(startDate: Expression, dayOfWeek: Expression) extends Expression

adrian-wang · 2015-06-30T04:21:31Z

Close it because #6782 (comment) for now

SparkQA · 2015-07-23T08:23:20Z

Test build #38195 has finished for PR 6986 at commit 5ef6a8c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-23T08:25:34Z

Test build #38197 has finished for PR 6986 at commit 1a9de3d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class LastDay(startDate: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class NextDay(left: Expression, right: Expression)

SparkQA · 2015-07-23T10:16:27Z

Test build #38203 has finished for PR 6986 at commit 2d0629b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class LastDay(startDate: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class NextDay(left: Expression, right: Expression)

adrian-wang · 2015-07-23T11:31:22Z

@rxin @chenghao-intel @zhichao-li

SparkQA · 2015-07-24T19:19:05Z

Test build #38382 has finished for PR 6986 at commit d09bcce.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class LastDay(startDate: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class NextDay(left: Expression, right: Expression)

rxin · 2015-07-26T07:53:00Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala

+/**
+ * Returns the last day of the month which the date belongs to.
+ */
+case class LastDay(startDate: Expression) extends UnaryExpression with ImplicitCastInputTypes {


add a prettyName last_day

SparkQA · 2015-07-27T11:34:50Z

Test build #38533 has finished for PR 6986 at commit 8846086.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class LastDay(startDate: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class NextDay(startDate: Expression, dayOfWeek: Expression)

rxin · 2015-07-27T20:19:43Z

...catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala

@@ -246,4 +247,19 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
    }
  }

+  test("last_day") {
+    checkEvaluation(LastDay(Literal(Date.valueOf("2015-07-23"))), Date.valueOf("2015-07-31"))


can you please add more unit tests?

you should test one for each month, and also test the leap years.

rxin · 2015-07-27T20:26:03Z

@adrian-wang I think this one is pretty close - we need to add more user facing documentation and internal comments, as well as unit tests.

Thanks.

rxin · 2015-07-28T02:07:15Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

+   * @group datetime_funcs
+   * @since 1.5.0
+   */
+  def next_day(sd: Column, dayOfWeek: Column): Column = NextDay(sd.expr, dayOfWeek.expr)


i think it is ok to not support that in the data frame api, since it is very unlikely somebody will be using a non literal.

rxin · 2015-07-28T02:08:27Z

Can you address these two issues?

rxin · 2015-07-28T02:08:48Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

@@ -2392,6 +2399,16 @@ object functions {
  def minute(columnName: String): Column = minute(Column(columnName))

  /**
+   * Returns the first date which is later than given date sd and named as dow.
+   * For example, next_day('2015-07-27', 'Sunday') would return 2015-08-02, which is the


quote next_day ... in backticks

adrian-wang · 2015-07-28T02:24:24Z

done

rxin · 2015-07-28T02:27:51Z

LGTM

SparkQA · 2015-07-28T02:42:59Z

Test build #38634 has finished for PR 6986 at commit 02b3426.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class LastDay(startDate: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class NextDay(startDate: Expression, dayOfWeek: Expression)

SparkQA · 2015-07-28T03:12:48Z

Test build #38645 has finished for PR 6986 at commit ef7e3da.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class LastDay(startDate: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class NextDay(startDate: Expression, dayOfWeek: Expression)

SparkQA · 2015-07-28T03:33:02Z

Test build #38627 has finished for PR 6986 at commit dc69630.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class LastDay(startDate: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class NextDay(startDate: Expression, dayOfWeek: Expression)

rxin · 2015-07-28T04:09:04Z

Thanks - I've merged this.

rxin · 2015-07-28T05:06:19Z

sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala

+      Seq(Row(Date.valueOf("2015-07-27")), Row(Date.valueOf("2015-07-27"))))
+    checkAnswer(
+      df2.select(next_day(col("t"), "th")),
+      Seq(Row(Date.valueOf("2015-07-30")), Row(null)))


note that this actually fails unit tests. I thought it passed but it was from a previous commit.

Can you please pay a little bit more attention in the future and at least run your own unit tests locally? Thanks.

Sorry for that...

adrian-wang closed this Jun 30, 2015

adrian-wang reopened this Jul 23, 2015

function next_day and last_day

1a9de3d

adrian-wang force-pushed the udfnlday branch from 5ef6a8c to 1a9de3d Compare July 23, 2015 08:17

adrian-wang changed the title ~~[SPARK-8195] [SPARK-8196] [WIP] [SQL] udf next_day last_day~~ [SPARK-8195] [SPARK-8196] [SQL] udf next_day last_day Jul 23, 2015

adrian-wang force-pushed the udfnlday branch from 2d0629b to 1a9de3d Compare July 24, 2015 17:35

multi fix

d09bcce

rxin reviewed Jul 26, 2015
View reviewed changes

address comments from rxin

8846086

rxin reviewed Jul 27, 2015
View reviewed changes

address comments from rxin

dc69630

rxin reviewed Jul 28, 2015
View reviewed changes

address 2 comments

02b3426

fix

ef7e3da

asfgit closed this in 2e7f99a Jul 28, 2015

rxin reviewed Jul 28, 2015
View reviewed changes

adrian-wang mentioned this pull request Jul 29, 2015

[SPARK-8186][SPARK-8187][SPARK-8194][SPARK-8198][SQL] functions: date_add, date_sub, add_months, months_between, time-interval calculation #7589

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8195] [SPARK-8196] [SQL] udf next_day last_day #6986

[SPARK-8195] [SPARK-8196] [SQL] udf next_day last_day #6986

adrian-wang commented Jun 24, 2015

SparkQA commented Jun 24, 2015

adrian-wang commented Jun 30, 2015

SparkQA commented Jul 23, 2015

SparkQA commented Jul 23, 2015

SparkQA commented Jul 23, 2015

adrian-wang commented Jul 23, 2015

SparkQA commented Jul 24, 2015

rxin Jul 26, 2015

adrian-wang Jul 27, 2015

SparkQA commented Jul 27, 2015

rxin Jul 27, 2015

rxin commented Jul 27, 2015

rxin Jul 28, 2015

rxin commented Jul 28, 2015

rxin Jul 28, 2015

adrian-wang commented Jul 28, 2015

rxin commented Jul 28, 2015

SparkQA commented Jul 28, 2015

SparkQA commented Jul 28, 2015

SparkQA commented Jul 28, 2015

rxin commented Jul 28, 2015

rxin Jul 28, 2015

adrian-wang Jul 28, 2015

[SPARK-8195] [SPARK-8196] [SQL] udf next_day last_day #6986

[SPARK-8195] [SPARK-8196] [SQL] udf next_day last_day #6986

Conversation

adrian-wang commented Jun 24, 2015

SparkQA commented Jun 24, 2015

adrian-wang commented Jun 30, 2015

SparkQA commented Jul 23, 2015

SparkQA commented Jul 23, 2015

SparkQA commented Jul 23, 2015

adrian-wang commented Jul 23, 2015

SparkQA commented Jul 24, 2015

rxin Jul 26, 2015

Choose a reason for hiding this comment

adrian-wang Jul 27, 2015

Choose a reason for hiding this comment

SparkQA commented Jul 27, 2015

rxin Jul 27, 2015

Choose a reason for hiding this comment

rxin commented Jul 27, 2015

rxin Jul 28, 2015

Choose a reason for hiding this comment

rxin commented Jul 28, 2015

rxin Jul 28, 2015

Choose a reason for hiding this comment

adrian-wang commented Jul 28, 2015

rxin commented Jul 28, 2015

SparkQA commented Jul 28, 2015

SparkQA commented Jul 28, 2015

SparkQA commented Jul 28, 2015

rxin commented Jul 28, 2015

rxin Jul 28, 2015

Choose a reason for hiding this comment

adrian-wang Jul 28, 2015

Choose a reason for hiding this comment