[SPARK-8186][SPARK-8187][SPARK-8194][SPARK-8198][SPARK-9133] [SPARK-9290] [SQL] functions: date_add, date_sub, add_months, months_between, time-interval calculation #7754

davies · 2015-07-29T18:39:55Z

This PR is based on #7589 , thanks to @adrian-wang

Added SQL function date_add, date_sub, add_months, month_between, also add a rule for
add/subtract of date/timestamp and interval.

Closes #7589

cc @rxin

Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala sql/core/src/main/scala/org/apache/spark/sql/functions.scala sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala

SparkQA · 2015-07-29T18:46:37Z

Test build #38865 has finished for PR 7754 at commit 989b5b9.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)
- case class AddMonths(startDate: Expression, numMonths: Expression)
- case class MonthsBetween(date1: Expression, date2: Expression)

SparkQA · 2015-07-29T19:05:52Z

Test build #38866 has finished for PR 7754 at commit e47ff2c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)
- case class AddMonths(startDate: Expression, numMonths: Expression)
- case class MonthsBetween(date1: Expression, date2: Expression)

rxin · 2015-07-29T22:35:17Z

I'm going to look through this. You will need to rebase because of #7745

rxin · 2015-07-29T22:37:15Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala

+        Add(right, left) // switch the order
+
+      case Add(left, right) if right.dataType == IntervalType =>
+        Cast(TimeAdd(Cast(left, TimestampType), right), left.dataType)


this implicitly casts all types to TimestampType. How about requiring left to be string, date, or timestamp?

rxin · 2015-07-29T22:51:00Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala

+   * Turns Add/Subtract of DateType/TimestampType/StringType and CalendarIntervalType
+   * to TimeAdd/TimeSub
+   */
+  object DateTimeOperations extends Rule[LogicalPlan] {


can you add a unit test for this rule?

make sure there is one test case just for this rule, and another that tests DateTimeOperations and ImplicitTypeCasts in combination to make sure they still work. I worry if we change ImplicitTypeCasts in the future, DateTimeOperations might break (e.g. the interval got converted to string type)

done, the combination of them will be covered by function tests.

SparkQA · 2015-07-29T22:58:17Z

Test build #38901 has finished for PR 7754 at commit bf0e9db.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2015-07-29T23:00:04Z

cc @yjshen to take a look at the date time function implementations too

rxin · 2015-07-29T23:02:03Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala

+
+  /**
+   * Returns number of months between time1 and time2. time1 and time2 are expressed in
+   * microseconds since 1.1.1970


we should document that this returns an integer if it is the same day in two different month

SparkQA · 2015-07-29T23:07:51Z

Test build #38902 has finished for PR 7754 at commit 061e012.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-29T23:35:11Z

Test build #38909 has finished for PR 7754 at commit 3efd86e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-30T00:59:18Z

Test build #38929 has finished for PR 7754 at commit 5b7af5b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)
- case class AddMonths(startDate: Expression, numMonths: Expression)
- case class MonthsBetween(date1: Expression, date2: Expression)

yjshen · 2015-07-30T04:30:00Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala

+    } else {
+      dayOfMonth
+    }
+    firstDayOfMonth(absoluteMonth) + currentDayInMonth - 1


getYear,getMonth, getDayOfMonth will call getYearAndDayInYear first, therefore 3 times in total. If performance is a consideration here, I think we could have a function (date) => (year, month, dayOfMonth)

yes, we can do that later.

SparkQA · 2015-07-30T06:07:23Z

Test build #39001 has finished for PR 7754 at commit 1d07de1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)
- case class AddMonths(startDate: Expression, numMonths: Expression)
- case class MonthsBetween(date1: Expression, date2: Expression)

SparkQA · 2015-07-30T07:04:09Z

Test build #39017 has finished for PR 7754 at commit 237452b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)
- case class AddMonths(startDate: Expression, numMonths: Expression)
- case class MonthsBetween(date1: Expression, date2: Expression)

SparkQA · 2015-07-30T07:51:16Z

Test build #39011 has finished for PR 7754 at commit 127596c.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)
- case class AddMonths(startDate: Expression, numMonths: Expression)
- case class MonthsBetween(date1: Expression, date2: Expression)

SparkQA · 2015-07-30T08:47:10Z

Test build #39020 has finished for PR 7754 at commit b923157.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)
- case class AddMonths(startDate: Expression, numMonths: Expression)
- case class MonthsBetween(date1: Expression, date2: Expression)

SparkQA · 2015-07-30T18:39:56Z

Test build #1239 has finished for PR 7754 at commit 1c4553d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-30T18:40:20Z

Test build #39070 has finished for PR 7754 at commit 6224ce4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)
- case class AddMonths(startDate: Expression, numMonths: Expression)
- case class MonthsBetween(date1: Expression, date2: Expression)

Conflicts: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala

davies · 2015-07-30T18:57:11Z

@rxin If no more objection, I will merge this once it pass the tests.

rxin · 2015-07-30T19:00:14Z

LGTM

SparkQA · 2015-07-30T20:14:18Z

Test build #39083 has finished for PR 7754 at commit 446f762.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class UnixTimestamp(timeExp: Expression, format: Expression)
- case class FromUnixTime(sec: Expression, format: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)
- case class AddMonths(startDate: Expression, numMonths: Expression)
- case class MonthsBetween(date1: Expression, date2: Expression)
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

SparkQA · 2015-07-30T20:40:34Z

Test build #39084 has finished for PR 7754 at commit 9e8e085.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class UnixTimestamp(timeExp: Expression, format: Expression)
- case class FromUnixTime(sec: Expression, format: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)
- case class AddMonths(startDate: Expression, numMonths: Expression)
- case class MonthsBetween(date1: Expression, date2: Expression)

adrian-wang and others added 8 commits July 22, 2015 00:04

function date_add , date_sub

c506661

poc of time interval calculation

1a68e03

function add_months, months_between and some fixes

87c4b77

fix style

42df486

fix

e8a639a

fix

522e91a

add python api, fix date functions

e47ff2c

davies force-pushed the date_add branch from 989b5b9 to e47ff2c Compare July 29, 2015 18:45

Merge branch 'master' of github.com:apache/spark into date_add

bd18cd4

rxin reviewed Jul 29, 2015
View reviewed changes

davies force-pushed the date_add branch from 061e012 to 3efd86e Compare July 29, 2015 23:28

davies force-pushed the date_add branch from 3efd86e to 5b7af5b Compare July 30, 2015 00:35

yjshen reviewed Jul 30, 2015
View reviewed changes

davies force-pushed the date_add branch from 5b7af5b to 1d07de1 Compare July 30, 2015 05:46

davies force-pushed the date_add branch 2 times, most recently from 127596c to 237452b Compare July 30, 2015 06:45

davies force-pushed the date_add branch from 237452b to b923157 Compare July 30, 2015 07:12

fix conclict

6224ce4

davies force-pushed the date_add branch from 1c4553d to 6224ce4 Compare July 30, 2015 16:55

Merge branch 'master' of github.com:apache/spark into date_add

9e8e085

Conflicts: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala

davies force-pushed the date_add branch from 446f762 to 9e8e085 Compare July 30, 2015 18:56

Merge branch 'master' of github.com:apache/spark into date_add

e8c633a

asfgit closed this in 1abf7dc Jul 30, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8186][SPARK-8187][SPARK-8194][SPARK-8198][SPARK-9133] [SPARK-9290] [SQL] functions: date_add, date_sub, add_months, months_between, time-interval calculation #7754

[SPARK-8186][SPARK-8187][SPARK-8194][SPARK-8198][SPARK-9133] [SPARK-9290] [SQL] functions: date_add, date_sub, add_months, months_between, time-interval calculation #7754

davies commented Jul 29, 2015

SparkQA commented Jul 29, 2015

SparkQA commented Jul 29, 2015

rxin commented Jul 29, 2015

rxin Jul 29, 2015

rxin Jul 29, 2015

davies Jul 30, 2015

SparkQA commented Jul 29, 2015

rxin commented Jul 29, 2015

rxin Jul 29, 2015

SparkQA commented Jul 29, 2015

SparkQA commented Jul 29, 2015

SparkQA commented Jul 30, 2015

yjshen Jul 30, 2015

davies Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015

davies commented Jul 30, 2015

rxin commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015

[SPARK-8186][SPARK-8187][SPARK-8194][SPARK-8198][SPARK-9133] [SPARK-9290] [SQL] functions: date_add, date_sub, add_months, months_between, time-interval calculation #7754

[SPARK-8186][SPARK-8187][SPARK-8194][SPARK-8198][SPARK-9133] [SPARK-9290] [SQL] functions: date_add, date_sub, add_months, months_between, time-interval calculation #7754

Conversation

davies commented Jul 29, 2015

SparkQA commented Jul 29, 2015

SparkQA commented Jul 29, 2015

rxin commented Jul 29, 2015

rxin Jul 29, 2015

Choose a reason for hiding this comment

rxin Jul 29, 2015

Choose a reason for hiding this comment

davies Jul 30, 2015

Choose a reason for hiding this comment

SparkQA commented Jul 29, 2015

rxin commented Jul 29, 2015

rxin Jul 29, 2015

Choose a reason for hiding this comment

SparkQA commented Jul 29, 2015

SparkQA commented Jul 29, 2015

SparkQA commented Jul 30, 2015

yjshen Jul 30, 2015

Choose a reason for hiding this comment

davies Jul 30, 2015

Choose a reason for hiding this comment

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015

davies commented Jul 30, 2015

rxin commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 30, 2015