[SPARK-8199][SPARK-8184][SPARK-8183][SPARK-8182][SPARK-8181][SPARK-8180][SPARK-8179][SPARK-8177][SPARK-8178][SPARK-9115][SQL] date functions #6981

tarekbecker · 2015-06-24T08:53:02Z

Jira:
https://issues.apache.org/jira/browse/SPARK-8199
https://issues.apache.org/jira/browse/SPARK-8184
https://issues.apache.org/jira/browse/SPARK-8183
https://issues.apache.org/jira/browse/SPARK-8182
https://issues.apache.org/jira/browse/SPARK-8181
https://issues.apache.org/jira/browse/SPARK-8180
https://issues.apache.org/jira/browse/SPARK-8179
https://issues.apache.org/jira/browse/SPARK-8177
https://issues.apache.org/jira/browse/SPARK-8179
https://issues.apache.org/jira/browse/SPARK-9115

Regarding dayand dayofmonth are both necessary?

~~I am going to add Quarter to this PR as well.~~ Done.

~~As soon as the Scala coding is reviewed and discussed, I'll add the python api.~~ Done

tarekbecker · 2015-06-24T08:54:55Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetime.scala

+  override def nullable: Boolean = true
+
+  override def eval(input: InternalRow): Any = {
+    DateFormat(child, Literal("y")).eval(input) match {


Is it okay to call DateFormat(child, Literal("y")).eval(input) or is there a more elegant way.

Is there a way to call genCode of DataFormat?

adrian-wang · 2015-06-24T09:02:09Z

I think this should wait #6782 , since we are creating a file with same usage.

fixed test cases dataframe test issues: - dateformat with date string is null - hours, minutes, seconds is zero for timestamp

tarekbecker · 2015-06-24T09:58:03Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetime.scala

+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+case class DateFormatClass(left: Expression, right: Expression)


If name is DateFormat build failes in line

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala

Line 72 in 6b7f2ce

def toString(days: Int): String = Cast.threadLocalDateFormat.get.format(toJavaDate(days))

tarekbecker · 2015-06-24T17:26:32Z

@davies @rxin
Is it possbile to use ExpectsInputTypes with a a type similar to Any. If I dont know, how to specifyDateTypeOR StringType. If I useoverride def expectedChildTypes: Seq[DataType] = Seq(DateType, StringType` it looses the time information. (Hour, Minute, Second test will fail)

tarekbecker · 2015-06-24T17:44:36Z

@adrian-wang I guess we can implement the stuff already. A rebase or merge should be trivial.

tarekbecker · 2015-06-24T18:55:53Z

Code generation is adopted from

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala

Line 191 in bc76a0f

protected def defineCodeGen(

yjshen · 2015-06-26T09:56:00Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetime.scala

+            UTF8String.fromString(sdf.format(DateTimeUtils.toJavaDate(valueLeft.asInstanceOf[Int])))
+          case StringType =>
+            UTF8String.fromString(
+              sdf.format(DateTimeUtils.stringToTime(valueLeft.asInstanceOf[UTF8String].toString)))


just valueLeft.toString

NAVER - http://www.naver.com/

sujkh@naver.com 님께 보내신 메일 <Re: [spark] [SPARK-8199][SPARK-8184][SPARK-8183][SPARK-8182][SPARK-8181][SPARK-8180][SPARK-8179][SPARK-8177][SPARK-8178][SQL] date functions (#6981)> 이 다음과 같은 이유로 전송 실패했습니다.

받는 사람이 회원님의 메일을 수신차단 하였습니다.

tarekbecker · 2015-06-26T23:03:56Z

@YijieSHEN Thanks for all your feedback. I changed the things you mentioned. Have you an idea to improve DateFormatClass? The problem is I have to provide format and for that particular case I have to evaluate format first. Is there a way to combine these two?

davies · 2015-06-29T03:52:26Z

ok to test

SparkQA · 2015-06-29T03:59:54Z

Test build #35952 has finished for PR 6981 at commit 356df78.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class DateFormatExpression extends UnaryExpression with ExpectsInputTypes
- case class DateFormatClass(left: Expression, right: Expression) extends BinaryExpression
- case class Year(child: Expression) extends DateFormatExpression with ExpectsInputTypes
- case class Quarter(child: Expression) extends DateFormatExpression
- case class Month(child: Expression) extends DateFormatExpression with ExpectsInputTypes
- case class Day(child: Expression) extends DateFormatExpression with ExpectsInputTypes
- case class Hour(child: Expression) extends DateFormatExpression with ExpectsInputTypes
- case class Minute(child: Expression) extends DateFormatExpression with ExpectsInputTypes
- case class Second(child: Expression) extends DateFormatExpression with ExpectsInputTypes
- case class WeekOfYear(child: Expression) extends DateFormatExpression with ExpectsInputTypes

SparkQA · 2015-07-18T03:26:23Z

Test build #37680 has finished for PR 6981 at commit 4afc09c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Hour(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Minute(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Second(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DayInYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Year(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Quarter(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Month(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Day(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DateFormatClass(left: Expression, right: Expression) extends BinaryExpression

…mment in apache#7488

SparkQA · 2015-07-18T14:37:47Z

Test build #37717 has finished for PR 6981 at commit 6e0c78f.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Hour(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Minute(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Second(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DayInYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Year(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Quarter(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Month(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Day(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DateFormatClass(left: Expression, right: Expression) extends BinaryExpression

SparkQA · 2015-07-18T16:19:04Z

Test build #37718 has finished for PR 6981 at commit 5983dcc.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Hour(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Minute(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Second(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DayInYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Year(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Quarter(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Month(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Day(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DateFormatClass(left: Expression, right: Expression) extends BinaryExpression

tarekbecker · 2015-07-18T18:07:46Z

@rxin / @davies First of all thanks for all your feedback so far. I removed the manual binary search. I guess we all agree that the if/else structure is much more readable.

@davies I removed testWithTimezone because of @cloud-fan comment in #7488.

@cloud-fan Could you have a look on this PR, if you have time? Would be great.

SparkQA · 2015-07-18T19:32:49Z

Test build #37725 has finished for PR 6981 at commit 256c357.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Hour(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Minute(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Second(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DayInYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Year(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Quarter(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Month(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Day(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DateFormatClass(left: Expression, right: Expression) extends BinaryExpression

davies · 2015-07-18T19:38:14Z

python/pyspark/sql/functions.py

+    specialized implementation.
+
+    >>> df0 = sqlContext.createDataFrame([('2015-04-08',)], ['a'])
+


minor: Why do we have this blank line?

davies · 2015-07-18T20:01:56Z

@tarekauel This is pretty close now, one thing left is that the int for DateType is in UTC, in order to get millisecond, we need to minus offset of timezone, see daysToMillis. But once you switch the timezone for Calendar to UTC, no offset anymore (it's zero).

SparkQA · 2015-07-18T20:43:35Z

Test build #37729 has finished for PR 6981 at commit 3e095ba.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Hour(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Minute(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Second(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DayInYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Year(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Quarter(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Month(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Day(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DateFormatClass(left: Expression, right: Expression) extends BinaryExpression

SparkQA · 2015-07-19T00:45:43Z

Test build #37742 has finished for PR 6981 at commit bb567b6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Hour(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Minute(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Second(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DayInYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Year(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Quarter(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Month(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Day(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DateFormatClass(left: Expression, right: Expression) extends BinaryExpression

SparkQA · 2015-07-19T03:10:12Z

Test build #37746 has finished for PR 6981 at commit f7b4c8c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Hour(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Minute(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Second(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DayInYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Year(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Quarter(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Month(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Day(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class DateFormatClass(left: Expression, right: Expression) extends BinaryExpression

davies · 2015-07-19T04:30:15Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateFunctionsSuite.scala

+        (0 to 5).foreach { i =>
+          val c = Calendar.getInstance()
+          c.set(y, m, 28, 0, 0, 0)
+          c.add(Calendar.DATE, 1)


rxin · 2015-07-19T05:47:53Z

Thanks @tarekauel. I'm going to merge this and push a commit to fix that two problems @davies pointed out.

davies · 2015-07-19T05:50:59Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateFunctionsSuite.scala

+        (0 to 5).foreach { i =>
+          val c = Calendar.getInstance()
+          c.set(y, m, 28, 0, 0, 0)
+          c.add(Calendar.DATE, 1)


rxin · 2015-07-19T05:51:19Z

@tarekauel congratulations. this is a big pr and a major step for date functions.

davies · 2015-07-19T05:51:38Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateFunctionsSuite.scala

+        (0 to 5).foreach { i =>
+          val c = Calendar.getInstance()
+          c.set(y, m, 28, 0, 0, 0)
+          c.add(Calendar.DATE, 1)


I changed this when I looked for the last bug. I'm going to create a follow PR

…stems. This pull request fixes some of the problems in #6981. - Added date functions to `__all__` so they get exposed - Rename day_of_month -> dayofmonth - Rename day_in_year -> dayofyear - Rename week_of_year -> weekofyear - Removed "day" from Scala/Python API since it is ambiguous. Only leaving the alias in SQL. Author: Reynold Xin <rxin@databricks.com> This patch had conflicts when merged, resolved by Committer: Reynold Xin <rxin@databricks.com> Closes #7506 from rxin/datetime and squashes the following commits: 0cb24d9 [Reynold Xin] Export all functions in Python. e44a4a0 [Reynold Xin] Removed day function from Scala and Python. 9c08fdc [Reynold Xin] [SQL] Make date/time functions more consistent with other database systems.

date functions

d0e2f99

tarekbecker reviewed Jun 24, 2015
View reviewed changes

tarekbecker changed the title ~~[SPARK-8199][SPARK-8184][SPARK-8183][SPARK-8182][SPARK-8181][SPARK-8180][SPARK-8179][SPARK-8177] date functions~~ [SPARK-8199][SPARK-8184][SPARK-8183][SPARK-8182][SPARK-8181][SPARK-8180][SPARK-8179][SPARK-8177][SQL] date functions Jun 24, 2015

resolved naming conflict

5ebb235

fixed test cases dataframe test issues: - dateformat with date string is null - hours, minutes, seconds is zero for timestamp

tarekbecker reviewed Jun 24, 2015
View reviewed changes

tarekbecker added 3 commits June 24, 2015 09:41

fixed tests and added type check

4d8049b

improved codegen

638596f

fixed stupid test

849fb41

added support for quarter SPARK-8178

c739788

tarekbecker changed the title ~~[SPARK-8199][SPARK-8184][SPARK-8183][SPARK-8182][SPARK-8181][SPARK-8180][SPARK-8179][SPARK-8177][SQL] date functions~~ [SPARK-8199][SPARK-8184][SPARK-8183][SPARK-8182][SPARK-8181][SPARK-8180][SPARK-8179][SPARK-8177][SPARK-8178][SQL] date functions Jun 24, 2015

added codegeneration to all functions

b680db6

added python api; changed test to be more meaningful

a5ea120

yjshen reviewed Jun 26, 2015
View reviewed changes

removed doubled code

02efc5d

rely on cast mechanism of Spark. Simplified implementation

356df78

fixed style

3bfac90

[SPARK-8199] removed setTimeZone in tests, according to cloud-fans co…

6e0c78f

…mment in apache#7488

[SPARK-8199] whitespace fix

5983dcc

[SPARK-8199] code cleanup

256c357

davies reviewed Jul 18, 2015
View reviewed changes

[SPARK-8199] style and timezone fix

3e095ba

[SPARK-8199] fixed test

bb567b6

[SPARK-8199] fixed bug in tests

f7b4c8c

davies reviewed Jul 19, 2015
View reviewed changes

asfgit closed this in 83b682b Jul 19, 2015

davies reviewed Jul 19, 2015
View reviewed changes

rxin mentioned this pull request Jul 19, 2015

[SQL] Make date/time functions more consistent with other database systems. #7506

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8199][SPARK-8184][SPARK-8183][SPARK-8182][SPARK-8181][SPARK-8180][SPARK-8179][SPARK-8177][SPARK-8178][SPARK-9115][SQL] date functions #6981

[SPARK-8199][SPARK-8184][SPARK-8183][SPARK-8182][SPARK-8181][SPARK-8180][SPARK-8179][SPARK-8177][SPARK-8178][SPARK-9115][SQL] date functions #6981

tarekbecker commented Jun 24, 2015

tarekbecker Jun 24, 2015

adrian-wang commented Jun 24, 2015

tarekbecker Jun 24, 2015

tarekbecker commented Jun 24, 2015

tarekbecker commented Jun 24, 2015

tarekbecker commented Jun 24, 2015

yjshen Jun 26, 2015

sujkh85 Jun 26, 2015

tarekbecker commented Jun 26, 2015

davies commented Jun 29, 2015

SparkQA commented Jun 29, 2015

SparkQA commented Jul 18, 2015

SparkQA commented Jul 18, 2015

SparkQA commented Jul 18, 2015

tarekbecker commented Jul 18, 2015

SparkQA commented Jul 18, 2015

davies Jul 18, 2015

davies commented Jul 18, 2015

SparkQA commented Jul 18, 2015

SparkQA commented Jul 19, 2015

SparkQA commented Jul 19, 2015

davies Jul 19, 2015

rxin commented Jul 19, 2015

davies Jul 19, 2015

rxin commented Jul 19, 2015

davies Jul 19, 2015

tarekbecker Jul 19, 2015

		specialized implementation.

		>>> df0 = sqlContext.createDataFrame([('2015-04-08',)], ['a'])

[SPARK-8199][SPARK-8184][SPARK-8183][SPARK-8182][SPARK-8181][SPARK-8180][SPARK-8179][SPARK-8177][SPARK-8178][SPARK-9115][SQL] date functions #6981

[SPARK-8199][SPARK-8184][SPARK-8183][SPARK-8182][SPARK-8181][SPARK-8180][SPARK-8179][SPARK-8177][SPARK-8178][SPARK-9115][SQL] date functions #6981

Conversation

tarekbecker commented Jun 24, 2015

Choose a reason for hiding this comment

adrian-wang commented Jun 24, 2015

Choose a reason for hiding this comment

tarekbecker commented Jun 24, 2015

tarekbecker commented Jun 24, 2015

tarekbecker commented Jun 24, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NAVER - http://www.naver.com/

tarekbecker commented Jun 26, 2015

davies commented Jun 29, 2015

SparkQA commented Jun 29, 2015

SparkQA commented Jul 18, 2015

SparkQA commented Jul 18, 2015

SparkQA commented Jul 18, 2015

tarekbecker commented Jul 18, 2015

SparkQA commented Jul 18, 2015

Choose a reason for hiding this comment

davies commented Jul 18, 2015

SparkQA commented Jul 18, 2015

SparkQA commented Jul 19, 2015

SparkQA commented Jul 19, 2015

Choose a reason for hiding this comment

rxin commented Jul 19, 2015

Choose a reason for hiding this comment

rxin commented Jul 19, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment