[SPARK-11396] [SQL] add native implementation of datetime function to_unix_timestamp #9347

adrian-wang · 2015-10-29T05:29:29Z

to_unix_timestamp is the deterministic version of unix_timestamp, as it accepts at least one parameters.

Since the behavior here is quite similar to unix_timestamp, I think the dataframe API is not necessary here.

SparkQA · 2015-10-29T07:44:08Z

Test build #44565 has finished for PR 9347 at commit 7c595fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class ToUnixTimestamp(timeExp: Expression, format: Expression) extends UnixTime\n * case class UnixTimestamp(timeExp: Expression, format: Expression) extends UnixTime\n * abstract class UnixTime extends BinaryExpression with ExpectsInputTypes\n

SparkQA · 2015-10-29T07:56:17Z

Test build #44566 has finished for PR 9347 at commit c684a7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class ToUnixTimestamp(timeExp: Expression, format: Expression) extends UnixTime\n * case class UnixTimestamp(timeExp: Expression, format: Expression) extends UnixTime\n * abstract class UnixTime extends BinaryExpression with ExpectsInputTypes\n

adrian-wang · 2015-10-30T05:00:24Z

@liancheng

davies · 2015-11-10T07:02:18Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala

How about move this into UnixTime?

davies · 2015-11-10T07:02:47Z

LGTM overall, just one minor comment.

SparkQA · 2015-11-10T08:54:29Z

Test build #45509 has finished for PR 9347 at commit e6201c5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class ToUnixTimestamp(timeExp: Expression, format: Expression)\n * abstract class UnixTime(timeExp: Expression, format: Expression)\n

adrian-wang · 2015-11-10T09:25:40Z

retest this please.

SparkQA · 2015-11-10T11:26:34Z

Test build #45513 has finished for PR 9347 at commit e6201c5.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class ToUnixTimestamp(timeExp: Expression, format: Expression)\n * abstract class UnixTime(timeExp: Expression, format: Expression)\n

yhuai · 2015-11-11T00:47:21Z

Seems a legitimate failure (java.io.InvalidClassException: org.apache.spark.sql.catalyst.expressions.UnixTimestamp; no valid constructor).

davies · 2015-11-11T00:49:52Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala

use format here

SparkQA · 2015-11-11T08:17:52Z

Test build #45602 has finished for PR 9347 at commit 533775f.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class ToUnixTimestamp(timeExp: Expression, format: Expression)\n * abstract class UnixTime(timeExp: Expression, format: Expression)\n

davies · 2015-11-11T08:32:59Z

Please fix the bug for SparkR

java.io.InvalidClassException: org.apache.spark.sql.catalyst.expressions.UnixTimestamp; no valid constructor
    at java.io.ObjectStreamClass$ExceptionInfo.newInvalidClassException(ObjectStreamClass.java:150)
    at java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:768)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1772)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
    at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
    at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

SparkQA · 2015-11-11T11:12:42Z

Test build #45627 has finished for PR 9347 at commit 1636f72.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class ToUnixTimestamp(timeExp: Expression, format: Expression)\n * abstract class UnixTime(timeExp: Expression, format: Expression)\n

adrian-wang · 2015-11-11T11:34:51Z

retest this please.

SparkQA · 2015-11-11T13:58:42Z

Test build #45629 has finished for PR 9347 at commit 1636f72.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class ToUnixTimestamp(timeExp: Expression, format: Expression)\n * abstract class UnixTime(timeExp: Expression, format: Expression)\n

adrian-wang · 2015-11-11T16:52:41Z

default constructor has conflict here, so maybe we have to avoid parameters in abstract class UnixTime.

SparkQA · 2015-11-11T16:52:45Z

Test build #45646 has started for PR 9347 at commit b2fca64.

SparkQA · 2015-11-11T18:50:15Z

Test build #2041 has finished for PR 9347 at commit b2fca64.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class ToUnixTimestamp(timeExp: Expression, format: Expression) extends UnixTime\n * case class UnixTimestamp(timeExp: Expression, format: Expression) extends UnixTime\n * abstract class UnixTime extends BinaryExpression with ExpectsInputTypes\n

adrian-wang · 2015-11-12T01:59:39Z

retest this please.

SparkQA · 2015-11-12T04:33:13Z

Test build #45699 has finished for PR 9347 at commit b2fca64.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class ToUnixTimestamp(timeExp: Expression, format: Expression) extends UnixTime\n * case class UnixTimestamp(timeExp: Expression, format: Expression) extends UnixTime\n * abstract class UnixTime extends BinaryExpression with ExpectsInputTypes\n

davies · 2015-11-12T04:36:06Z

LGTM, merging this into master and 1.6.

…_unix_timestamp `to_unix_timestamp` is the deterministic version of `unix_timestamp`, as it accepts at least one parameters. Since the behavior here is quite similar to `unix_timestamp`, I think the dataframe API is not necessary here. Author: Daoyuan Wang <daoyuan.wang@intel.com> Closes #9347 from adrian-wang/to_unix_timestamp. (cherry picked from commit 39b1e36) Signed-off-by: Davies Liu <davies.liu@gmail.com>

…_unix_timestamp `to_unix_timestamp` is the deterministic version of `unix_timestamp`, as it accepts at least one parameters. Since the behavior here is quite similar to `unix_timestamp`, I think the dataframe API is not necessary here. Author: Daoyuan Wang <daoyuan.wang@intel.com> Closes apache#9347 from adrian-wang/to_unix_timestamp.

adrian-wang added 2 commits October 28, 2015 22:20

add native implementation of to_unix_timestamp

7c595fd

add doc

c684a7b

davies reviewed Nov 10, 2015
View reviewed changes

move left and right to parent class

e6201c5

davies reviewed Nov 11, 2015
View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala

Copy link

Contributor

davies Nov 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use format here

retest this please

533775f

add self reference

1636f72

not using this abstract class

b2fca64

asfgit closed this in 39b1e36 Nov 12, 2015

[SPARK-11396] [SQL] add native implementation of datetime function to_unix_timestamp #9347

[SPARK-11396] [SQL] add native implementation of datetime function to_unix_timestamp #9347

Uh oh!

Conversation

adrian-wang commented Oct 29, 2015

Uh oh!

SparkQA commented Oct 29, 2015

Uh oh!

SparkQA commented Oct 29, 2015

Uh oh!

adrian-wang commented Oct 30, 2015

Uh oh!

davies Nov 10, 2015

Choose a reason for hiding this comment

Uh oh!

davies commented Nov 10, 2015

Uh oh!

SparkQA commented Nov 10, 2015

Uh oh!

adrian-wang commented Nov 10, 2015

Uh oh!

SparkQA commented Nov 10, 2015

Uh oh!

yhuai commented Nov 11, 2015

Uh oh!

davies Nov 11, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 11, 2015

Uh oh!

davies commented Nov 11, 2015

Uh oh!

SparkQA commented Nov 11, 2015

Uh oh!

adrian-wang commented Nov 11, 2015

Uh oh!

SparkQA commented Nov 11, 2015

Uh oh!

adrian-wang commented Nov 11, 2015

Uh oh!

SparkQA commented Nov 11, 2015

Uh oh!

SparkQA commented Nov 11, 2015

Uh oh!

adrian-wang commented Nov 12, 2015

Uh oh!

SparkQA commented Nov 12, 2015

Uh oh!

davies commented Nov 12, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants