[SPARK-2190][SQL] Specialized ColumnType for Timestamp by liancheng · Pull Request #1440 · apache/spark

liancheng · 2014-07-16T10:09:31Z

Added specialized in-memory column type for Timestamp. Whitelisted all timestamp related Hive tests except timestamp_udf, which is timezone sensitive.

SparkQA · 2014-07-16T10:12:55Z

QA tests have started for PR 1440. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16725/consoleFull

SparkQA · 2014-07-16T11:48:53Z

QA results for PR 1440:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16725/consoleFull

liancheng · 2014-07-16T12:02:26Z

Hmm, just realized Timestamp.toString normalizes date and time according to current timezone and makes almost all timestamp related tests timezone sensitive. (Wouldn't notice this if I were in US...) Guess we have to blacklist them for now, and this will revert part of #1396.

ueshin · 2014-07-16T12:07:56Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

Hi, SimpleDateFormat is not thread-safe, so def should be used instead of val.

Just checked TimestampWritable.java, it's indeed handled with a thread local variable. Thanks for pointing this out!

liancheng · 2014-07-16T12:14:04Z

Confirmed that the following test cases are timezone sensitive and blacklisted them (by first remove all timestamp related golden answers, run them in my local timezone to generate new golden answers, then manually change my timezone settings and rerun these tests):

timestamp_1
timestamp_2
timestamp_3 *
timestamp_udf *

[*] Reverted from #1396.

SparkQA · 2014-07-16T12:22:53Z

QA tests have started for PR 1440. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16728/consoleFull

SparkQA · 2014-07-16T12:37:53Z

QA tests have started for PR 1440. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16729/consoleFull

SparkQA · 2014-07-16T13:59:07Z

QA results for PR 1440:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16728/consoleFull

SparkQA · 2014-07-16T14:14:04Z

QA results for PR 1440:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16729/consoleFull

marmbrus · 2014-07-16T14:52:25Z

sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnStats.scala

Nit: Spark style would probably prefer != here.

I think I read somewhere ne has better performance in certain cases...

Oh, I see... I forgot that ne/eq do reference equality... in the case of null I would imagine there is no difference, but this is probably fine then.

SparkQA · 2014-07-16T22:53:00Z

QA tests have started for PR 1440. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16749/consoleFull

SparkQA · 2014-07-17T00:31:54Z

QA results for PR 1440:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16749/consoleFull

SparkQA · 2014-07-21T05:18:11Z

QA tests have started for PR 1440. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16894/consoleFull

liancheng · 2014-07-21T05:26:37Z

Rebased to the most recent master, generated all golden answer files after setting timezone of my local machine to America/Los_Angeles and hardcoded the timezone in HiveCompatibilitySuite to enable those timezone sensitive tests. Hope Jenkins is happy with this.

SparkQA · 2014-07-21T05:28:10Z

QA tests have started for PR 1440. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16895/consoleFull

SparkQA · 2014-07-21T06:56:21Z

QA results for PR 1440:
- This patch PASSES unit tests.

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16894/consoleFull

SparkQA · 2014-07-21T07:10:27Z

QA results for PR 1440:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16895/consoleFull

marmbrus · 2014-07-21T07:47:47Z

Thanks! I've merged this into master.

JIRA issue: [SPARK-2190](https://issues.apache.org/jira/browse/SPARK-2190) Added specialized in-memory column type for `Timestamp`. Whitelisted all timestamp related Hive tests except `timestamp_udf`, which is timezone sensitive. Author: Cheng Lian <lian.cs.zju@gmail.com> Closes apache#1440 from liancheng/timestamp-column-type and squashes the following commits: e682175 [Cheng Lian] Enabled more timezone sensitive Hive tests. 53a358f [Cheng Lian] Fixed failed test suites 01b592d [Cheng Lian] Fixed SimpleDateFormat thread safety issue 2a59343 [Cheng Lian] Removed timezone sensitive Hive timestamp tests 45dd05d [Cheng Lian] Added Timestamp specific in-memory columnar representation

ueshin reviewed Jul 16, 2014
View reviewed changes

marmbrus reviewed Jul 16, 2014
View reviewed changes

liancheng added 3 commits July 21, 2014 13:15

Added Timestamp specific in-memory columnar representation

45dd05d

Removed timezone sensitive Hive timestamp tests

2a59343

Fixed SimpleDateFormat thread safety issue

01b592d

liancheng added 2 commits July 21, 2014 13:23

Fixed failed test suites

53a358f

Enabled more timezone sensitive Hive tests.

e682175

asfgit closed this in cd273a2 Jul 21, 2014

liancheng deleted the timestamp-column-type branch September 5, 2014 20:52

Conversation

liancheng commented Jul 16, 2014

Uh oh!

SparkQA commented Jul 16, 2014

Uh oh!

SparkQA commented Jul 16, 2014

Uh oh!

liancheng commented Jul 16, 2014

Uh oh!

ueshin Jul 16, 2014

Choose a reason for hiding this comment

Uh oh!

liancheng Jul 16, 2014

Choose a reason for hiding this comment

Uh oh!

liancheng commented Jul 16, 2014

Uh oh!

SparkQA commented Jul 16, 2014

Uh oh!

SparkQA commented Jul 16, 2014

Uh oh!

SparkQA commented Jul 16, 2014

Uh oh!

SparkQA commented Jul 16, 2014

Uh oh!

marmbrus Jul 16, 2014

Choose a reason for hiding this comment

Uh oh!

concretevitamin Jul 16, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus Jul 16, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 16, 2014

Uh oh!

SparkQA commented Jul 17, 2014

Uh oh!

SparkQA commented Jul 21, 2014

Uh oh!

liancheng commented Jul 21, 2014

Uh oh!

SparkQA commented Jul 21, 2014

Uh oh!

SparkQA commented Jul 21, 2014

Uh oh!

SparkQA commented Jul 21, 2014

Uh oh!

marmbrus commented Jul 21, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants