Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-10522] [SQL] Nanoseconds of Timestamp in Parquet should be positive #8674

Closed
wants to merge 1 commit into from
Closed

[SPARK-10522] [SQL] Nanoseconds of Timestamp in Parquet should be positive #8674

wants to merge 1 commit into from

Conversation

davies
Copy link
Contributor

@davies davies commented Sep 9, 2015

Or Hive can't read it back correctly.

Thanks @vanzin for report this.

vanzin
vanzin reviewed Sep 9, 2015
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
val secondsInDay = seconds % SECONDS_PER_DAY
val nanos = (us % MICROS_PER_SECOND) * 1000L
(day.toInt, secondsInDay * NANOS_PER_SECOND + nanos)
val julian_us = us + JULIAN_DAY_OF_EPOCH * MICROS_PER_DAY
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without the checks I'm adding in #8606 this can overflow and write garbage to the file...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We haven't handle overflow in many places, the behavior is undefined, it means will be garbage in the file, I think it's fine for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we throw an exception, the job will crash if there are some garbage rows in it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. I'll close my PR and leave the bug open.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long (2^63) is enough for 584942 years before overflow, so don't worry about it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about adding some comments at here?

@vanzin
Copy link
Contributor

vanzin commented Sep 9, 2015

LGTM.

@SparkQA
Copy link

SparkQA commented Sep 10, 2015

Test build #42221 has finished for PR 8674 at commit 337727f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@davies
Copy link
Contributor Author

davies commented Sep 14, 2015

Merged into master and 1.5 branch (added comment while merging).

asfgit pushed a commit that referenced this pull request Sep 14, 2015
@davies
[SPARK-10522] [SQL] Nanoseconds of Timestamp in Parquet should be pos…
a0d564a 
…itive

Or Hive can't read it back correctly.

Thanks vanzin for report this.

Author: Davies Liu <davies@databricks.com>

Closes #8674 from davies/positive_nano.

(cherry picked from commit 7e32387)
Signed-off-by: Davies Liu <davies.liu@gmail.com>
@asfgit asfgit closed this in 7e32387 Sep 14, 2015
ashangit pushed a commit to ashangit/spark that referenced this pull request Oct 19, 2016
[SPARK-10522] [SQL] Nanoseconds of Timestamp in Parquet should be pos…
94decd8 
…itive

Or Hive can't read it back correctly.

Thanks vanzin for report this.

Author: Davies Liu <davies@databricks.com>

Closes apache#8674 from davies/positive_nano.

(cherry picked from commit 7e32387)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@davies @vanzin @SparkQA @yhuai