Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Parquet reads of CORRECTED timestamps prior to 1590 are wrong #132

Open
revans2 opened this issue Jun 9, 2020 · 2 comments
Open

[BUG] Parquet reads of CORRECTED timestamps prior to 1590 are wrong #132

revans2 opened this issue Jun 9, 2020 · 2 comments
Labels
bug Something isn't working P2 Not required for release SQL part of the SQL/Dataframe plugin

Comments

@revans2
Copy link
Collaborator

revans2 commented Jun 9, 2020

Describe the bug
Reading parquet timetamps prior to 1590 that were written with
'spark.sql.legacy.parquet.datetimeRebaseModeInWrite': 'CORRECTED' are off

Steps/Code to reproduce bug
An integration test is being added for this.
@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify SQL part of the SQL/Dataframe plugin labels Jun 9, 2020
@revans2
Copy link
Collaborator Author

revans2 commented Jul 27, 2020

As of #435 reading legacy timestamps prior to 1900 is an error. Corrected timestamps are not showing issues from 1900 to 1590, but we should retest everything.

@sameerz sameerz added P2 Not required for release and removed ? - Needs Triage Need team to review and classify labels Aug 18, 2020
@jlowe jlowe mentioned this issue Oct 28, 2020
Closed
@jlowe
Copy link
Member

jlowe commented Aug 11, 2021 •
edited
Loading

It looks like this works on Spark 3.1+. Probably not high priority to allow it if nobody is asking for it.

tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Merge pull request NVIDIA#132 from NVIDIA/bot-auto-merge-branch-22.04
b5aff75 
[auto-merge] bot-auto-merge-branch-22.04 to branch-22.06 [skip ci] [bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working P2 Not required for release SQL part of the SQL/Dataframe plugin
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jlowe @revans2 @sameerz