Skip to content

Spark 3.3: [FOLLOWUP] Add TODO for TimeTravel via DataframeReader using timestamp in seconds and CatalogOptions#5160

Closed
singhpk234 wants to merge 2 commits intoapache:masterfrom
singhpk234:fix/workaround-SPARK-39633
Closed

Spark 3.3: [FOLLOWUP] Add TODO for TimeTravel via DataframeReader using timestamp in seconds and CatalogOptions#5160
singhpk234 wants to merge 2 commits intoapache:masterfrom
singhpk234:fix/workaround-SPARK-39633

Conversation

@singhpk234
Copy link
Contributor

@singhpk234 singhpk234 commented Jun 29, 2022

About the changes

  • Document limitation of TT via DataframeReader using timestamp in seconds and CatalogOptions.

Testing Done

cc @rdblue

@singhpk234 singhpk234 changed the title Spark 3.3: [FOLLOWUP] TimeTravel using SPARK SQL and DataFrameReaders Spark 3.3: [FOLLOWUP] Add TODO for TimeTravel via Dataframe Reader using timestamp in seconds and CatalogOptions Jul 1, 2022
@singhpk234 singhpk234 changed the title Spark 3.3: [FOLLOWUP] Add TODO for TimeTravel via Dataframe Reader using timestamp in seconds and CatalogOptions Spark 3.3: [FOLLOWUP] Add TODO for TimeTravel via DataframeReader using timestamp in seconds and CatalogOptions Jul 1, 2022
@rdblue
Copy link
Contributor

rdblue commented Jul 3, 2022

Should we close this since it won't be fixed in Iceberg?

@singhpk234
Copy link
Contributor Author

Sure, I just thought it would nice to document this atleast, so that we don't miss this when we upgrade to 3.3.1, but I think we should be fine either ways !

Thanks @rdblue for the review :) !! I learned about timestamp handling in iceberg / spark from this.

@singhpk234 singhpk234 closed this Jul 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments