Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SEDONA-94] GeoParquet Reader Writer #652

Merged
merged 20 commits into from
Aug 22, 2022
Merged

Conversation

ashar236
Copy link
Contributor

@ashar236 ashar236 commented Jul 14, 2022

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

GeoParquet reader and writer support for Sedona is implemented in this PR.

  • This implementation is based on a fork of Spark 3.1 Parquet reader/writer because there was a heavy internal Parquet API change from Spark 3.2 to Spark 3.3.
  • This fork is also modified to support Spark 3.0 - 3.3. But the users should expect its non-geospatial behavior identical to Spark 3.0/3.1/3.2.
  • VectorizedReader is removed in this reader for compatibility issues since Geometry type is not atomic type anyway.
  • We have a neat version that nicely supports Spark 3.3 only. It will be released when Sedona no longer needs to support Spark 3.0 - 3.2.
  • Geometry filter on BBox is not implemented. It will be introduced in a follow-up PR.
  • Spark 2.4 is not supported. We have no plan to support it since Sedona on Spark 2.4 support will be completely dropped in the next Sedona major release.

Additional notes:

  1. This PR will be merged after we release Sedona 1.2.1 and drop Spark 2.4 support
  2. Before merging, we need to remove 'fail-fast: false' from the Scala/Java CI.

How was this patch tested?

Unit tests have been added.

Did this PR include necessary documentation updates?

  • Yes, I have updated the documentation update.

@jiayuasu jiayuasu changed the title GeoParquet Reader Writer [SEDONA-94] GeoParquet Reader Writer Jul 15, 2022
@jiayuasu jiayuasu added this to the sedona-1.2.1 milestone Jul 15, 2022
@ashar236 ashar236 force-pushed the master branch 3 times, most recently from d02660f to 1b7da2a Compare July 25, 2022 02:42
@jiayuasu jiayuasu modified the milestones: sedona-1.2.1, sedona-1.3.0 Jul 25, 2022
@jiayuasu jiayuasu self-assigned this Aug 4, 2022
@jiayuasu jiayuasu merged commit 547ff40 into apache:master Aug 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants