Skip to content

[SEDONA-94] GeoParquet Reader Writer#652

Merged
jiayuasu merged 20 commits into
apache:masterfrom
ashar236:master
Aug 22, 2022
Merged

[SEDONA-94] GeoParquet Reader Writer#652
jiayuasu merged 20 commits into
apache:masterfrom
ashar236:master

Conversation

@ashar236
Copy link
Copy Markdown
Contributor

@ashar236 ashar236 commented Jul 14, 2022

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

GeoParquet reader and writer support for Sedona is implemented in this PR.

  • This implementation is based on a fork of Spark 3.1 Parquet reader/writer because there was a heavy internal Parquet API change from Spark 3.2 to Spark 3.3.
  • This fork is also modified to support Spark 3.0 - 3.3. But the users should expect its non-geospatial behavior identical to Spark 3.0/3.1/3.2.
  • VectorizedReader is removed in this reader for compatibility issues since Geometry type is not atomic type anyway.
  • We have a neat version that nicely supports Spark 3.3 only. It will be released when Sedona no longer needs to support Spark 3.0 - 3.2.
  • Geometry filter on BBox is not implemented. It will be introduced in a follow-up PR.
  • Spark 2.4 is not supported. We have no plan to support it since Sedona on Spark 2.4 support will be completely dropped in the next Sedona major release.

Additional notes:

  1. This PR will be merged after we release Sedona 1.2.1 and drop Spark 2.4 support
  2. Before merging, we need to remove 'fail-fast: false' from the Scala/Java CI.

How was this patch tested?

Unit tests have been added.

Did this PR include necessary documentation updates?

  • Yes, I have updated the documentation update.

@jiayuasu jiayuasu changed the title GeoParquet Reader Writer [SEDONA-94] GeoParquet Reader Writer Jul 15, 2022
@jiayuasu jiayuasu added this to the sedona-1.2.1 milestone Jul 15, 2022
@ashar236 ashar236 force-pushed the master branch 3 times, most recently from d02660f to 1b7da2a Compare July 25, 2022 02:42
@jiayuasu jiayuasu modified the milestones: sedona-1.2.1, sedona-1.3.0 Jul 25, 2022
@jiayuasu jiayuasu self-assigned this Aug 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants