Why not GeoParquet?

[Daniel WIlson (dwilson1988)]

This goes for all categories, not just Buildings.

Parquet is a great format, but with just a small amount of extra metadata, this data could be made much more usable by existing geospatial tools. The OGC has create the GeoParquet standard which is under active development, but under a 1.0.0-beta1 release. Many public datasets, including the Microsoft Building Footprints in the Microsoft Planetary computer have already been released. Tools like Apache Sedona and Esri GeoAnalytics Engine can work with GeoParquet natively.

https://github.com/opengeospatial/geoparquet
https://geoparquet.org/

Primarily GeoParquet is just additional metadata in each partition's header and geometries are encoded as WKB already so it seems like it would be a very small change that would make these datasets infinitely more usable.

Any thoughts on this?

[Jia Yu (jiayuasu)]

Daniel WIlson (@dwilson1988) I totally agree. I believe this is because the Overture developers thought they do not have scalable tools that can generate a huge GeoParquet file. However, Apache Sedona is the first scalable geospatial data processing engine that can read / write GeoParquet file (GeoParquet spec 1.0.0). See [1] [2]

I propose that Wherobots, the company founded by the original creators of Apache Sedona, lead the effort or help the team generate GeoParquet version of this data, using Apache Sedona.

[1] https://sedona.apache.org/
[2] https://sedona.apache.org/1.4.1/tutorial/sql/#save-geoparquet

[Daniel WIlson (dwilson1988)]

Jia Yu (@jiayuasu) Well, I don't even know that they would need to change their processes all that much since the GeoParquet spec is fairly simple (in this case, the only needed change would be to write the GeoParquet metadata to each partition). Apache Sedona would be an option for sure, but not entirely necessary if it would require a major process change. Assuming Spark is already in the mix though, writing the output via Sedona shouldn't be too heavy of a lift.

Of course, I say that not knowing what the current processes in place are though. We make heavy use of GeoParquet at SeerAI and are happy to lend assistance if needed.

[Jia Yu (jiayuasu)]

Daniel WIlson (@dwilson1988) Hi all, today Wherobots released the spatially indexed GeoParquet format of Overture Maps data using Apache Sedona (sedona.apache.org). With this dataset, a spatial range query on the 120GB Building dataset takes around 3 minute to finish on a single-node 4GB Apache Sedona docker container.

The data is free and public on AWS S3. For more details, please see my blog post on Medium: medium.com/@dr.jiayu/harnessing-overture-maps-data-apache-sedonas-journey-from-parquet-to-geoparquet-d99f7767a499