Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Overturemap parquet files #849

Open
bchapuis opened this issue Apr 16, 2024 · 2 comments
Open

Add support for Overturemap parquet files #849

bchapuis opened this issue Apr 16, 2024 · 2 comments

Comments

@bchapuis
Copy link
Member

https://github.com/OvertureMaps/data

@bchapuis
Copy link
Member Author

bchapuis commented May 1, 2024

@sebr72 as discussed, I'm not really satisfied with my current experiment in the overturemap branch. The geoparquet format contains semi structured data which require some changes in the DataTable abstraction. Also, it requires a deep understanding of the geoparquet format.

One avenue (probably the best) could be to use the parser available in sedona (the project is written in scala):
https://sedona.apache.org/latest-snapshot/tutorial/sql/#__tabbed_9_2

Another avenue could be to build upon my throw-away overturemaps branch, but I'm not sure about the effort needed to have something robust.

In both cases, adding parquet or sedona will result in a lot of new dependencies (hadoop, spark).

@bchapuis
Copy link
Member Author

bchapuis commented May 1, 2024

@sebr72 There may also be a third option which is to rely on parquet support in postgresql. I have no experience with this extention.
https://github.com/adjust/parquet_fdw

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant