Repository navigation
Replies: 2 comments
|
Geo is hard and the community interested in getting the details right is mostly just non intersecting with the Arrow/Parquet/DataFusion communities. The parquet and parquet-geospatial crates do have some type logic needed to integrate the Parquet geospatial types and I think that has been positive (used, for example, by iceberg-rust to implement their spatial types), but functions, file formats, and specialized operators I think are best left to dedicated communities.
There's ongoing work on integrating extension types in DataFusion as something approaching first class citizens, or at the very least extra special metadata. |
What do you mean "into DataFusion"? Do you mean to be adopted and maintained in an ASF maintained repo? If so I think we would need to see some maintainers step up to say they would help maintain and drive it forward, along with doing an appropriate IP clearance for what is already here |
Uh oh!
There was an error while loading. Please reload this page.
Thanks to @kylebarron's existing great work, I have pushed some efforts to the test framework, some missing functions, performance optimizations, and documentation polish. Although this project is not that mature yet, maybe it's time for us to think about how to get geodatafusion into DataFusion so that our community can have less work to integrate them all.
To start, I have very little understanding of geo things. I just jumped into this because https://github.com/lance-format/lance needs it. At first glance, it seems we have a geo variant for everything inside the community, like GeoArrow, GeoParquet, and this, geodatafusion.
I'm wondering what blocker prevents us from merging them in. What's the next step?
cc @kylebarron, @alamb, @Ant59 and @paleolimbot for the discussion, thank you in advance.
All reactions