Add test data covering different native (geoarrow-based) encodings

[jorisvandenbossche]

Addressing part of #199 (small example data)

And similar as #134, but focusing here on the new geoarrow-based encodings (I should also update that older PR)

This adds some tiny example data for the different geometry types:

• for each I am saving the data as WKB and the native encoding (with some default metadata), and additionally also as plain text in case this would be useful for another tool as the source of truth to compare with (eg to check reading the parquet file was correct)
• this scripts only relies on pyarrow for creating the data and writing the parquet files (and shapely to create the WKB data), so nothing geoarrow-specific (eg no arrow extension types, no interleaved coordinates, so just what is in the geoparquet spec)
• for now I only added 2D data, not yet variants with z coordinates. And there are probably also more corner cases that could be added (eg a MultiPolygon containing an empty Polygon, and things like that?)

In #134, I also saved the metadata in a separate file as well, which allows easier validation / review that this is correct. I could do that here as well, in case that is useful.
(I should still add validation of the generated files with the json schema)

Does this look useful in this form? Any suggestions to approach it differently?

[paleolimbot]

Just a few notes! (I haven't checked the actual files yet)

[paleolimbot]

These look like .csv files to me...should they have a .csv extension?

[paleolimbot]

Should there also be a version of these that contain NULLs or a version that contains Z values?

[jorisvandenbossche]

Added some null values!

[kylebarron]

Maybe it would be less error-prone to construct these geometries from the WKT? Then we don't have to manually check that the WKT is equivalent to here.

Or maybe it would be better to use GeoJSON and pyarrow's JSON reader to infer geometry columns? Though then you'd need to convert from interleaved to separated coords, which is a bit involved for this test data here 😕

[jorisvandenbossche]

Yeah, I actually tried to automate writing this hard-coded data a bit, by parsing the WKT with shapely and converting to GeoJSON. The main problem is that then I need to convert the inner lists to tuples (as json is only lists), to have the construction as a separated struct type work with pyarrow (a tuple element can be converted to a struct, but a list element not). And that gives some convoluted logic ..

[jorisvandenbossche]

cc @rouault would this (have been) useful for testing in GDAL? (xref #189 (comment))
(I have been checking the files generated here with GDAL main, and they read fine)

[rouault]

would this (have been) useful for testing in GDAL?

yes, thanks

[rouault]

I've detected that there's a typo on geometry_type which should be geometry_types . Generated files should be regenerated

[jorisvandenbossche]

I've detected that there's a typo on geometry_type which should be geometry_types . Generated files should be regenerated

Ah yes, that's because I started from #134 which I wrote before we made that name change .. Thanks for the catch!

[cholmes]

I'm going to go ahead and merge this in, as it's been open and approved for awhile, and seems like a good edition. Any more work can be done as PR's to main.

[tschaub]

@jorisvandenbossche - The scripts directory has a generate_example.py script and a readme that describes how to install the dependencies for it. Could we move this new generate_test_data.py script to that same directory and update the readme if needed to describe how to install the dependencies for it (or update the pyproject.toml or the poetry.lock as needed)?

In addition, it would be nice to pull the version identifier out of the format-specs/schema.json (as is done in the generate_example.py script) to limit the number of places we have to update version numbers for a release.

I'll leave the same comment on #232 (as I think it would be nice to consolidate the examples/test data into one thing that is easier to maintain).

[jorisvandenbossche]

In addition, it would be nice to pull the version identifier out of the format-specs/schema.json (as is done in the generate_example.py script) to limit the number of places we have to update version numbers for a release.

Yes, I realized last week with the release I suboptimally hardcoded the number here in this PR.

Will try to look at it tomorrow.