GPKG: disable by default multi-threaded ArrowArray interface. Make it…

[rouault]

… opt-in with the OGR_GPKG_NUM_THREADS config option

Cf https://lists.osgeo.org/pipermail/gdal-dev/2024-January/058177.html

[jratike80]

Out of interest, does the "consecutive feature ID numbering" support cases with missing ID numbers when some rows have been deleted from the middle of a table?

[rouault]

does the "consecutive feature ID numbering" support cases with missing ID numbers when some rows have been deleted from the middle of a table?

no, consecutive feature ID numbering means no holes and N features with id = 1, 2, .... N where N is the total number of features.

[coveralls]

coverage: 68.734% (-0.001%) from 68.735%
when pulling c4d5e5d on rouault:OGR_GPKG_NUM_THREADS
into 98c3491 on OSGeo:master.

[jratike80]

no, consecutive feature ID numbering means no holes and N features with id = 1, 2, .... N where N is the total number of features.

Is that tested so that user cannot activate numthreads>1 and ArrowArray interface if that is doomed to fail?
With a valid GeoPackage it should be enough to test that max(fid)=count(*). Or rather use the feature_count in the gpkg_ogr_contents for comparison.

[rouault]

Is that tested so that user cannot activate numthreads>1 and ArrowArray interface if that is doomed to fail?

yes, the optimization checks that max(fid) = count(*) ( actually using the gpkg_ogr_contents special table to get count(*) in a fast way). So it should be safe to set OGR_GPKG_NUM_THREADS unconditionally, in theory. I'm not sure why the user on the mailing list hit that issue. I've never hit myself.

[leofuhrmann]

User from the mailing list here, here's a GeoPackage to hopefully (?) reproduce the problem. The original dataset contains geometries that were removed for the example, leaving only the fid column.

ogr2ogr test.gpkg example.gpkg
throws "ERROR 1: Worker thread task has not expected m_iStartShapeId value"

ogr2ogr test.gpkg example.gpkg -where 1
doesn't produce the error.

example.gpkg.zip

[rouault]

User from the mailing list here, here's a GeoPackage to hopefully (?) reproduce the problem

Thanks. I did reproduce. The issue was related to the number of features vs the batch size. As far as I can see, there should be no corruption and the error message could thus be ignored. fix in #9026