QGIS cannot load feature classes from a .gdb with a 64bit ObjectID

[coefficient1900]

What is the bug or the crash?

When trying to load a feature class from a .gdb created with ArcGIS Pro > 3.2 it is possbile that this operation fails.
The crucial point seems to be that ESRI introduced new data types like 64bit field types for example for ObjectIDs.

Executing such an operation with a "classic" .gdb there is no problem.

First a knew .gdb and a feature class is created in ArcGISPro:

and then dropped in QGIS, everything is fine:

Steps to reproduce the issue

But when I used the tool "Migrate Object ID To 64 Bit" in ArcGISPro:

the capability of QGIS to read this feature-class is gone:

Versions

<style type="text/css"> p, li { white-space: pre-wrap; } </style>

QGIS-Version	3.28.15-Firenze	QGIS-Codeversion	252ad49
Qt-Version	5.15.3
Python-Version	3.9.18
GDAL-Version	3.8.3
PROJ-Version	9.3.1
EPSG-Registraturdatenbankversion	v10.098 (2023-11-24)
GEOS-Version	3.12.1-CAPI-1.18.1
SQLite-Version	3.41.1
PDAL-Version	2.6.0
PostgreSQL-Client-Version	unknown
SpatiaLite-Version	5.1.0
QWT-Version	6.1.6
QScintilla2-Version	2.13.4
BS-Version	Windows 10 Version 2009
Aktive Python-Erweiterungen
alkisplugin	2.0.54
HCMGIS	22.9.9
processing_saga_nextgen	1.0.0
qfieldsync	v4.2.0
wfsclient	0.9.11
db_manager	0.1.20
grassprovider	2.12.99
MetaSearch	0.3.6
processing	2.12.99
sagaprovider	2.12.99

QGIS-Version 3.28.15-Firenze QGIS-Codeversion [252ad49](https://github.com/qgis/QGIS/commit/252ad49ddc) Qt-Version 5.15.3 Python-Version 3.9.18 GDAL-Version 3.8.3 PROJ-Version 9.3.1 EPSG-Registraturdatenbankversion v10.098 (2023-11-24) GEOS-Version 3.12.1-CAPI-1.18.1 SQLite-Version 3.41.1 PDAL-Version 2.6.0 PostgreSQL-Client-Version unknown SpatiaLite-Version 5.1.0 QWT-Version 6.1.6 QScintilla2-Version 2.13.4 BS-Version Windows 10 Version 2009

Aktive Python-Erweiterungen
alkisplugin
2.0.54
HCMGIS
22.9.9
processing_saga_nextgen
1.0.0
qfieldsync
v4.2.0
wfsclient
0.9.11
db_manager
0.1.20
grassprovider
2.12.99
MetaSearch
0.3.6
processing
2.12.99
sagaprovider
2.12.99

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

Additional context

I know 3.28.15 is outdated, but it does not work with 3.36.x either.

[rouault]

This should normally be fixed when you'll have a QGIS build against GDAL 3.9 which adds support for those new data types per OSGeo/gdal#8866

However I'm not totally sure about 64-bit OBJECTIDs. Would you mind attaching to this ticket a zipped .gdb with a 64-bit OBJECTID so I can check?

[coefficient1900]

In this .gdb the feature from the pictures above is included. With 64bit ObjectID of course.

MyProject1.gdb.zip

[rouault]

@coefficient1900 Thanks for the example. Which shows that 64-bit objectIds will need more work on the GDAL side. The file format version number in that example has changed, and there are other backward incompatible changes in header fields that prevent GDAL from correctly reading those files. I would need more examples to try understanding the exact changes in the file format. Typically (all below created with 64-bit objectIds)

• an example with 0 features (empty table, with no feature ever created in it)
• an example with 2 or more features
• an example with 2 or more features initially created, and then all deleted
• an example where there have been several tens of thousand of features created with a hole of 2048 or more features of consecutive objectid (not totally sure this can be done on creation using ESRI software. it may require creating them, and deleting them afterwards). To make this last example easier, I'm including such a regular FileGDB that you hopefully may try to migrate to 64 bit: with_holes.gdb.zip

It has been generated with the below GDAL Python script:

from osgeo import gdal, ogr, osr

ds = ogr.GetDriverByName("OpenFileGDB").CreateDataSource("with_holes.gdb")
srs = osr.SpatialReference()
srs.ImportFromEPSG(4326)
lyr = ds.CreateLayer("with_holes", geom_type = ogr.wkbPoint, srs = srs)
for i in range(10):
    f = ogr.Feature(lyr.GetLayerDefn())
    f.SetGeometry(ogr.CreateGeometryFromWkt("POINT(%f %f)" % (i / 10000, i / 10000)))
    lyr.CreateFeature(f)
for i in range(10):
    f = ogr.Feature(lyr.GetLayerDefn())
    objectid = 20 * 1024 + i + 1
    f.SetFID(objectid)
    f.SetGeometry(ogr.CreateGeometryFromWkt("POINT(%f %f)" % (objectid / 10000, objectid / 10000)))
    lyr.CreateFeature(f)

I'm also including a version with OBJECTIDs from 1 to 21504 (points from (0,0) to (2.1503,2.1503)). It might be useful to try to convert it to 64bit objectids, and then deleting 2048 or more consecutive OBJECTIDs. no_holes.gdb.zip

from osgeo import gdal, ogr, osr

ds = ogr.GetDriverByName("OpenFileGDB").CreateDataSource("no_holes.gdb")
srs = osr.SpatialReference()
srs.ImportFromEPSG(4326)
lyr = ds.CreateLayer("no_holes", geom_type = ogr.wkbPoint, srs = srs)
for i in range(21 * 1024):
    f = ogr.Feature(lyr.GetLayerDefn())
    f.SetGeometry(ogr.CreateGeometryFromWkt("POINT(%f %f)" % (i / 10000, i / 10000)))
    lyr.CreateFeature(f)

[coefficient1900]

Thanks a lot @rouault

• here is the empty one (created a new 32bit ObjectID feature class and converted to 64bit):
  empty.gdb.zip

• a .gdb with a feature class containing 3 features (64 bit ObjectIDs):
  3features.gdb.zip

• deleted this three features:
  3features_thendeleted.gdb.zip

• the "with_holes.gdb" converted to 64bit:
  with_holes.gdb.zip

• and the "no-holes.gdb" first converted to 64bit ObjectIDs an the deleted features 1000-5000:
  no_holes.gdb.zip

I hope this helps.

[rouault]

@coefficient1900 This does help a lot. The with_holes is hard to decypher however. I've prepared a number of variations of input files around it to hopefully manage to make progress. In the attached with_holes_234567.zip, there are with_holes_2.gdb to with_holes_7.gdb. Would you mind converting them to 64 bit and attach back the results?
Also, do you know how to create a 64-bit FileGDB with a feature without object id is greater than 32bit? Let's say create a feature whose feature id is 5 billion ? That would be helpful too to have access to such file

[coefficient1900]

Here are the _2 to _7 .gdbs converted to 64bit IDs:
with_holes_234567_64bit.zip

For the second case, should the .gdb contain a feature-class that include exclusively features with IDs outside the 32bit range? Did I get it right?

[rouault]

should the .gdb contain a feature-class that include exclusively features with IDs outside the 32bit range? Did I get it right?

yes that was my idea. That said both cases could potentially be useful. One with an ID > 32 bit, and another one with one ID < 32 bit and one > 32 bit.

[rouault]

I'm also attaching with_holes_8.gdb.zip that would be worth converting to 64 bit. It contains several tables, as it might be easier to convert on your side. Tables with_holes_8_e and with_holes_8_f contain an object of id larger than 1 billion. I hope the conversion tool doesn't choke on this!

[coefficient1900]

This is the coverted "with_holes8.gdb":
with_holes_8.gdb.zip

Do you still need the .gdb with feature IDs > 32bit? It is a little bit challenging to create such a feature class.

[rouault]

Do you still need the .gdb with feature IDs > 32bit?

At that point, no I don't think that would help me better understanding the format of sparse tables. I understand it partly, but not sufficiently. Thanks for all the converted datasets!

[rouault]

(Partial) fix in OSGeo/gdal#9980