Handle NULL values in features correctly

[evetion]

If I create the following geopackage (shown with ogrinfo)

Geometry Column = geom
date: DateTime (0.0)
test: Integer(Boolean) (0.0)
name: String (255.0)
number: Integer (0.0)
OGRFeature(null):1
  date (DateTime) = (null)         <-- note these nulls
  test (Integer(Boolean)) = 0
  name (String) = test
  number (Integer) = 1
  POINT (0.029535864978903 0.578059071729958)

OGRFeature(null):2
  date (DateTime) = 2021/03/12 00:00:00
  test (Integer(Boolean)) = 1
  name (String) = (null)         <-- note these nulls
  number (Integer) = (null)    <-- note these nulls
  POINT (-0.355836849507736 0.20393811533052)

and apply getfield to it, ArchGDAL fails with:

ERROR: Failed to fetch datetime at index 0
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] asdatetime(::ArchGDAL.Feature, ::Int32) at /Users/evetion/.julia/packages/ArchGDAL/j9NPL/src/ogr/feature.jl:318
 [3] getfield(::ArchGDAL.Feature, ::Int32) at /Users/evetion/.julia/packages/ArchGDAL/j9NPL/src/ogr/feature.jl:378
 [4] getfield(::ArchGDAL.Feature, ::Symbol) at /Users/evetion/.julia/packages/ArchGDAL/j9NPL/src/ogr/feature.jl:384

It seems that isfieldset doesn't trigger correctly for NULL DateTime fields. If we actually fill it, we get a result, but ArchGDAL uses getdefault, which returns actual data values (name and number should both be null). These should be either nothing or missing.

(date = Dates.DateTime("2021-03-12T00:00:00"), test = 1, name = "", number = 0, geom = Geometry: POINT (-0.355836849507736 0.20393811533052))

On a last note, we don't parse the width of integers, so we don't parse the test=1 as the bool it is.

Ping to @vernimmen who ran into this when opening a GeoPackage with NULLs via GeoDataFrames.jl.

[yeesian]

Sorry for being late to respond here: is it possible to provide a small working example here?

[evetion]

Sure, I've uploaded a small test file.

using ArchGDAL

fn = download("https://github.com/evetion/GeoDataFrames.jl/raw/null-test-data/test/null.gpkg")
ds = ArchGDAL.read(fn)
layer = ArchGDAL.getlayer(ds, 0)

Purely the show method will already crash there:

Layer: null
Geometry 0 (geom): [wkbPoint], POINT (0.02953586497...), ...
Error showing value of type ArchGDAL.IFeatureLayer:
ERROR: Failed to fetch datetime at index 0

[yeesian]

In #177 (comment), I'm also curious about "On a last note, we don't parse the width of integers, so we don't parse the test=1 as the bool it is."

Geometry Column = geom
date: DateTime (0.0)
test: Integer(Boolean) (0.0)         <-- note the Boolean (rather than the width of 0)
name: String (255.0)
number: Integer (0.0)
OGRFeature(null):1
  date (DateTime) = (null)
  test (Integer(Boolean)) = 0
  name (String) = test
  number (Integer) = 1
  POINT (0.029535864978903 0.578059071729958)

...

I suspect I should be going by the fieldsubtype (per https://github.com/OSGeo/gdal/blob/fec15b146f8a750c23c5e765cac12ed5fc9c2b85/gdal/apps/ogrinfo.cpp#L396). What do you think?

[yeesian]

I dug into the code for ogrfeature.cpp (which is very instructive), and realize that it checks for nullity of the fields.

We should wrap the corresponding method from GDAL.jl and use it to check for nullity in ArchGDAL.jl

[evetion]

I suspect I should be going by the fieldsubtype (per https://github.com/OSGeo/gdal/blob/fec15b146f8a750c23c5e765cac12ed5fc9c2b85/gdal/apps/ogrinfo.cpp#L396). What do you think?

Agreed. It also means our current method of a single convert (https://github.com/yeesian/ArchGDAL.jl/blob/master/src/tables.jl#L7) isn't good enough, as we need two converts and some extra logic.

[yeesian]

Also linking to Toblerity/Fiona#460 (comment) here.