Handling of missing data in tables

[SouthEndMusic]

As mentioned here.

[visr]

Thanks for creating this issue. This is partially supported right now, e.g. see

Ribasim/core/src/utils.jl

Line 198 in ed4fbef

if !isnan(val)

, where we ignore NaN values when updating the forcing.

Both GeoPackage and Arrow have a general concept of missing data, separate from NaN which only exists for floats, and therefore is not an option for optional string fields like control_state. In julia NULL is normally encoded as missing, so a float that can be missing is Union{Float64, Missing}. So probably we should use that for everything that is allowed to be missing, unless there are other downsides compared to using NaN.

[SouthEndMusic]

@visr I guess this issue is solved now?

[visr]

I don't think so. It's gotten better with your implementation of defaults in the julia core, but we still need testing and a clear story on how we handle optional data across ribasim-python and the core.

[visr]

I think this can be closed now. Occasionally we adjust this a bit like in #1493. One thing worth adding is that we should always use NULL / missing rather that NaN in out input data.