Repository navigation
Support NetCDF input tables as Arrow alternative - #2542
Conversation
a9944d6 to
84d3152
Compare
|
This doesn't support all tables listen in #2434 yet, but those that are needed soon are there. It includes tests, but no docs. Essentially this is an experimental feature for now. |
evetion
left a comment
There was a problem hiding this comment.
This could use some documentation. I know it's supposed to be experimental (even though you're ignoring the current flags for that), but that should also be documented somewhere.
There was a problem hiding this comment.
Hmm yeah I considered it, but I believe this is easier. Otherwise we have to document that users can only do certain things if they have certain optional deps installed, which can be confusing. Technically pyarrow could also be an optional dep, which may make the default install smaller, but not more usable.
| # copy state results to input | ||
| write_results(model) | ||
| state_path = Ribasim.results_path(config, Ribasim.RESULTS_FILENAME.basin_state) | ||
| cp(state_path, Ribasim.input_path(config, "warm_state.nc")) |
There was a problem hiding this comment.
Might be nice to have a model utility for that in our Python codebase?
There was a problem hiding this comment.
Yeah possibly, although I mainly see Delft-FEWS doing this. Don't think it is worth doing it now.
I agree, though this is basically an existing issue #2039, that now is valid for both Arrow and NetCDF. The way it works is so buggy that I'd rather fix it first before documenting it. |
Fixes most of #2434, not all because only a subset of the tables are supported.
This essentially gives the same treatment we currently have for Arrow input files, but for NetCDF files. We choose between the file format based on whether the path has a
.arrowor.ncextension.So in Python if you do:
(Note you need this if you read the model from disk.)
Ribasim Python will convert the
df.to_xarray().to_netcdf(), and if you read it back in with Pythonds.to_dataframe().In the TOML you get:
The core then also reads based on the file extension. To keep things simple, when reading data into the core we convert the N dimensional arrays in the NetCDF to tables, so the rest of the initialization doesn't need to handle both data structures.
We should also think about what to do for irregular tables that don't fit in a structured array. Perhaps just throw an error?