Skip to content

Support NetCDF input tables as Arrow alternative - #2542

Merged
visr merged 3 commits into
mainfrom
netcdf-input
Sep 9, 2025
Merged

visr merged 3 commits into
mainfrom
netcdf-input

Conversation

@visr

@visr visr commented Aug 25, 2025 •

Copy link
Copy Markdown
Member

Fixes most of #2434, not all because only a subset of the tables are supported.

This essentially gives the same treatment we currently have for Arrow input files, but for NetCDF files. We choose between the file format based on whether the path has a .arrow or .nc extension.

So in Python if you do:

model.basin.state.set_filepath(Path("basin-state.nc"))

(Note you need this if you read the model from disk.)

Ribasim Python will convert the df.to_xarray().to_netcdf(), and if you read it back in with Python ds.to_dataframe().
In the TOML you get:

[basin]
state = "basin-state.nc"

The core then also reads based on the file extension. To keep things simple, when reading data into the core we convert the N dimensional arrays in the NetCDF to tables, so the rest of the initialization doesn't need to handle both data structures.

We should also think about what to do for irregular tables that don't fit in a structured array. Perhaps just throw an error?

@visr
visr force-pushed the netcdf-input branch 2 times, most recently from a9944d6 to 84d3152 Compare September 8, 2025 12:24
@visr
visr marked this pull request as ready for review September 8, 2025 12:28
@visr

visr commented Sep 8, 2025

Copy link
Copy Markdown
Member Author

This doesn't support all tables listen in #2434 yet, but those that are needed soon are there. It includes tests, but no docs. Essentially this is an experimental feature for now.

@visr
visr requested a review from evetion September 8, 2025 12:30
@visr visr mentioned this pull request Sep 8, 2025
4 tasks done
visr added a commit that referenced this pull request Sep 9, 2025
This is optimistically targeting a release date of tomorrow, and already
includes these assuming the will be in the release.

- [ ] #2564
- [ ] #2562
- [ ] #2544
- [ ] #2542

@evetion evetion left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could use some documentation. I know it's supposed to be experimental (even though you're ignoring the current flags for that), but that should also be documented somewhere.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these be optional deps?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm yeah I considered it, but I believe this is easier. Otherwise we have to document that users can only do certain things if they have certain optional deps installed, which can be confusing. Technically pyarrow could also be an optional dep, which may make the default install smaller, but not more usable.

Comment thread python/ribasim/tests/test_io.py Resolved
Comment thread python/ribasim/ribasim/input_base.py Resolved
Comment thread python/ribasim/ribasim/input_base.py Resolved
Comment thread python/ribasim/ribasim/input_base.py Resolved
Comment thread core/test/io_test.jl
# copy state results to input
write_results(model)
state_path = Ribasim.results_path(config, Ribasim.RESULTS_FILENAME.basin_state)
cp(state_path, Ribasim.input_path(config, "warm_state.nc"))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be nice to have a model utility for that in our Python codebase?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah possibly, although I mainly see Delft-FEWS doing this. Don't think it is worth doing it now.

Comment thread core/test/io_test.jl Resolved
@visr

visr commented Sep 9, 2025

Copy link
Copy Markdown
Member Author

This could use some documentation.

I agree, though this is basically an existing issue #2039, that now is valid for both Arrow and NetCDF. The way it works is so buggy that I'd rather fix it first before documenting it.

@visr
visr merged commit 1e71aa6 into main Sep 9, 2025
@visr
visr deleted the netcdf-input branch September 9, 2025 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants