Skip to content

Modflow6Simulation.from_file() or even Package instantiation could always eagerly materialize all non-spatial data #1689

@Huite

Description

@Huite

This has some relation to #1686

My finding there is that .item() doesn't work on dask-backed scalar values. This looks like an issue/inconsistency with the xarray zarr backend (in comparison with netCDF). The workaround is to simply force all non-spatial variables into memory.

However, this workaround might have general benefits: forcing a compute on these small (non-spatial) variables will decrease IO overhead in general, such that it may be worthwhile to always call a .load() or .compute(). Alternatively, it has no downsides (apart from the single loop over the dataset variables at instantiation).

An argument could be made that e.g. the contents of the DIS package should always be loaded into RAM as well.

We could even take this further: guarantee that some arrays are materialized (i.e. numpy backed instead of dask backed) at package instantiation. The resulting memory pressure is small, but it might increase performance predictability.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    📯 New

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions