-
Notifications
You must be signed in to change notification settings - Fork 5
Description
Related to #1686: it may be worthwhile to detect if the model to split is unchunked. As mentioned in the PR, this will lead to massive IO waste when writing the parallel partitions. I'm not sure on how best to word this.
There is also the distinction between structured and unstsructured inputs. A structured model can be trivially chunked along x and y. To solve the partition-related overhead, it would be sufficient to set the relevant encoding in a dump method.
This isn't the case for the unstructured case, where the internal topology ordering is essential. We could define a reorder method on the model, but it feels a bit messy to me. For structured data, I guess it would be a no-op. Perhaps a pragmatic solution is to allow it to be called internally (as an xugrid method) in the dump method: if the data is unstructured and not already ordered, then it ordered automatically before being written to Zarr or netCDF chunks.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status