-
Notifications
You must be signed in to change notification settings - Fork 1
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
It should be possible to persist in-memory slice datasets to a temporarily Zarr before appending them to the target dataset.
The background for this requirement is that often in-memory datasets contain data variables whose dask arrays represent complex and/or expensive processing graphs. The target dataset's chunking may cause that these graphs are computed multiple times if the source chunking overlaps multiple target chunks, potentially causing large resource overheads while recomputing same source chunks. The overheads comprise same source files being reloaded multiple times and high memory and CPU loads due to recomputation of same regions.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request