Skip to content

Persist in-memory slices before appending #11

@forman

Description

@forman

It should be possible to persist in-memory slice datasets to a temporarily Zarr before appending them to the target dataset.

The background for this requirement is that often in-memory datasets contain data variables whose dask arrays represent complex and/or expensive processing graphs. The target dataset's chunking may cause that these graphs are computed multiple times if the source chunking overlaps multiple target chunks, potentially causing large resource overheads while recomputing same source chunks. The overheads comprise same source files being reloaded multiple times and high memory and CPU loads due to recomputation of same regions.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions