Mutable stores, appending and deleting references.

[betolink]

Data producers are often in the need of virtualizing living datasets, these datasets are modified by 2 operations: append and delete. I'd like to understand the current state of things regarding mutable stores with virtualizarr and the broader ecosystem. Let's assume we virtualize a collection with 100k items and then Id like to understand what happens if we try to append and/or delete items from it, and what happens when we try to access a store that has mutated.

Virtualizing

More complicated examples with mixed native chunks and virtual references are excluded for now

vds = open_virtual_mfdataset(inputs, **opts)

vds.virtualize.to_icechunk()
vds.virtualize.to_kerchunk() # parquet  

virtualizarr wrote the chunk manifests to disk and if I'm not mistaken it embeds the coordinate dimension values (at least for kerchunk). This store can be opened by compatible clients -most of them in the Python ecosystem-. Icechunk has support in Javascript and Rust.

Appending

Requirements:

• identical schema (all variables should be the same and present along the concatenated dimension)
• identical chunking/encoding (variable length/encoding is coming)

Opening a virtual store from disk as a virtual store in virtualizarr seems to be limited by a couple factors, for Icechunk I don't think there is a parser that will read an icechunk store and create the pure ManifestStore representation, maybe this is not a big issue since Icechunk can be opened directly by xarray. For Kerchunk, the limitation lies in reading those coordinates that were serialized as native chunks, they are not references. This has workarounds and seems like a permanent fix is coming soon.

Kerchunk

kerchunk_vds = open_virtual_dataset(
    uri.parquet,
    registry=registry, 
    parser=KerchunkParquetParser(skip_variables=materialized_dims)
)

new_vds = open_virtual_dataset(
    new_items,
    registry=registry,
    parser=Parser() # any of the supported parsers
)

updated_vds = xr.concat([new_vds, kerchunk_vds], concat_dim="time")
# we serialize the updated vds back to the original location
updated_vds.to_kerchunk(uri.parquet)

Notice that we are manually concatenating the virtual datasets, perhaps we need a vds.virtualize.to_kerchunk(uri.parquet, append_dim="time") for better user experience even if this is just replacing the parquet file not actually appending (for small collections). For collections with potentially millions of items appending in a columnar format makes sense. Also notice that appending json is not a thing so we should avoid it at all costs.

IceChunk

Appending operations are easier with Icechunk, we open a writable session pointed to the IceChunk store and use the VDS of the new items with that session store.

# the config has the info about the IceChunk store
repo = icechunk.Repository.open_or_create(storage, config, credentials)
session = repo.writable_session("main")

new_vds = open_virtual_dataset(
    new_items,
    registry=registry,
    parser=Parser() # some of the supported parsers
)

new_vds.to_icechunk(session.store, append_dim="time")
session.commit("wrote 100 new items")

Appending with VirtualiZarr is definitely doable and could lower the operational costs for big collections, however we still need to deal with the other operation: deleted granules

Deleting

Once in a while data producers delete files from a collection, sometimes is a quality issue or a reprocessing campaign that affects only a subset of files, maybe is just bad data. In any of these scenarios a file that was processed and is referenced in a virtual store has changed or has been deleted. In a traditional database, updating a row is as simple as DELETE FROM table WHERE unique_id = value;

In the virtual store world is not that simple. Parquet files are immutable by definition and we don't have a simple mechanism to delete rows in a columnar format (as is an anti-pattern). This leaves us with full rewrites. Maybe we can implement a "smart" full re-write by not having to parse the archive from scratch. If a verification method is implemented in the Kerchunk parsers we can very well "prune" the missing references so when we serialize the store we don't include them.

This goes back to what information should be in the chunk manifest, currently we only have byte ranges and codecs information, it would be very useful if we also had metadata like checksum and lastModified

kerchunk_vds = open_virtual_dataset(
    uri.parquet,
    registry=registry, 
    parser=KerchunkParquetParser(skip_variables=materialized_dims, validate=True)
)

This validation flag will test that all references are at least reachable (HTTP HEAD) and will skip those that aren't. It will be slower than just loading the references into memory but I suspect way faster than parsing all from scratch.

Another aspect of this problem is what happens in the clients (focusing on xarray for now) when we load virtual stores with missing chunks.

Operation	Kerchunk	Icechunk
Open dataset: xr.open_zarr() or xr.open_dataset()	✅ succeeds	✅ succeeds
Validate references	❌ none	✅ metadata-based,
Read missing file: any operation that triggers a read()	❌ runtime error	❌ runtime error (earlier + clearer)

On top of implementing a kerchunk validation mechanism, it would be great if xarray/dask had the same fail-safe, if a chunk is not reachable, it be counted as NullValue, FillValue or just skip it, right now a runtime error of this kind is a showstopper unless we manually handle(hack) the exceptions.

More deletion strategies¹

Beyond validation and pruning, a more robust deletion model for virtual stores could include the following ideas:

• Deletion manifests – Instead of rewriting the entire Parquet file, maintain a separate "deletion log" (e.g., a small Parquet or JSONL file) listing the unique identifiers (e.g., granule IDs or time indices) that have been deleted. When opening the virtual dataset, virtualizarr would filter out any references found in the deletion log. This turns deletion into an O(1) append operation, though it adds some complexity to the read path but it will avoid the need for an actual HTTP request like in validate=True.

• Compaction (AKA garbage collection)– Periodically, rewrite the main Parquet file to incorporate deletions and remove the deletion log. This can be done offline, avoiding the cost of a full rewrite on every delete. Perhaps VirtualiZarr could implement a high-level method like vds.virtualize.compact() for Kerchunk, it would rewrite the paruqet and delete the deletion manifest; for IceChunk, it could create a new versioned commit that excludes the specified granules. It's just UX as a user can do it manually but this would be simpler.

Implementing any of these approaches would improve the usability of virtual stores for living datasets, especially when deletions are frequent. The current "full rewrite or validation + prune" model is acceptable but it may not scale well for collections with millions of granules.

Interoperability

Finally a note on interoperability, this is orthogonal to mutable collections. It would be useful for data producers to be able to open any chunk manifest format (icechunk and kerchunk and who knows in the future: MinecraftChunk), mutate it and save it in another format or even better, mix and matched formats. It may be also useful to have an schema at the chunk level that focuses on virtual references. This has been recently discussed at the ESIP cloud computing cluster.

Feel free to comment and edit or correct my misunderstandings, the overall goal to me is to ease user friction with virtual stores specially when they are used via earthaccess.

Footnotes

1. I think some (if not all) of these ideas are already present in Icechunk/ArrayLake. I wonder if in the near future Icechunk would be the only technology we will need to accomplish data virtualization. ↩

[TomNicholas]

This is broadly correct. I'm not really sure what you're asking though.

virtualizarr wrote the chunk manifests to disk and if I'm not mistaken it embeds the coordinate dimension values (at least for kerchunk)

Specifically, any in-memory variables (i.e. loadable_variables, which are the 1D coordinates by default) are written as native chunks, not virtual chunk refs. VirtualiZarr does this for both Kerchunk and Icechunk.

I don't think there is a parser that will read an icechunk store and create the pure ManifestStore representation

There is not, but there could be. See earth-mover/icechunk#104.

In the virtual store world [deleting stale references] is not that simple.

Deleting chunks / arrays can be done. See https://icechunk.io/en/latest/getting-started/howto/#delete-a-group

for IceChunk, it could create a new versioned commit that excludes the specified granules

So you want some kind of utility which, given a list of deleted granules (files), deletes all virtual chunks in the store that reference those files? That sounds doable.

Compaction (AKA garbage collection)

See the icechunk docs on Data Expiration and Garbage Collection ! And the Arraylake docs on Garbage Collection and Expiration, which run the GC jobs for you on a schedule! ("Compaction" is a slightly different thing, but you could also do that with Icechunk).

maintain a separate "deletion log"

Icechunk has a transaction log, which includes deletions.

what information should be in the chunk manifest, currently we only have byte ranges and codecs information, it would be very useful if we also had metadata like checksum and lastModified

Those are already in the Icechunk manifests. They just aren't super accessible from python right now.

---

What is the reason for using Parquet / Kerchunk here? It seems like every feature you want is something that Icechunk was either specifically designed to provide or could potentially provide without too much work.

[betolink]

This is broadly correct. I'm not really sure what you're asking though.

Not a question, this is meant to understand the current state of mutable stores in VirtualiZarr. The follow up topic would be how data producers can interact with both formats with less friction, mainly because Kerchunk is still widely used.

I think a couple ideas that can improve the way virtualizarr uses Kerchunk are within reach, e.g. validating the references when we open a store, and adding append_dim toto_kerchunk() would you be open to PRs implementing these?

Agreed on IceChunk, (see the footnote).

[TomNicholas]

e.g. validating the references when we open a store

Not quite sure what you mean. Opening what store?

adding append_dim to to_kerchunk()

That seems fine.

would you be open to PRs implementing these?

In general yes, and we'll keep support for Kerchunk, but I'm trying to understand why you even want to write to Kerchunk at all at this point.

Agreed on IceChunk, (see the footnote).

I wonder if in the near future Icechunk would be the only technology we will need to accomplish data virtualization.

I don't see any technical reason to use Kerchunk at this point, and a large number of serious reasons not to.

[betolink]

Not quite sure what you mean. Opening what store?

Say I created mystore.parquet with 100k references, since its creation ~100 of these were deleted. I don't want to recreate the store from scratch, I could just verify which of the references are still reachable (in the absence of a transaction manifest like in Icechunk)

filtered_vds = open_virtual_dataset(
    mystore.parquet,
    parser=KerchunkParquetParser(skip_variables=materialized_dims, validate=True)
)
unfiltered_vds =  open_virtual_dataset(
    mystore.parquet,
    parser=KerchunkParquetParser(skip_variables=materialized_dims, validate=False)
)

The validate kwarg will result in HTTP HEAD requests, this way we can persist the store filtering out 404 references. It's simple.

I'm trying to understand why you even want to write to Kerchunk at all at this point.

Because there are a few stores already in Kerchunk.

[betolink]

Also, I'm probably misusing the term validate, this is more like a reachable=True

[sharkinsspatial]

@betolink A few points. I don't think we've written an issue for it yet but we will likely be undertaking work to build an Icechunk parser for Virtualizarr during some upcoming sprints. The goal for this is as you described to increase reference portability and also inform how we might consider building a reference serialization specification for native Zarr as well.

With respect to appending and deleting, Parquet datasets are not a transactional store. Most organizations using Parquet or Zarr in operational systems require some type of transactional layer over storage to address the issues you are describing. These requirements were the driving principles in the development of Iceberg and Icechunk.

As @TomNicholas mentioned, there is no compelling reason to choose Kerchunk/Parquet as a serialization target for virtual references any longer and a host of issues as reasons not to.

[betolink]

These requirements were the driving principles in the development of Iceberg and Icechunk

I'm aware, I'm not trying to implement all the transactional features of Icechunk. I don't think having these 2 features (filter deleted chunks, appending chunks) would be bad for cases where Kerchunk is already being used.

[sharkinsspatial]

As a corollary, I've done work on a few large scale pipelines for managing operational data in Parquet. We investigated options for managing our own in-situ row group overwriting logic (as well as dynamic repartitioning) but using an established technology like Iceberg that has wide community support was an easy choice over building bespoke logic to do the same thing. At this point, I think our development efforts might be better spent assisting users who have existing Kerchunk parquet archives with migrating to Icechunk as a serialization target.