ITS_LIVE data virtualization

[betolink]

We just had a good meeting with @sharkinsspatial and @TomNicholas about virtualizing ITS_LIVE data granules (level 2) and prospects for streamlining the creation of our zarr data cubes.

I'll try to summarize some of the topics we discussed.

• ITS_LIVE is producing netcdf granules from all Landsat and Sentinel satellites at a rate of ~20k new granules per day. (total so far we have ~42 million granules)
• These granules are currently used to produce UTM-aligned zarr data cubes that get rechunked in the time dimension (20k items x 32x32) a.k.a. data churros. We do this async like twice a year.

We are interested in streamlining the data cube generation to bring it as closer to real time as possible. Tom mentioned that this is a good case of that hybrid chunking where new data gets appended as is and historical data gets rechunked. We are also interested in using data virtualization to provide a better access to the raw data (and maybe it can help with the data ingestion pipeline as well)

When we first started exploring the idea of virtualizing our granules we ran into a series of blockers due the nature of the data, e.g. https://github.com/nasa-jpl/its_live_production/blob/sorted_granules_datacube_june2026/src/dev_notebooks/issues/virtualizarr/cube_virtualizarr_bugReport_883.ipynb

We talked about how the potential solution to this was going to be driven by the use of schema validation, similar to the methods used to virtualize GOES 16 and how this is not necessarily a core virtualizarr feature. Some points that I got from the meeting, feel free to comment on them.

• Seems like Virtualizarr won't require refactoring of the ChunkManifest to accomodate ragged arrays? also see Sean's notebook
• Documentation needs to be updated in Virtualizarr to guide users on how to apply these transformations and schema validation to the virtualized output (in this case padding)
• Having a UTM virtual data cube brings compromises in how can we search for individual metadata. Something zarr-datafusion is trying to address. https://developmentseed.org/zarr-datafusion-search/ we are interested but not a priority.
• We are looking into using Icechunk as the main store for the cubes and have them available through earthmover's data market place. Same with the virtual cubes but first we need to create them.
• There are many potential optimizations the the pipeline that can serve as a blueprint of what's possible with NASA data in the near future and Alex is on board to pursue them (within budget constraints) .

cc. @alex-s-gardner @jhkennedy @mliukis