Pyramiding breakout group at Zarr sprint on Feb 8, 2024

[maxrjones]

Originally posted in zarr-developers/geozarr-spec#33 (comment), I'll be leading a 1.45 hour virtual breakout group on Zarr pyramids in association with the upcoming Zarr sprint organized by the Cloud-Native Geospatial Foundation. Information is copied below:

Date: February 8, 2024
Time: 2:30 - 4:15 pm ET
Where: Zoom
How to register: https://lu.ma/Zarr-NYC
How to get more information: Join the cloud-native geospatial slack and add the "zarr-sprint" channel

Long description
A Zarr Sprint track focused on geospatial multi-scales / pyramids.

Our focus is on identifying and addressing shortcomings of the ndpyramid utility, either though development in that library or deciding where else development would need to happen.

ndpyramid is a utility for generating pyramids for Zarr datasets to enable performant visualization. The library was built specifically for use with the @carbonplan/maps toolkit and produces pyramids conforming to this schema. There has been persistent community interest in broader support and standards for pyramids in Zarr. The focus of this sprint is on establishing whether ndpyramid could provide the foundation for this support, and, if so, develop towards that goal, or, if not, establish where else development would happen.

We anticipate that this sprint will progress development towards geospatial pyramids in Zarr that can be used broadly for dynamic client visualization approaches, tiling servers, and multi-scale analysis. This will serve data providers, front-end developers, and researchers.

This will be achieved when:

• We demonstrate the generation and usage of Zarr pyramids with @carbonplan/maps, titiler-xarray, QGIS, Datashader, and a SRCNN, all sharing the same schema.

Some potential goals for this breakout group session:

• Bring your own data: Try out generating pyramids on your dataset and identify where current tools fall short
• Define how different Tile Matrix Sets would be represented in the pyramid schema
• Implement a proof of concept for producing a World CRS84 Quad pyramid using ndpyramid
• Test a proof of concept for loading raw data and pyramids stored separately using a single datatree entrypoint

The types of skills needed complete this task are moderate familiarity with Python and Zarr. We would especially encourage participate from those familiar with geospatial projections, multiscale representations, and metadata conventions.

We expect the level of difficulty to complete this to be medium.

We'll have a discussion about specific priorities on the slack channel on February 7, so if you're interested in the breakout group please try to join by then.

[maxrjones]

I posted a short summary of the outcomes here:

In the Zarr pyramids breakout group, Thomas Maschler and I discussed the motivations for following the OGC TileMatrixSet 2.0 specification within the GeoZarr specification, which will be shared as a new issue to supersede zarr-developers/geozarr-spec#30. We also discussed reading those TMS into rio-tiler using Xarray and started a refactor of ndpyramid to support the TMS specification.