titiler-pgstac and titiler-xarray code profiling using CMIP6 Daily TAS on AWS PDS

[abarciauskas-bgse]

Execution of the work described in stac-utils/titiler-pgstac#97, specifically stac-utils/titiler-pgstac#97 (comment), copying most content that discussion.

Description

We would like to compare performance of tiling the same dataset using titiler-pgstac (COGs) and titiler-xarray (Zarr)

We believe this is the best approximation of comparing tiling between the two approaches because of the following methodology:

1. Assume you are starting with a known collection
2. Metadata read step
   a. For COG: you register a query with pgstac for the collection id, variable and time step of interest
   b. For Zarr: You read consolidated metadata and "lazily load" the variable and time step (slice) of interest
   c. For now - I'm going to store both of these metadata components (pgstac and kerchunk reference) locally, but it might be of interest to move pgSTAC and kerchunk to the cloud...
3. Generate tiles
   a. COG: use mosaic ID to read chunks from COGs on S3
   b. Zarr: Use zarr metadata to read chunks from NetCDFs or Zarrs on S3

There is a public archive of COGs generated for CMIP6 for a VEDA related dashboard, using NetCDFs we have stored in S3 as well. There is a public archive is here: https://nex-gddp-cmip6-cog.s3.us-west-2.amazonaws.com/ and we have the original NetCDFs in S3 as well. So we can:

For titiler-pgstac:

• Seed a pgSTAC collection of these files (started on this for https://nex-gddp-cmip6-cog.s3.us-west-2.amazonaws.com/index.html#monthly/CMIP6_ensemble_median/tas/ (2,808 entries))
• Determine how to generate a mosaic + tile request
• Write profiling code for titiler-pgstac for a single tile
• Use daily CMIP6 COG files from the NEX GDDP CMIP6 bucket #20
• Deploy and seed an AWS RDS pgSTAC database with CMIP collection and items #16

For titiler-xarray

• Create a kerchunk reference for the NetCDFs in S3
• Write profiling code for titiler-xarray for a single tile (this is already started in sketch profiling titiler-multidim#4)
• Use daily CMIP6 NetCDF files from the NEX GDDP CMIP6 bucket #19
• Store kerchunk reference file in S3 #17
• Create initial zarr store for CMIP 6 daily data and add to profile.ipynb #25
• Create different chunking + pyramiding zarr stores
• Run tests + report
• (Maybe) Test with higher resolution dataset

For both:

• Update profile.ipynb notebook to use temporal resolution options and remote storage options #21
• Test a series of tiles #24

[abarciauskas-bgse]

Done