This is a simplification of my old workflow. It still handles requests, but the collection is now primarily updated by cycling through popular searches. This is done by simply specifying the keyword values of the type of data desired. As before, the ESGF repo is searched for relevant data and a comparison with the GC catalog is made to see what new data is available.
The notebook GetSpecified.ipynb is now what I use day-to-day to update our GC collection.
The notebook GetRequested.ipynb is now what I use day-to-day to handle data requests (may be be phased out in the future)
The esm-collection-spec CSV file for the Pangeo Google Cloud datasets is pangeo-cmip6.csv.
My working request tracker for updating the Google Cloud CMIP6 zarr collection is here.