Skip to content
This repository has been archived by the owner on Dec 7, 2023. It is now read-only.

Shutting down Heroku instance #238

Closed
2 tasks done
cisaacstern opened this issue Dec 5, 2023 · 6 comments
Closed
2 tasks done

Shutting down Heroku instance #238

cisaacstern opened this issue Dec 5, 2023 · 6 comments

Comments

@cisaacstern
Copy link
Member

cisaacstern commented Dec 5, 2023 •
edited
Loading

Following pangeo-forge/pangeo-forge.org#430, I'll be shutting down the Heroku instance.

Here are some related to-dos:
@cisaacstern
Copy link
Member Author

Export existing catalog, and migrate it to some other public location

Here's a glance at what's cataloged currently in the api:

from dataclasses import dataclass
import requests

api = "https://api.pangeo-forge.org"
feedstocks = requests.get(api + "/feedstocks").json()
datasets = {}
for f in feedstocks:
    dss = requests.get(f"{api}/feedstocks/{f['id']}/datasets?type=production").json()
    if dss:
        datasets.update({f["spec"]: dss})

@dataclass
class Dataset:
    recipe_id: str
    dataset_url: str

typed_datasets = {
    k: [
        Dataset(ds["recipe_id"], ds["dataset_public_url"])
        for ds in v
    ]
    for k, v in datasets.items()
}

for k, v in typed_datasets.items():
    print(k)
    for ds in v:
        print(f"  - {ds.recipe_id}: {ds.dataset_url}")
# stylizing this output as yaml for readability

pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock
  - noaa-coastwatch-geopolar-sst: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock/noaa-coastwatch-geopolar-sst.zarr
pangeo-forge/WOA_1degree_monthly-feedstock
  - woa18-1deg-monthly: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/WOA_1degree_monthly-feedstock/woa18-1deg-monthly.zarr
pangeo-forge/cmip6-feedstock
  - test_surface: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/test_surface.zarr
  - test_full_depth: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/test_full_depth.zarr
  - CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429.zarr
  - CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429.zarr
  - CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701.zarr
  - CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r13i1p2f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r13i1p2f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r3i1p1f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r3i1p1f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
pangeo-forge/CMIP6_static_grids-feedstock
  - NorESM2-LM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-LM.zarr
  - GFDL-ESM4: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/GFDL-ESM4.zarr
  - NorESM2-MM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-MM.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
  - MPI-ESM1-2-LR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-LR.zarr
  - NorESM2-MM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-MM.zarr
  - GFDL-ESM4: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/GFDL-ESM4.zarr
  - NorESM2-LM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-LM.zarr
  - MPI-ESM1-2-LR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-LR.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
pangeo-forge/gpcp-feedstock
  - gpcp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr
pangeo-forge/HadISST-feedstock
  - hadisst: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/HadISST-feedstock/hadisst.zarr
pangeo-forge/AGDC-feedstock
  - AGCD: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/AGDC-feedstock/AGCD.zarr
pangeo-forge/CMIP6-PMIP-feedstock
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
  - CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.v20200120: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.v20200120.zarr
  - CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.v20200318: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.v20200318.zarr
pangeo-forge/GPM_3IMERGDL-feedstock
  - gpm3imergdl: s3://yuvipanda-test1/cmr/gpm3imergdl.zarr/
pangeo-forge/eVolv2k-feedstock
  - eVolv2k: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/eVolv2k-feedstock/eVolv2k.zarr
  - eVolv2k: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/eVolv2k-feedstock/eVolv2k.zarr
pangeo-forge/LMRv2p1_MCruns_ensemble_gridded-feedstock
  - LMRv2p1_MCruns_ensemble_gridded: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/LMRv2p1_MCruns_ensemble_gridded-feedstock/LMRv2p1_MCruns_ensemble_gridded.zarr
pangeo-forge/EOBS-feedstock
  - eobs-wind-speed: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-wind-speed.zarr
  - eobs-surface-downwelling: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-surface-downwelling.zarr
  - eobs-tg-tn-tx-rr-hu-pp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-tg-tn-tx-rr-hu-pp.zarr
  - eobs-wind-speed: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-wind-speed.zarr
  - eobs-surface-downwelling: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-surface-downwelling.zarr
  - eobs-tg-tn-tx-rr-hu-pp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-tg-tn-tx-rr-hu-pp.zarr
pangeo-forge/liveocean-feedstock
  - liveocean: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/liveocean-feedstock/liveocean.zarr
pangeo-forge/aws-noaa-oisst-feedstock
  - aws-noaa-oisst-avhrr-only: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/aws-noaa-oisst-feedstock/aws-noaa-oisst-avhrr-only.zarr

@cisaacstern
Copy link
Member Author

Continuing with python session from previous comment, looks like all but the following three datasets are openable:

import warnings
import xarray as xr

for k, v in typed_datasets.items():
    for ds in v:
        try:
            with warnings.catch_warnings():
                warnings.simplefilter('ignore')  # ignore xarray time decode warnings
                _ = xr.open_dataset(ds.dataset_url, engine="zarr")
        except:
            print(k, ds.recipe_id)
pangeo-forge/GPM_3IMERGDL-feedstock gpm3imergdl
pangeo-forge/liveocean-feedstock liveocean
pangeo-forge/aws-noaa-oisst-feedstock aws-noaa-oisst-avhrr-only

@cisaacstern
Copy link
Member Author

So the openable datasets are:

openable_datasets = {
    k: v
    for k, v in typed_datasets.items()
    if k not in [
         "pangeo-forge/GPM_3IMERGDL-feedstock",
         "pangeo-forge/liveocean-feedstock",
         "pangeo-forge/aws-noaa-oisst-feedstock",
    ]
}
len(openable_datasets) # -> 11

for k, v in openable_datasets.items():
    print(k)
    for ds in v:
    print(f"  - {ds.recipe_id}: {ds.dataset_url}")
pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock
  - noaa-coastwatch-geopolar-sst: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock/noaa-coastwatch-geopolar-sst.zarr
pangeo-forge/WOA_1degree_monthly-feedstock
  - woa18-1deg-monthly: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/WOA_1degree_monthly-feedstock/woa18-1deg-monthly.zarr
pangeo-forge/cmip6-feedstock
  - test_surface: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/test_surface.zarr
  - test_full_depth: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/test_full_depth.zarr
  - CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429.zarr
  - CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429.zarr
  - CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701.zarr
  - CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r13i1p2f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r13i1p2f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r3i1p1f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r3i1p1f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
pangeo-forge/CMIP6_static_grids-feedstock
  - NorESM2-LM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-LM.zarr
  - GFDL-ESM4: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/GFDL-ESM4.zarr
  - NorESM2-MM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-MM.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
  - MPI-ESM1-2-LR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-LR.zarr
  - NorESM2-MM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-MM.zarr
  - GFDL-ESM4: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/GFDL-ESM4.zarr
  - NorESM2-LM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-LM.zarr
  - MPI-ESM1-2-LR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-LR.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
pangeo-forge/gpcp-feedstock
  - gpcp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr
pangeo-forge/HadISST-feedstock
  - hadisst: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/HadISST-feedstock/hadisst.zarr
pangeo-forge/AGDC-feedstock
  - AGCD: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/AGDC-feedstock/AGCD.zarr
pangeo-forge/CMIP6-PMIP-feedstock
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
  - CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.v20200120: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.v20200120.zarr
  - CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.v20200318: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.v20200318.zarr
pangeo-forge/eVolv2k-feedstock
  - eVolv2k: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/eVolv2k-feedstock/eVolv2k.zarr
  - eVolv2k: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/eVolv2k-feedstock/eVolv2k.zarr
pangeo-forge/LMRv2p1_MCruns_ensemble_gridded-feedstock
  - LMRv2p1_MCruns_ensemble_gridded: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/LMRv2p1_MCruns_ensemble_gridded-feedstock/LMRv2p1_MCruns_ensemble_gridded.zarr
pangeo-forge/EOBS-feedstock
  - eobs-wind-speed: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-wind-speed.zarr
  - eobs-surface-downwelling: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-surface-downwelling.zarr
  - eobs-tg-tn-tx-rr-hu-pp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-tg-tn-tx-rr-hu-pp.zarr
  - eobs-wind-speed: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-wind-speed.zarr
  - eobs-surface-downwelling: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-surface-downwelling.zarr
  - eobs-tg-tn-tx-rr-hu-pp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-tg-tn-tx-rr-hu-pp.zarr

@cisaacstern
Copy link
Member Author

Moving towards a PR to the leap data catalog :

import yaml

@dataclass
class FlatDataset:
    feedstock: str
    recipe_id: str
    dataset_url: str

flat_datasets = []
for k, v in openable_datasets.items():
    for ds in v:
        flat_datasets.append(
            FlatDataset(k, ds.recipe_id, ds.dataset_url)
        )

@dataclass
class Store:
    name: str
    href: str

@dataclass
class Maintainer:
    name: str
    github_username: str

@dataclass
class LEAPCatalogEntry:
    name: str
    description: str
    stores: list[Store]
    maintainers: list[Maintainer]

leap_catalog_entries = []

for ds in flat_datasets:
    meta_text = requests.get(
        f"https://raw.githubusercontent.com/{ds.feedstock}/main/feedstock/meta.yaml"
    ).text
    meta_yaml = yaml.safe_load(meta_text)
    
    maintainers = [Maintainer(m["name"], m["github"]) for m in meta_yaml["maintainers"]]
    store = Store(name="Pangeo Forge Open Storage Network (OSN)", href=ds.dataset_url)
    
    lce = LEAPCatalogEntry(
        name=ds.recipe_id,
        description=meta_yaml["description"],
        stores=[store],
        maintainers=maintainers,
    )
    leap_catalog_entries.append(lce)

@cisaacstern
Copy link
Member Author

Oops I just realized that I had the structure of the LEAP Catalog wrong, this should be closer to the mark:

leap_catalog_entries_2 = []

for feedstock, datasets in openable_datasets.items():
    meta_text = requests.get(
        f"https://raw.githubusercontent.com/{feedstock}/main/feedstock/meta.yaml"
    ).text
    meta_yaml = yaml.safe_load(meta_text)
    
    maintainers = [Maintainer(m["name"], m["github"]) for m in meta_yaml["maintainers"]]
    
    stores = []
    for ds in datasets:
        stores.append(Store(name=ds.recipe_id, href=ds.dataset_url))
    
    lce = LEAPCatalogEntry(
        name=meta_yaml["title"],
        description=meta_yaml["description"],
        stores=stores,
        maintainers=maintainers,
    )
    leap_catalog_entries_2.append(lce)

@cisaacstern cisaacstern mentioned this issue Dec 6, 2023
Closed
@cisaacstern
Copy link
Member Author

Just deleted the Heroku deployment from the Heroku dashboard, so this is resolved!

@cisaacstern cisaacstern mentioned this issue Dec 7, 2023
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cisaacstern