KeyError: 'HTTPServer'

[raybellwaves]

A combination of #37 and #26

I was trying the same logic to access the GFS archive

url = "simplecache::https://rda.ucar.edu/thredds/catalog/files/g/ds084.1/2020/20200101/catalog.xml"
intake.open_thredds_merged(
    url,
    path=["gfs.0p25.2020010100.f*.grib2"],
    driver="netcdf",
    concat_kwargs={"dim": "time"},
    xarray_kwargs=dict(
        engine="cfgrib", backend_kwargs=dict(filter_by_keys={"cfVarName": "t2m"})
    ),
).to_dask()

This gives KeyError: 'HTTPServer' with the Traceback below

KeyError                                  Traceback (most recent call last)
<ipython-input-6-d2ab48b9bee8> in <module>
      1 url = "simplecache::https://rda.ucar.edu/thredds/catalog/files/g/ds084.1/2020/20200101/catalog.xml"
----> 2 intake.open_thredds_merged(
      3     url,
      4     path=["gfs.0p25.2020010100.f*.grib2"],
      5     driver="netcdf",

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/intake_xarray/base.py in to_dask(self)
     67     def to_dask(self):
     68         """Return xarray object where variables are dask arrays"""
---> 69         return self.read_chunked()
     70 
     71     def close(self):

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/intake_xarray/base.py in read_chunked(self)
     42     def read_chunked(self):
     43         """Return xarray object (which will have chunks)"""
---> 44         self._load_metadata()
     45         return self._ds
     46 

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/intake/source/base.py in _load_metadata(self)
    234         """load metadata only if needed"""
    235         if self._schema is None:
--> 236             self._schema = self._get_schema()
    237             self.dtype = self._schema.dtype
    238             self.shape = self._schema.shape

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/intake_xarray/base.py in _get_schema(self)
     16 
     17         if self._ds is None:
---> 18             self._open_dataset()
     19 
     20             metadata = {

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/intake_thredds/source.py in _open_dataset(self)
     95 
     96         if self._ds is None:
---> 97             cat = ThreddsCatalog(self.urlpath, driver=self.driver)
     98             for i in range(len(self.path)):
     99                 part = self.path[i]

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/intake_thredds/cat.py in __init__(self, url, driver, **kwargs)
     28         self.url = url
     29         self.driver = driver
---> 30         super().__init__(**kwargs)
     31 
     32     def _load(self):

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/intake/catalog/base.py in __init__(self, entries, name, description, metadata, ttl, getenv, getshell, persist_mode, storage_options)
     98         self.updated = time.time()
     99         self._entries = entries if entries is not None else self._make_entries_container()
--> 100         self.force_reload()
    101 
    102     @classmethod

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/intake/catalog/base.py in force_reload(self)
    156         """Imperative reload data now"""
    157         self.updated = time.time()
--> 158         self._load()
    159 
    160     def reload(self):

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/intake_thredds/cat.py in _load(self)
     79 
     80         self._entries.update(
---> 81             {
     82                 ds.name: LocalCatalogEntry(
     83                     ds.name,

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/intake_thredds/cat.py in <dictcomp>(.0)
     85                     self.driver,
     86                     True,
---> 87                     {'urlpath': access_urls(ds, self), 'chunks': {}},
     88                     [],
     89                     [],

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/intake_thredds/cat.py in access_urls(ds, self)
     73             elif self.driver == 'netcdf':
     74                 driver_for_access_urls = 'HTTPServer'
---> 75             url = ds.access_urls[driver_for_access_urls]
     76             if 'fsspec_pre_url' in self.metadata.keys():
     77                 url = f'{self.metadata["fsspec_pre_url"]}{url}'

~/opt/miniconda3/envs/test_env/lib/python3.9/site-packages/siphon/catalog.py in __getitem__(self, key)
    219     def __getitem__(self, key):
    220         """Return value from case-insensitive lookup of ``key``."""
--> 221         return super(CaseInsensitiveDict, self).__getitem__(CaseInsensitiveStr(key))
    222 
    223     def __setitem__(self, key, value):

[raybellwaves]

If I try and get at one file

url = "https://rda.ucar.edu/thredds/catalog/files/g/ds084.1/2020/20200101/catalog.xml"
source = intake.open_thredds_cat(url)
source['gfs.0p25.2020010100.f000.grib2']

it'll use driver: intake_xarray.opendap.OpenDapSource. This will failure later without some cfgrib like filtering.

If I add simplecaching and driver="netcdf" I get the HTTPServer error:

url = "simplecache::https://rda.ucar.edu/thredds/catalog/files/g/ds084.1/2020/20200101/catalog.xml"
source = intake.open_thredds_cat(url, driver="netcdf")

Similarly I get the HTTPServer error using open_thredds_merged

url = "simplecache::https://rda.ucar.edu/thredds/catalog/files/g/ds084.1/2020/20200101/catalog.xml"
source = intake.open_thredds_merged(url, path=["gfs.0p25.2020010100.f000.grib2"], driver="netcdf")

[aaronspring]

On this thredds server, there seems to be no HTTPServer, see https://rda.ucar.edu/thredds/catalog/files/g/ds084.1/2020/20200101/catalog.html?dataset=files/g/ds084.1/2020/20200101/gfs.0p25.2020010100.f000.grib2.

A quick check is always to exchange xml with html and browse the files by mouse. Opendap is available and therefore works, but opendap and caching are incompatible.