HTTP client_kwargs authentication not used with simplecache

[scottyhq]

Using fsspec='0.8.4+70.g33d532f' if a simplecache:: prefix is used with HTTP:// it seems that authentication settings are not used leading to ClientResponseError: 401, message='Unauthorized'

This works:

import fsspec
import xarray as xr
import netrc
import aiohttp

(username, account, password) = netrc.netrc().authenticators("urs.earthdata.nasa.gov")
uri = 'https://grfn.asf.alaska.edu/door/download/S1-GUNW-A-R-137-tops-20200311_20200305-020013-43055N_41518N-PP-00fe-v2_0_2.nc'
with fsspec.open(item.data.urlpath, 
                 client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) as f:
    ds = xr.open_dataset(f, group='/science/grids/data', engine='h5netcdf', chunks={})

This does not:

with fsspec.open('simplecache::'+item.data.urlpath, 
                 client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) as f:
    ds = xr.open_dataset(f, group='/science/grids/data', engine='h5netcdf', chunks={})

---------------------------------------------------------------------------
ClientResponseError                       Traceback (most recent call last)
<ipython-input-17-a30d2d9da865> in <module>
      1 # with simplecache
      2 with fsspec.open('simplecache::'+item.data.urlpath, 
----> 3                  client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) as f:
      4     ds = xr.open_dataset(f, group='/science/grids/data', engine='h5netcdf', chunks={})
      5 

~/GitHub/filesystem_spec/fsspec/core.py in __enter__(self)
    100         mode = self.mode.replace("t", "").replace("b", "") + "b"
    101 
--> 102         f = self.fs.open(self.path, mode=mode)
    103 
    104         self.fobjects = [f]

~/GitHub/filesystem_spec/fsspec/implementations/cached.py in <lambda>(*args, **kw)
    384             # all the methods defined in this class. Note `open` here, since
    385             # it calls `_open`, but is actually in superclass
--> 386             return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
    387         if item in ["__reduce_ex__"]:
    388             raise AttributeError

~/GitHub/filesystem_spec/fsspec/spec.py in open(self, path, mode, block_size, cache_options, **kwargs)
    912                 autocommit=ac,
    913                 cache_options=cache_options,
--> 914                 **kwargs
    915             )
    916             if not ac:

~/GitHub/filesystem_spec/fsspec/implementations/cached.py in <lambda>(*args, **kw)
    384             # all the methods defined in this class. Note `open` here, since
    385             # it calls `_open`, but is actually in superclass
--> 386             return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
    387         if item in ["__reduce_ex__"]:
    388             raise AttributeError

~/GitHub/filesystem_spec/fsspec/implementations/cached.py in _open(self, path, mode, **kwargs)
    637                     f2.write(data)
    638         else:
--> 639             self.fs.get(path, fn)
    640         return self._open(path, mode)
    641 

~/GitHub/filesystem_spec/fsspec/asyn.py in get(self, rpath, lpath, recursive, **kwargs)
    279         lpaths = other_paths(rpaths, lpath)
    280         [os.makedirs(os.path.dirname(lp), exist_ok=True) for lp in lpaths]
--> 281         return sync(self.loop, self._get, rpaths, lpaths)
    282 
    283 

~/GitHub/filesystem_spec/fsspec/asyn.py in sync(loop, func, callback_timeout, *args, **kwargs)
     69     if error[0]:
     70         typ, exc, tb = error[0]
---> 71         raise exc.with_traceback(tb)
     72     else:
     73         return result[0]

~/GitHub/filesystem_spec/fsspec/asyn.py in f()
     53             if callback_timeout is not None:
     54                 future = asyncio.wait_for(future, callback_timeout)
---> 55             result[0] = await future
     56         except Exception:
     57             error[0] = sys.exc_info()

~/GitHub/filesystem_spec/fsspec/asyn.py in _get(self, rpaths, lpaths, **kwargs)
    267             *[
    268                 self._get_file(rpath, lpath, **kwargs)
--> 269                 for lpath, rpath in zip(lpaths, rpaths)
    270             ]
    271         )

~/GitHub/filesystem_spec/fsspec/implementations/http.py in _get_file(self, rpath, lpath, chunk_size, **kwargs)
    177             if r.status == 404:
    178                 raise FileNotFoundError(rpath)
--> 179             r.raise_for_status()
    180             with open(lpath, "wb") as fd:
    181                 chunk = True

~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/aiohttp/client_reqrep.py in raise_for_status(self)
    944                 status=self.status,
    945                 message=self.reason,
--> 946                 headers=self.headers)
    947 
    948     def _cleanup_writer(self) -> None:

ClientResponseError: 401, message='Unauthorized', url=URL('https://urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=iwntGSgHy9yoog7Mjag0dQ&response_type=code&redirect_uri=https://grfn.asf.alaska.edu/door/oauth&state=aHR0cDovL2dyZm4uYXNmLmFsYXNrYS5lZHUvZG9vci9kb3dubG9hZC9TMS1HVU5XLUEtUi0xMzctdG9wcy0yMDIwMDMxMV8yMDIwMDMwNS0wMjAwMTMtNDMwNTVOXzQxNTE4Ti1QUC0wMGZlLXYyXzBfMi5uYw')

I also tried entering authentication settings via fsspec.config.conf['https'] = dict(client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) but get the same traceback.

[martindurant]

with fsspec.open(item.data.urlpath, 
                 client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) as f:

->

with fsspec.open(item.data.urlpath, 
                 {"http": dict(client_kwargs={'auth': aiohttp.BasicAuth(username, password)})}) as f:

because otherwise fsspec thinks that client_kwargs is an argument to simplecache.

I also tried entering authentication settings via fsspec.config.conf['https'] = dict(client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) but get the same traceback.

It may be you need "http" - I'm not sure I tried the case where the implementation has two protocols associated with it.

[scottyhq]

Thanks @martindurant for the suggestions!

I've finally hit the correct combination:

uri = 'https://grfn.asf.alaska.edu/door/download/S1-GUNW-A-R-137-tops-20200311_20200305-020013-43055N_41518N-PP-00fe-v2_0_2.nc'
with fsspec.open('simplecache::'+uri,
                 https=dict(client_kwargs={'auth': aiohttp.BasicAuth(username, password)}),
                 ) as f:
    ds = xr.open_dataset(f, group='/science/grids/data', engine='h5netcdf', chunks={})

ds

However https= is critical. http does not work. I'd be nice if either one worked, or perhaps just a note in the docs would be sufficient since from fsspec.registry import known_implementations shows both "http" and "https" pointing to 'fsspec.implementations.http.HTTPFileSystem',`.

It seems I initially got really confused trying the following:

fsspec.config.conf['http'] = dict(client_kwargs={'auth': aiohttp.BasicAuth(username, password)})
uri = 'https://grfn.asf.alaska.edu/door/download/S1-GUNW-A-R-137-tops-20200311_20200305-020013-43055N_41518N-PP-00fe-v2_0_2.nc'
with fsspec.open('simplecache::'+uri) as f:
    ds = xr.open_dataset(f, group='/science/grids/data', engine='h5netcdf', chunks={})

ds

Which gives the traceback. Even changing the config setting to https results in the traceback b/c of caching I think. Only when starting a fresh kernel with the https keyword does this work.

[martindurant]

Yeah, for the arguments to open, this is done by parsing the URL, so the protocol needs to match. In theory, the protocols in the registry are dynamic, and not necessarily tied to the .protocol attribute of the implementation class, so I don' think there's more that I can do about that.

Even changing the config setting to https results in the traceback b/c of caching I think. Only when starting a fresh kernel with the https keyword does this work.

That's worth investigating. It could be that the instance is cached only on the passed arguments, not the implicit ones from the config - but I didn't think that was the case. In any case, you can always state that you want to ignore the cache at instantiation ( skip_instance_cache=True ) or clear the cache ( Implementation.clear_instance_cache() ) so not have to restart the kernel.