Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fsmap can't 'get' on simplecache wrapped on adlfs #794

Open
majidaldo opened this issue Oct 9, 2021 · 1 comment
Open

fsmap can't 'get' on simplecache wrapped on adlfs #794

majidaldo opened this issue Oct 9, 2021 · 1 comment

Comments

@majidaldo
Copy link

fs = fsspec.filesystem('simplecache', fs=fsspec.filesystem('az', **kw),)
fsm = fs.get_mapper('/some/where')

In [130]: next(iter(fsm.values()))
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-130-88122b9ca828> in <module>
----> 1 next(iter(fsm.values()))

~\programfiles\miniconda\envs\fixtures-cache\lib\_collections_abc.py in __iter__(self)
    867     def __iter__(self):
    868         for key in self._mapping:
--> 869             yield self._mapping[key]
    870
    871

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\mapping.py in __getitem__(self, key, default)
    133         k = self._key_to_str(key)
    134         try:
--> 135             result = self.fs.cat(k)
    136         except self.missing_exceptions:
    137             if default is not None:

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\implementations\cached.py in <lambda>(*args, **kw)
    391             # all the methods defined in this class. Note `open` here, since
    392             # it calls `_open`, but is actually in superclass
--> 393             return lambda *args, **kw: getattr(type(self), item).__get__(self)(
    394                 *args, **kw
    395             )

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\implementations\cached.py in cat(self, path, recursive, on_error, callback, **kwargs)
    568             fns.append(fn)
    569         if getpaths:
--> 570             self.fs.get(getpaths, storepaths)
    571             self.save_cache()
    572

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in wrapper(*args, **kwargs)
     86     def wrapper(*args, **kwargs):
     87         self = obj or args[0]
---> 88         return sync(self.loop, func, *args, **kwargs)
     89
     90     return wrapper

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in sync(loop, func, timeout, *args, **kwargs)
     67         raise FSTimeoutError
     68     if isinstance(result[0], BaseException):
---> 69         raise result[0]
     70     return result[0]
     71

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in _runner(event, coro, result, timeout)
     23         coro = asyncio.wait_for(coro, timeout=timeout)
     24     try:
---> 25         result[0] = await coro
     26     except Exception as ex:
     27         result[0] = ex

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\fsspec\asyn.py in _get(self, rpath, lpath, recursive, callback, **kwargs)
    422         from fsspec.implementations.local import make_path_posix
    423
--> 424         rpath = self._strip_protocol(rpath)
    425         lpath = make_path_posix(lpath)
    426         rpaths = await self._expand_path(rpath, recursive=recursive)

~\programfiles\miniconda\envs\fixtures-cache\lib\site-packages\adlfs\spec.py in _strip_protocol(cls, path)
    454         STORE_SUFFIX = ".dfs.core.windows.net"
    455         logger.debug(f"_strip_protocol for {path}")
--> 456         if not path.startswith(("abfs://", "az://", "abfss://")):
    457             path = path.lstrip("/")
    458             path = "abfs://" + path

AttributeError: 'list' object has no attribute 'startswith'

AzureDataLakeFS is derived from AsyncFileSystem. Somewhere WholeFileCache creates a list of paths to get by the wrapped fs self.fs.get. But AsyncFilesystem's _get is only expecting one path.

@majidaldo majidaldo changed the title fsmap can't get on simplecache wrapped on adlfs fsmap can't 'get' on simplecache wrapped on adlfs Oct 9, 2021
@martindurant
Copy link
Member

You can presumably show that ABFS's get does not accept multiple paths, although the superclass says that it should (and that the mapper and cache are not related). If that's the case in the current main branch, please raise an issue with them, and link this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants