TypeError: 'coroutine' object is not iterable when reading parquet partitions via s3fs >= 0.5 with pyarrow

[asfimport]

Trying to read partitioned parquet files using updated s3fs >= 0.5 (using via ``dask), and got this error:

 

410 files = set()
 411 
--> 412 for key in list(self.fs._ls(path, refresh=refresh)):
 413 path = key['Key']
 414 if key['StorageClass'] == 'DIRECTORY':

TypeError: 'coroutine' object is not iterable

coming from

arrow/python/pyarrow/filesystem.py

Line 415 in 9baa123

for key in list(self.fs._ls(path, refresh=refresh)):

 

 

Seems related to switching s3fs to asyncio in https://github.com/dask/s3fs/pull/336.

Reporter: Ivan Necas

Related issues:

• [Python] pyarrow doesn't work with s3fs>=0.5 (is duplicated by)
• [Python] pyarrow doesn't work with s3fs>=0.5 (relates to)

_{Note: This issue was originally created as ARROW-10921. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~inecas] thanks for the report! It's good to have an issue with this error, so others can find it. But so as you already noted yourself (on the PR), this is a duplicate of ARROW-10433 and fixed on master (to be released in pyarrow 3.0.0 in January). For now the best solution is indeed to pin s3fs to <0.5).