Cannot delete directory

[sweco]

I'm unable to delete an S3 directory, i.e. a path ending with / using rm nor bulk_delete. There is no error raised but the actual directory does not get deleted.

Code to reproduce:

# Create a folder in an S3 bucket via S3 console
s3fs = S3FileSystem()
s3fs.rm('bucket/folder/')
# Observe the directory remained in S3

I guess this is the result of calling the S3FileSystem.split_path(path) which calls the AbstractFileSystem._strip_protocol(path) which also strips the trailing slash.

[mariusvniekerk]

S3 doesn't actually have directories, only files

[sweco]

You're right, I basically meant a path ending with / which then appears as a directory in the S3 console.

[martindurant]

Agree, stripping of trailing "/" should only happen at list time, not for rm/open. Uncertain about info.

[martindurant]

Will port fsspec/gcsfs#273 to fix this

[abulka]

In the current s3fs 0.4.2 I still cannot delete a

path ending with / which then appears as a directory in the S3 console.

even though s3fs.exists('bucket/folder/') reports True. I'm having to work around it by going to native boto3

s3resource = boto3.resource('s3', ...)
s3resource.Object('bucket', 'folder/').delete()

Repro

Create the offending dir (yeah I know there are really no such things as folders in S3, only keys with a trailing /)

BUCKET = 'bucket'
s3client = boto3.client('s3', ...)
s3client.put_object(Bucket=BUCKET, Key=('folder/'))

s3fs can see it but can't delete it

print(fs.exists(os.path.join(BUCKET, 'folder/')))  # True
fs.rm('bucket/folder/') # FileNotFoundError
fs.rm('bucket/folder/', recursive=True)  # FileNotFoundError

The workaround

s3resource.Object(BUCKET, 'folder/').delete()

[martindurant]

Does the placeholder key show up with find, and does it have the "/" on the end? Does it show up as a file or a directory?

[rom1504]

I'm hitting a similar issue when trying to use fs.rm(folder, recursive=True) : it works (everything is deleted) but fails with FileNotFoundError for the folder

[martindurant]

Thanks for the ping, I'll look into it.

[leeprevost]

I get this error for either s3://bucket/path/ and /bucket/path, or bucket/path:

An error was encountered:
_run_coros_in_chunks() got an unexpected keyword argument 'nofiles'
Traceback (most recent call last):
  File "/home/glue_user/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 88, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/glue_user/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 69, in sync
    raise result[0]
  File "/home/glue_user/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/glue_user/.local/lib/python3.10/site-packages/s3fs/core.py", line 1821, in _rm
    await _run_coros_in_chunks(
TypeError: _run_coros_in_chunks() got an unexpected keyword argument 'nofiles'

[martindurant]

@leeprevost , please state exactly what you did, and the versions you are using. Your case seems to be very different from the rest of this issue.

[leeprevost]

I'm running this in an ETL pyspark script and testing on an AWS glue container.

s3fs.__version__ = '2022.11.0'
fs = s3fs.S3FileSystem()  # credentials through a bound drive to my .aws folder
file_path = "s3://my_bucket/folder/"   # my acct-id is in my file_path so left out
assert fs.exists(file_path)   # no assertion error
fs.glob(file_path + '**')
[ big listing of parquet files]
fs.rm(file_path, rercursive = True)

An error was encountered:
_run_coros_in_chunks() got an unexpected keyword argument 'nofiles'
Traceback (most recent call last):
  File "/home/glue_user/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 88, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/glue_user/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 69, in sync
    raise result[0]
  File "/home/glue_user/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/glue_user/.local/lib/python3.10/site-packages/s3fs/core.py", line 1821, in _rm
    await _run_coros_in_chunks(
TypeError: _run_coros_in_chunks() got an unexpected keyword argument 'nofiles'

[martindurant]

'2022.11.0'

This is pretty old; I expect that maybe you have a version of fsspec which doesn't match. The best solution would be to update both to a consistent version (they are released together).