Directories (CommonPrefixes) are shown even after deletion - inconsistency with AWS S3 API

[tomekit]

I am creating this to understand what's MinIO view on this issue and if this is classified as a bug (and will eventually be fixed) or is a feature (https://xkcd.com/1172/)

There was a similar one here: #10914 (comment)
Its title had: "[working as intended]" prefix appended, but given the fact that MinIO is pretty unique with this behavior in the S3 world (I haven't spotted this anywhere else, including AWS S3 API) I think it's worth revisiting this topic again.

Versioning must be enabled on the bucket level.

Expected Behavior

This is behavior that you can find on AWS S3 and many other providers: https://github.com/s3drive/app/blob/master/S3_PROVIDERS.md

Upload/Delete/Listing on AWS or non-MinIO S3

BUCKET_NAME=bucket

touch somefile.txt

### Upload file
aws s3 cp somefile.txt  s3://$BUCKET_NAME/directoryPrefix/somefile.txt
upload: ./somefile.txt to s3://bucket/directoryPrefix/somefile.txt

### Delete file
aws s3 rm s3://$BUCKET_NAME/directoryPrefix/somefile.txt
delete: s3://bucket/directoryPrefix/somefile.txt

### Directory listing does not appear - CORRECT
aws s3 ls s3://$BUCKET_NAME
<NOTHING GETS RETURNED - CORRECT>

Current Behavior

This is behavior that you can find on current MinIO version: RELEASE.2023-08-04T17-40-21Z

Upload/Delete/Listing on MinIO

## Setup
BUCKET_NAME=miniobucket
ENDPOINT=https://your.minio.endpoint
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin

touch somefile.txt

## Upload file
aws s3 --endpoint=$ENDPOINT cp somefile.txt  s3://$BUCKET_NAME/directoryPrefix/somefile.txt
upload: ./somefile.txt to s3://miniobucket/directoryPrefix/somefile.txt

## Delete file
aws s3 --endpoint=$ENDPOINT rm s3://$BUCKET_NAME/directoryPrefix/somefile.txt
delete: s3://miniobucket/directoryPrefix/somefile.txt

## Directory listing is returned despite no current file - INCORRECT
aws s3 --endpoint=$ENDPOINT ls s3://$BUCKET_NAME
    PRE directoryPrefix/

Context

We're running S3Drive (https://s3drive.app/) which is a file manager running on top of the S3 protocol. We're relying purely on S3 API response to display files&directories listings back to the user. Because of this issue, users see the folder structure even after it's being successfully deleted, this is unexpected.

Other projects, e.g. NextCloud which heavily utilize MinIO, also faced this issue and developed a MinIO specfic workaround: nextcloud/files_versions_s3#11 (comment) to hide these entries.

In our case we don't run separate files metadata store which we could easily use to store info about deleted paths and query against to determine what entries we should hide. Such workaround it's not feasible on our end, because we rely purely on correctness of S3 API. I also think that such workaround goes against the idea of file management and versioning concepts.

You can compare this to deleting a folders and subfolders on your OS whether Linux, Windows or macOS.
After deletion, directories and files end up versioned (they latest version is versioned) in Trash/Recycle Bin, but user wouldn't expect to see them in the original directory location, but that's pretty much what MinIO does currently.

Your Environment

RELEASE.2023-08-04T17-40-21Z

podman run -p 443:9000 -p 9001:9001 --name minio -v /data:/data -v /miniocerts:/miniocerts quay.io/minio/minio server /data --console-address ":9001" --certs-dir /miniocerts

[klauspost]

ACK. The solution isn't trivial, and we have not had any commercial interest for this, so it has not been a high priority issue. We will discuss internally.

[harshavardhana]

Doing this in a performant manner is not possible as it stands today, converting every non-recursive listing into a recursive listing will not work in terms of the number of IOPs spent to do that.

Showing empty folders is not necessarily a real issue, it does show for a real reason that you do not have your data deleted yet. In hindsight, it will help you know that there is data left at those prefixes.

[tomekit]

Thank you for your response. I understand that MinIO's past storage format design decisions doesn't make this viable as it stands now. I appreciate your response and technical insight.

We could really finish it here, but please do not try to convince me that this broken API response is in fact one of the great strengths of MinIO. It's not. It's a limitation which stands out from commonly accepted and implemented S3 API.

it does show for a real reason that you do not have your data deleted yet. In hindsight, it will help you know that there is data left at those prefixes.

If one wants to know that there is any version left they use the version API (https://docs.aws.amazon.com/cli/latest/reference/s3api/list-object-versions.html) instead of relying on coincidental broken MinIO output from list-objects-v2.

Standard listing method (https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects-v2.html) is supposed to show the current state.

Deleted items which no longer appear in the listings are no longer current, since they're versioned with the DeleteMarker being at the top. MinIO's list-objects-v2 mixes both current and deleted state, how can that be any good?

From: #10914 (comment) by @harshavardhana

This is behaving in accordance with AWS S3 behavior.

It's not. Please check AWS S3 API example provided, they handle that correctly: #17810 (comment)
It's also the case with Backblaze, Wasabi, Scaleway, iDrive and many other major S3 providers. MinIO is an exception here.

Showing empty folders is not necessarily a real issue

It's an issue for anyone who relies on the correctness of the S3 API and aim to implement files & directories concept.

It's an issue for projects like: https://rclone.org/, https://github.com/kahing/goofys, https://github.com/s3fs-fuse/s3fs-fuse ... and many others who aim to emulate FS (POSIX compatible or not) using S3 API.

It was an issue for NextCloud who had to develop a workaround: nextcloud/files_versions_s3#11 (comment)

It's an issue even for your own web browser which shows deleted objects despite that: "Show deleted objects" isn't toggled on.

It also shall be an issue for MinIO as a project if you really aim to deliver on your publicly listed goals (https://min.io/):

AWS S3 Compatible Object Storage
S3 compatibility is a hard requirement
adherence to the API alternative to AWS S3
ensures that no other AWS alternative is more compatible.
MinIO’s massive community ensures that no other AWS alternative is more compatible.

Don't get me wrong, hats off to MinIO and its contributors for building such a great tool and providing a robust implementation of one of the most popular blob storage protocols.

It's just if you simply admitted the issue, mentioned about the technical challenges and business viability, I think everyone would understand and this post wouldn't be necessary.

[harshavardhana]

CommonPrefixes for DELETE marker to show up as a flat key is not a cornerstone of S3 compatibility, you may see it that way but that isn't true at all. Such a line of thought simply ignores the rest of the stuff.

Otherwise, MinIO won't be used by any applications out there. Yes, filesystem emulations such as s3fs-fuse might have some need here but that is a niche.

We took this approach willingly to keep many things simple by design

• Design parts are simple such as the choice of using direct object
  names on the backend, so that during operational issues this doesn't
  become a hindrance.
• No DB model - relies on the filesystem namespace for constructing the
  namespace, which has the pitfall mentioned in this issue.

It isn't that we do not know how to address it - there are multiple ways but supporting this simply means overcomplicating everything else, even compromising on some design goals - just to satisfy what AWS S3 returns for a bucket that is versioned and returns a common-prefix in the non-recursive listing.

Currently, we have chosen the current design of keeping object-name to path a one-to-one mapping. If we were to change that supporting AWS S3 behavior could become trivial impacting operation costs. Erasure-coded backend would simply become too detail-oriented.

Right now there is no such pressing need, you have some valid points regarding POSIX mounts but that is an area we are willing to sacrifice for keeping our on-disk format simplicity.