[wontfix] Minio doesn't support S3 compatible folder structure

[vikrantrathore]

We are using pyfilesystem to abstract the calls to files stored in cloud storage. Pyfilesystem has an api called opendir which works with S3, google file system, local filesystem, NAS. It fails in case of minio.

This is due to fact that minio fails to return directories with files which is supported by S3 API. Minio is not compatible with S3 as it does not return response in this case similar to S3. If we change the code it will become very specific to minio. So I believe it will be nice if this behaviour works like S3, as minio claims to be S3 compatible but in this case it isn’t.

There is a ticket in pyfilesystem project which was closed because it follows S3, but minio is an exception.

Expected Behavior

Minio returns directories with files as S3, to be S3 compatible.

Current Behavior

Minio does not return directories with files, it is a behaviour not compatible with S3.

[nitisht]

Can you add the response returned by S3 vs MinIO with this API call?

[vikrantrathore]

Steps to reproduce

The output in aws:

% python main.py
dirwithfiles/ 0
dirwithfiles/firstfile 7
emptydir/ 0

The output in minio:

% python main.py
dirwithfiles/firstfile 7
emptydir/ 0

create following dir structure to minio:

emptydir/ (this is just empty directory)
dirwithfiles/firstfile (file inside directory)

/storage/test2 # ls -lR
.:
total 8
drwxr-xr-x    2 root     root          4096 Jun 15 05:44 dirwithfiles
drwxr-xr-x    2 root     root          4096 Jun 15 05:44 emptydir

./dirwithfiles:
total 4
-rw-r--r--    1 root     root             7 Jun 15 05:46 firstfile

./emptydir:
total 0

Listing everything:

        response = self.client.list_objects_v2(
            Bucket=self.bucket,
            Prefix='/',
        )
        for obj in response.get('Contents', []):
            print("%s %d" % (obj['Key'], obj['Size']))

[harshavardhana]

This is expected as we used backend filesystems for the namespace.

We have explained our stance on this in the past, this is a broken feature from AWS S3 and better left not implemented.

Most applications that are portable never use this type of behavior, and we do not want to encourage applications that need this type of behavior either.

Search for older issues on why we didn't implement it.

[vikrantrathore]

So there is no intention of minio to be S3 API compatible which is fine.

It will nice to be honest and update the front page of marketing instead of false and misleading message “Amazon’s S3 API is the defacto standard in the object storage world. MinIO is the defacto standard for S3 compatibility.“

[harshavardhana]

So there is no intention of minio to be S3 API compatible which is fine.

It will nice to be honest and update the front page of marketing instead of false and misleading message “Amazon’s S3 API is the defacto standard in the object storage world. MinIO is the defacto standard for S3 compatibility.“

Supporting this pattern is not the corner stone of S3 compatibility, we implement and support 100s of applications and almost all the features in AWS S3 and more.

Ignoring this simple fact is simply just ignoring the underlying effort MinIO took to keep the product simple.

I don't think messaging needs any change here. It's the application that needs to be fixed which is not written in a portable manner.