Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUGFIX] Using headObject to proof that a folder exists does not work in every case #45

Closed
wants to merge 1 commit into from

Conversation

geldmacher
Copy link

If files in the bucket were created by a 3rd party tool and the folder structure is not created explicity beforehand (which is not necesssary because S3 allows a structured filesystem just by prefixes), the headObject method returns a 404, because there is no folder object to fetch meta data from.

We can use listObjects in this case. If it delivers min. 1 object, we can assume that it is a folder.

… in every case

If files in the bucket were created by a 3rd party tool and the folder structure is not created explicity beforehand (which is not necesssary because S3 allows a structured filesystem just by prefixes), the headObject method returns a 404, because there is no folder object to fetch meta data from. 

We can use listObjects in this case. If it delivers min. 1 object, we can assume that it is a folder.
@GerDner
Copy link

GerDner commented Mar 20, 2019

The listObjects case does not work, or does not work with API v4.

The following code snipped does work for me.

    /**
     * Checks if a prefix exists
     *
     * @param string $identifier
     * @return bool
     */
    protected function prefixExists($identifier)
    {

        if($this->objectExists($identifier)){
            return true;
        }
        $objects = $this->getListObjects($identifier);

        return $objects['KeyCount'] > 0;
    }

@Lagerregal Lagerregal closed this in 1671cea Mar 3, 2021
cweiske added a commit to mogic-le/aus_driver_amazon_s3 that referenced this pull request Jul 2, 2021
aus_driver_amazon_s3 falls back to ListObjectsV2 when HEAD requests fail
on folders since commit 1671cea.
This is because folders are created implicitly in S3
(because files have slashes in their names), and thus HEAD requests
on such implicit folders fail.
(See issue andersundsehr#45
and andersundsehr#57)

This degrades performance when implicit folders are used:
A HEAD request is sent, followed by a ListObjectsV2 request.

We *always* need only one HTTP request when doing the ListObjectsV2
request first.

This means we could completely remove the HEAD request,
but the MinIO storage backend has some strange behavior/bug[1]:
- non-empty directory appears in ListObjectsV2, but not HEAD
- an empty directory does not appear in ListObjectsV2, but does in HEAD

Thus the HEAD request stays, but only as fallback to support empty
directories on the MinIO storage backend.

[1] minio/minio#6010 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants