Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-5101. Fix Maxkeys is invalid when delimiter is not null #2154

Closed
wants to merge 3 commits into from

Conversation

wzhallright
Copy link
Contributor

JIRA: https://issues.apache.org/jira/browse/HDDS-5101?filter=-2

When get keys from s3g, the keys in my cluster is 
  main/chunk/00000000 main/chunk/00000001 ... main/chunk/99999999
if delimiter is not null, until all keys are queried, the count is always 1 ,so the max-keys is invalid.
image

@wzhallright
Copy link
Contributor Author

The failed UT does not seem to be caused by my modification

Copy link
Member

@elek elek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @wzhallright to report this problem

Can you please give more information what is the original problem? What was your s3 request, and what was the expected result?

For me, the current behavior seems to be fine:

I tested it with creating keys in real AWS:

aws s3api list-objects --bucket=ozonetest3  | grep Key
            "Key": "copyobject/f1",
            "Key": "key1",
            "Key": "test/dir1/file1",
            "Key": "test/dir1/file2",
            "Key": "test/dir1/file3",
            "Key": "test/dir2/file4",
            "Key": "test/dir2/file5",
            "Key": "test/dir2/file6",
            "Key": "test/dir3/file7",

And executing aws cli:

aws s3api list-objects --bucket=ozonetest3 --prefix=test/ --delimiter=/ --max-keys=2 {
    "IsTruncated": true,
    "Marker": "",
    "NextMarker": "test/dir2/",
    "Name": "ozonetest3",
    "Prefix": "test/",
    "Delimiter": "/",
    "MaxKeys": 2,
    "CommonPrefixes": [
        {
            "Prefix": "test/dir1/"
        },
        {
            "Prefix": "test/dir2/"
        }
    ],
    "EncodingType": "url"
}

The result looks good to me. As the documentation says:

These rolled-up keys are not returned elsewhere in the response. Each rolled-up result counts as only one return against the MaxKeys value.

But this behavior is implemented with the existing code in Ozone.

What did I miss?

@wzhallright wzhallright deleted the HDDS-5101 branch April 30, 2021 09:38
@wzhallright wzhallright restored the HDDS-5101 branch April 30, 2021 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants