-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: oss list bucket return all records #12084
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you. Can you add some notes in the code? Is there any reference to the 100 records limit?
https://help.aliyun.com/zh/oss/user-guide/list-objects-18?spm=a2c4g.11186623.0.0.275f7711UTgi3H
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we also bump max-keys to 1000?
Maybe we need a configuration to limit it, because some customers' oss objects are indeed relatively large, exceeding 1000. |
/CI |
@@ -236,12 +236,21 @@ func (ossDriver *ArtifactDriver) ListObjects(artifact *wfv1.Artifact) ([]string, | |||
if err != nil { | |||
return !isTransientOSSErr(err), err | |||
} | |||
results, err := bucket.ListObjectsV2(oss.Prefix(artifact.OSS.Key)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oss.Prefix is removed accidentally?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i forget it. thanks.
workflow/artifacts/oss/oss.go
Outdated
continueToken = results.NextContinuationToken | ||
if !results.IsTruncated { | ||
break | ||
|
||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should check whether it's truncated before accessing next continuation token
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes.
workflow/artifacts/oss/oss.go
Outdated
if err != nil { | ||
return !isTransientOSSErr(err), err | ||
} | ||
// Add to files. By default, 100 records are returned at a time. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok.
Can you address my comments and then run some tests if you have OSS available? |
Can you test it against actual OSS environment and see if everything is working correctly? |
OK. I'll run some tests later. |
I made a test use the file. It works well.
|
Signed-off-by: shuangkun <tsk2013uestc@163.com> Signed-off-by: zjgemi <liuxin_zijian@163.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
Signed-off-by: Dillen Padhiar <dillen_padhiar@intuit.com>
when list a bucket over 100 object, the response can only return 100 records, so wrote a loop to return all records.
Fixes #TODO
Motivation
Modifications
Verification