Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

awss3 List method ignores NextMarker #66

Closed
peczenyj opened this issue Aug 23, 2018 · 0 comments · Fixed by #72
Closed

awss3 List method ignores NextMarker #66

peczenyj opened this issue Aug 23, 2018 · 0 comments · Fixed by #72

Comments

@peczenyj
Copy link
Contributor

Hello

I had some troubles to list the entire content of one s3 bucket - it only returns until the PageSize value but does not give me the next marker when the response is Truncated

q := cloudstorage.NewQuery(path)
or, err := s.StoreReader.List(context.Background(), q)
...
spew.Dump(or.NextMarker) // always empty

when I investigate I find two problems here:

https://github.com/lytics/cloudstorage/blob/master/awss3/store.go#L309

	if resp.IsTruncated != nil && *resp.IsTruncated {
		q.Marker = *resp.Contents[len(resp.Contents)-1].Key
	}

first, the query q is not a reference/pointer to a Query, it is passed by copy, so set the marker into q is useless

second, it does not fill the objResp.NextMarker ( like Azure https://github.com/lytics/cloudstorage/blob/master/azure/store.go#L265 )

This may affect other backends, I have an workaround for this but will be great a fix -- I will try to submit a pull request.

peczenyj added a commit to Weborama/cloudstorage that referenced this issue Aug 23, 2018
I am not sure if I can provide a unit test for this case, my problem is iterate over a huge s3 bucket without manually handle the next marker
araddon pushed a commit that referenced this issue Aug 25, 2018
araddon added a commit that referenced this issue Aug 25, 2018
fix issue #66 aws-s3 paging/next-marker is not correct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant