Folder only showing 1000 files

[cyberduck]

8719699 created the issue

S3 folder only showing 1000 files. I have folders where there are 20,000 files or more and I now can't access them anymore. To my knowledge, they are still there, just not showing in the Cyberduck Browser. I think the new update broke this. I'm on 7.1.0.31395

[cyberduck]

@dkocher commented

I cannot reproduce this issue with a bucket in eu-north-1 with 1037 files uploaded. We should issue multiple requests to fetch keys using marker

GET /?encoding-type=url&max-keys=1000&prefix=TRAC-10811%2F&delimiter=%2F HTTP/1.1
Date: Thu, 19 Sep 2019 07:45:21 GMT
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: test-eu-north-1-cyberduck.s3.amazonaws.com
x-amz-date: 20190919T074521Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.1.31445 (Mac OS X/10.14.6) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: d8DbY3SvmfDop0DY9mtDQ9jwdt02XwaRPt2fblJlxyFBkNF0MMHzset8PBo4iPF0tLtzoCTMJvI=
x-amz-request-id: DF2F772BF05CD1D1
Date: Thu, 19 Sep 2019 07:45:22 GMT
x-amz-bucket-region: eu-north-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?encoding-type=url&max-keys=1000&prefix=TRAC-10811%2F&delimiter=%2F&marker=TRAC-10811%2F1568878850200075000 HTTP/1.1
Date: Thu, 19 Sep 2019 07:45:21 GMT
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: test-eu-north-1-cyberduck.s3.amazonaws.com
x-amz-date: 20190919T074521Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.1.31445 (Mac OS X/10.14.6) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: Who1ZcC3bPW2xN/dKHDUPOsxXXHgHpEZFI24w/NYAbDyiBjM3oJMmaP4fbD0C55myrcuJPenj88=
x-amz-request-id: 593DEA8EA4511C7F
Date: Thu, 19 Sep 2019 07:45:22 GMT
x-amz-bucket-region: eu-north-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3

[cyberduck]

@dkocher commented

Please post the transcript from the log drawer (⌘-L).

[cyberduck]

8719699 commented

This is what I got from Ctrl-L and refreshing the browser window. Looks like the max keys are set to 1000? how do I change that. Never had an issue before.

GET /?encoding-type=url&max-keys=1000&prefix=Repertoire%2FRepertoire%20Current%2F&delimiter=%2F HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:22 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: dwmedia1.s3.amazonaws.com
x-amz-date: 20190919T200622Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.0.31395 (Windows 8.1/6.3) (amd64)
HTTP/1.1 200 OK
x-amz-id-2: TJd6++ButicPmwUNPzule/GYppZUpNB9ywhNkcMbN3CDcPGTvdOH0jcuFep7RK7l4ZtZTMpxPAw=
x-amz-request-id: A7DB4C0E58E7E9AA
Date: Thu, 19 Sep 2019 20:06:31 GMT
x-amz-bucket-region: us-east-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?encoding-type=url&max-keys=1000&prefix=Repertoire%2FRepertoire%20Current%2F&delimiter=%2F&marker=Repertoire%2FRepertoire%2BCurrent%2FHelp%2B-%2BThe%2BBeatles.pdf HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:23 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: dwmedia1.s3.amazonaws.com
x-amz-date: 20190919T200623Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.0.31395 (Windows 8.1/6.3) (amd64)
HTTP/1.1 200 OK
x-amz-id-2: oIZyYuSF/4Z2BRguDqBTg4SgBijjo5Bt9UGCyOalm/UdTWaaY0umHJn2cjH6FzoHjnby3OrixqY=
x-amz-request-id: F9EC3F57D115AC92
Date: Thu, 19 Sep 2019 20:06:32 GMT
x-amz-bucket-region: us-east-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?prefix=Repertoire%2FRepertoire%20Current%2F&uploads HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:23 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: dwmedia1.s3.amazonaws.com
x-amz-date: 20190919T200623Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.0.31395 (Windows 8.1/6.3) (amd64)
HTTP/1.1 200 OK
x-amz-id-2: +12VIpOsviETMoY1EzFA6NMmbDwKf7JElVbe/RzOhDiWwMcNj/WWj8y154oWBel+7fzLD7P0U2A=
x-amz-request-id: 58F38262BC38AE26
Date: Thu, 19 Sep 2019 20:06:33 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3

[cyberduck]

@dkocher commented

I can confirm the max-keys is set to 1000 by default. Thus for directories containing more files we issue multiple GET request with a marker parameter.

GET /?encoding-type=url&max-keys=1000&prefix=Repertoire%2FRepertoire%20Current%2F&delimiter=%2F&marker=Repertoire%2FRepertoire%2BCurrent%2FHelp%2B-%2BThe%2BBeatles.pdf HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:23 GMT

Given the two requests in the above transcript you should see more than 1000 files but less than 2000.

[cyberduck]

8719699 commented

Currently it brings up exactly 1000. I know barely anything about coding/your system (I took one coding class in python in college and can do very basic HTML/CSS). How can I make it pull all of my files up again?

[cyberduck]

8719699 commented

Hey, just wanted to check in. Any updates on this? Need anything more from me to figure it out? More than happy to adjust settings over here if needed, but couldn't find anything in the different menus that helped with this. Thank you!

[cyberduck]

@dkocher commented

Please update to the latest snapshot build available and let us know if the issue can still be reproduced.

[cyberduck]

8719699 commented

Still having the same issues on the snapshot build 7.1.2.31581. No difference in how it handles the huge folders/buckets (only showing exactly 1000 files)

[cyberduck]

@dkocher commented

In d61df59.