Not all files listed

[dstreufert]

In the case where I have around 17,000 files in a directory, only the first 600-900 are shown in the directory listing.

Any ideas?

[Erisa]

This is because R2 has a limit on how many objects can be listed in a single call to the API, which is 1,000 maximum but (If I recall correctly) can be lower based on the total byte size of the result. https://developers.cloudflare.com/r2/api/workers/workers-api-reference/#r2listoptions

Listing more than this using the Workers bindings requires a change to make multiple list calls, which can dramatically increase the latency and cost of the request, so would be tricky to implement in a sensible way.

[kotx]

We could add a ...see more... link at the bottom of the page if there are more objects to be returned.
This would add a ?cursor=... to the url, which would then return all results after the cursor. You could then list all objects similarly to this method in cloudflare docs.

The question is if you need a configurable limit of objects per page (for example, so you can limit results to 50 per page), and/or an option for listing all files in a single request if the number of subrequests doesn't go over 50 on free, or 1k on paid.
Let me know what works for your needs, I can try to add it in the next few days.

[dstreufert]

I think “see more” and/or option for “unlimited” files would be great!

…

________________________________ From: Kot C ***@***.***> Sent: Tuesday, January 23, 2024 7:26:17 PM To: kotx/render ***@***.***> Cc: Dan Streufert ***@***.***>; Author ***@***.***> Subject: Re: [kotx/render] Not all files listed (Issue #39) We could add a ...see more... link at the bottom of the page if there are more objects to be returned. This would add a ?cursor=... to the url, which would then return all results after the cursor. You could then list all objects similarly to this method in cloudflare docs<https://developers.cloudflare.com/r2/api/workers/workers-api-reference/#r2listoptions>. The question is if you need a configurable limit of objects per page (for example, so you can limit results to 50 per page), and/or an option for listing all files in a single request if the number of subrequests doesn't go over 50 on free, or 1k on paid. Let me know what works for your needs, I can try to add it in the next few days. — Reply to this email directly, view it on GitHub<#39 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFCEOH4TD5OWZYNYNTAIQP3YQBWMTAVCNFSM6AAAAABCH7KUFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBXGI2DGNZRGI>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[dstreufert]

Even dirs with 255 subdirectories are only showing the first 10-15…

…

________________________________ From: Dan Streufert ***@***.***> Sent: Tuesday, January 23, 2024 7:28:12 PM To: kotx/render ***@***.***>; kotx/render ***@***.***> Cc: Author ***@***.***> Subject: Re: [kotx/render] Not all files listed (Issue #39) I think “see more” and/or option for “unlimited” files would be great!

________________________________ From: Kot C ***@***.***> Sent: Tuesday, January 23, 2024 7:26:17 PM To: kotx/render ***@***.***> Cc: Dan Streufert ***@***.***>; Author ***@***.***> Subject: Re: [kotx/render] Not all files listed (Issue #39) We could add a ...see more... link at the bottom of the page if there are more objects to be returned. This would add a ?cursor=... to the url, which would then return all results after the cursor. You could then list all objects similarly to this method in cloudflare docs<https://developers.cloudflare.com/r2/api/workers/workers-api-reference/#r2listoptions>. The question is if you need a configurable limit of objects per page (for example, so you can limit results to 50 per page), and/or an option for listing all files in a single request if the number of subrequests doesn't go over 50 on free, or 1k on paid. Let me know what works for your needs, I can try to add it in the next few days. — Reply to this email directly, view it on GitHub<#39 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFCEOH4TD5OWZYNYNTAIQP3YQBWMTAVCNFSM6AAAAABCH7KUFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBXGI2DGNZRGI>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[kotx]

Just pushed a release (1.4.0) supporting pagination in page listings with ?cursor= as a query.

Even dirs with 255 subdirectories are only showing the first 10-15…

This may be a R2 API limitation. What I think is happening here is you have files in those subdirectories, reaching the listing limit of 1000 so there are only 10-15 unique /-delimited prefixes (=subdirectories).