Rate limit r2 bulk put

[vicb]

r2 bulk put is based on the Cloudflare API which is rate limited

Make sure we do not exceed the limit when bulk uploading to R2

---

• Tests
  • Tests included
  • Tests not necessary because: validated locally by bulk putting 5000 files
• Public documentation
  • Cloudflare docs PR(s):
  • Documentation not necessary because: experimental command
• Wrangler V3 Backport
  • Wrangler PR:
  • Not necessary because: not in v3

[changeset-bot]

🦋 Changeset detected

Latest commit: 9a2a516

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

Failed to automatically backport this PR's changes to Wrangler v3. Please manually create a PR targeting the v3-maintenance branch with your changes. Thank you for helping us keep Wrangler v3 supported!

Depending on your changes, running git rebase --onto v3-maintenance main vicb/r2-rate-limit might be a good starting point.

Notes:

• your PR branch should be named v3-backport-11367
• add the skip-v3-pr label to the current PR to stop this workflow from failing

[pkg-pr-new]

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@11367

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@11367

miniflare

npm i https://pkg.pr.new/miniflare@11367

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@11367

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@11367

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@11367

@cloudflare/vitest-pool-workers

npm i https://pkg.pr.new/@cloudflare/vitest-pool-workers@11367

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@11367

@cloudflare/workers-utils

npm i https://pkg.pr.new/@cloudflare/workers-utils@11367

wrangler

npm i https://pkg.pr.new/wrangler@11367

commit: 9a2a516

[petebacondarwin]

Nice solution.
Does this mean, though, that this approach is fundamentally constrained to be blocked for 5 mins on scenarios where there are more than 5000 files to be cached?

[vicb]

Does this mean, though, that this approach is fundamentally constrained to be blocked for 5 mins on scenarios where there are more than 5000 files to be cached?

yes, more than 1200 remote objects actually, unfortunately

(edit: still faster than 1 object/s without bulking but we probably need to improve that)