Bug Report - Large listings/transfers (~3k objects) hang via Cloudflare Worker S3 proxy; small directories work

[diegoripley]

Setup

• Multistore deployed at: https://s3.labs.dataforcanada.org
• rclone setup as such

Configuration

• wrangler.dataforcanada.toml:

compatibility_date = "2024-11-11"
main = "build/worker/shim.mjs"
name = "dataforcanada-s3-labs"

[build]
command = "cargo install worker-build && worker-build --release"

[vars]
VIRTUAL_HOST_DOMAIN = "s3.labs.dataforcanada.org"

# ── Custom Domain ────────────────────────────────────────────────────
# Routes the worker to s3.dataforcanada.org.
# Requires the domain to be added to your Cloudflare account.
[[routes]]
pattern = "s3.labs.dataforcanada.org"
custom_domain = true

# ── Proxy Configuration ─────────────────────────────────────────────
[vars.PROXY_CONFIG]

# dataforcanada bucket — proxies Source Cooperative's public S3 bucket.
# Anonymous read access for everyone; authenticated writes require proxy credentials.
[[vars.PROXY_CONFIG.buckets]]
allowed_roles = []
anonymous_access = true
backend_type = "s3"
backend_prefix = "dataforcanada/"
name = "dataforcanada"

[vars.PROXY_CONFIG.buckets.backend_options]
bucket_name = "us-west-2.opendata.source.coop"
endpoint = "https://s3.us-west-2.amazonaws.com"
region = "us-west-2"
access_key_id = "CHANGEME"
secret_access_key = "CHANGEME"
skip_signature = "true"

# ── Placement & Observability ────────────────────────────────────────
[placement]
mode = "smart"

[observability]
enabled = true

Observed behavior

• When running rclone on a directory with a low number of files, downloads work. Command used:

rclone copy --progress dataforcanada-s3:dataforcanada/d4c-datapkg-foundation/ .

• When downloading from a folder with approximately 3,000 files, it hangs. Command to replicate:

rclone copy --progress dataforcanada-s3:dataforcanada/d4c-datapkg-orthoimagery/archive/ca-qc_government_and_municipalities_of_quebec-2026A000224_d4c-datapkg-orthoimagery_orthorectified_imagery_from_quebec/ .