Skip to content

fix: add limit to nix copy to avoid hitting s3 upload limits#2039

Merged
samrose merged 1 commit intodevelopfrom
max-jobs-copy
Feb 7, 2026
Merged

fix: add limit to nix copy to avoid hitting s3 upload limits#2039
samrose merged 1 commit intodevelopfrom
max-jobs-copy

Conversation

@samrose
Copy link
Copy Markdown
Collaborator

@samrose samrose commented Feb 7, 2026

in PR ci runs like https://github.com/supabase/postgres/actions/runs/21781372775/job/62845918939?pr=2028 we hit s3 upload limits since no max-jobs was set, nix copy has no limit.

We have multiple CI jobs potentially uploading concurrently to the same /nar/ prefix, a conservative per-job limit would be:

  • 5 concurrent uploads (nix copy --max-jobs 5)
  • Or roughly 50 requests/second per job

This should help with the fact that we have potential to launch 5-10 CI runs at once.

@samrose samrose marked this pull request as ready for review February 7, 2026 14:55
@samrose samrose requested review from a team as code owners February 7, 2026 14:55
Copy link
Copy Markdown
Contributor

@valigula valigula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@samrose samrose added this pull request to the merge queue Feb 7, 2026
Merged via the queue into develop with commit 6a6be67 Feb 7, 2026
28 checks passed
@samrose samrose deleted the max-jobs-copy branch February 7, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants