Skip to content

perf(upload): bounded-concurrency multipart uploader - #3

Merged
chogarcia merged 1 commit into
mainfrom
perf/multipart-concurrency
Jul 3, 2026
Merged

perf(upload): bounded-concurrency multipart uploader#3
chogarcia merged 1 commit into
mainfrom
perf/multipart-concurrency

Conversation

@chogarcia

Copy link
Copy Markdown
Contributor

What

Upload multipart parts with bounded concurrency instead of strictly one at a time. Completes the deferred "concurrent parts" stretch from #1 (per-part 5xx retry) and #2 (argon2id).

The old loop encrypted and PUT one part at a time, so the network sat idle during each chunk's encrypt and the CPU sat idle during each PUT — a multi-part file transferred at the speed of a single connection.

How

  • Run up to K parts at once. Default 3; override with SF_UPLOAD_CONCURRENCY (clamped 1..6). SF_UPLOAD_CONCURRENCY=1 reproduces the exact old serial behaviour.
  • URL pool: Backblaze needs one b2_upload_part URL per concurrent connection. K parts draw from a small pool seeded with the session URL, grown on demand, and returned after each success for reuse. A URL whose part failed is dropped, not reused.
  • Backpressure: at most K parts in flight, gated by Promise.race over the in-flight set (which is provably non-empty when raced).
  • Ordering integrity (the crux): part SHA-1s are keyed by chunkIndex, not finish order, and the finalize manifest is rebuilt densely in strict chunk order. So b2_finish_large_file's ordered part list and the multipart ciphertextHash (SHA-1 over the concatenated 20-byte part digests) are byte-identical to the serial version even though parts now land in network order.
  • Unchanged: per-chunk nonce (prefix||chunkIndex), first-chunk-only upload proof, and the ciphertext bytes.
  • Failure semantics: any part error sets a sticky first-error, drains in-flight work, and rethrows — no partial file is ever finalized; a missing slot is a hard error.
  • Memory ceiling ≈ 2 · K · chunkSize (K plaintext + K ciphertext chunks resident).

Tests

40/40 pass (node --test).

  • The two existing multipart tests are pinned to SF_UPLOAD_CONCURRENCY=1 (they assert arrival order + a single URL refresh — serial-only invariants).
  • New: runs 5 fully concurrent parts, forces completion order 5,4,3,2,1, and asserts the finalized partSha1Array is back in chunk order (not completion order), the ciphertextHash matches the chunk-order digests, one URL is drawn per connection, and the reassembled/decrypted plaintext equals the original file.

Adversarially reviewed (3 independent lenses: ordering integrity, data races/buffer aliasing, failure-abort) — zero confirmed findings. Verified that readFileChunks yields independent copies (new Uint8Array(buffer.subarray(...)) allocates a fresh backing buffer), so concurrent parts never alias the reused read buffer.

Verify (post-merge)

Needs one real sf push of a > 5 MiB file (multi-part) against prod to confirm end-to-end, plus a SF_UPLOAD_CONCURRENCY=1 run to confirm the serial path still works.

The multipart uploader encrypted and PUT one part at a time, so the
network sat idle during each chunk's encrypt and the CPU sat idle during
each PUT. On a multi-part file that serialises the whole transfer at the
speed of one connection.

Run up to K parts at once (default 3, override with SF_UPLOAD_CONCURRENCY,
clamped 1..6). Backblaze needs one upload URL per concurrent connection,
so K parts draw URLs from a small pool that is seeded with the session URL,
grown on demand, and returned after each successful part for reuse; a URL
whose part failed is dropped rather than reused. Backpressure holds at most
K parts in flight via Promise.race over the in-flight set.

Because parts now finish in network order, part SHA-1s are keyed by
chunkIndex (not push order) and the finalize manifest is rebuilt densely
in strict chunk order, so b2_finish_large_file and the multipart
ciphertextHash are byte-identical to the serial version. The first-chunk
upload proof, per-chunk nonce, and ciphertext are unchanged. Any part
failure sets a sticky first-error, drains in-flight work, and rethrows —
no partial file is ever finalized; a missing slot is a hard error.

SF_UPLOAD_CONCURRENCY=1 reproduces the exact serial behaviour. The two
existing multipart tests are pinned to =1 (they assert arrival order and a
single URL refresh); a new test runs 5 fully concurrent parts, forces
completion order 5..1, and asserts the finalized manifest is back in chunk
order, the ciphertextHash matches, and the reassembled plaintext equals the
original file. 40/40 tests pass.
@chogarcia
chogarcia merged commit b94a0f4 into main Jul 3, 2026
1 check passed
@chogarcia
chogarcia deleted the perf/multipart-concurrency branch July 3, 2026 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant