perf(upload): bounded-concurrency multipart uploader - #3
Merged
Conversation
The multipart uploader encrypted and PUT one part at a time, so the network sat idle during each chunk's encrypt and the CPU sat idle during each PUT. On a multi-part file that serialises the whole transfer at the speed of one connection. Run up to K parts at once (default 3, override with SF_UPLOAD_CONCURRENCY, clamped 1..6). Backblaze needs one upload URL per concurrent connection, so K parts draw URLs from a small pool that is seeded with the session URL, grown on demand, and returned after each successful part for reuse; a URL whose part failed is dropped rather than reused. Backpressure holds at most K parts in flight via Promise.race over the in-flight set. Because parts now finish in network order, part SHA-1s are keyed by chunkIndex (not push order) and the finalize manifest is rebuilt densely in strict chunk order, so b2_finish_large_file and the multipart ciphertextHash are byte-identical to the serial version. The first-chunk upload proof, per-chunk nonce, and ciphertext are unchanged. Any part failure sets a sticky first-error, drains in-flight work, and rethrows — no partial file is ever finalized; a missing slot is a hard error. SF_UPLOAD_CONCURRENCY=1 reproduces the exact serial behaviour. The two existing multipart tests are pinned to =1 (they assert arrival order and a single URL refresh); a new test runs 5 fully concurrent parts, forces completion order 5..1, and asserts the finalized manifest is back in chunk order, the ciphertextHash matches, and the reassembled plaintext equals the original file. 40/40 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Upload multipart parts with bounded concurrency instead of strictly one at a time. Completes the deferred "concurrent parts" stretch from #1 (per-part 5xx retry) and #2 (argon2id).
The old loop encrypted and PUT one part at a time, so the network sat idle during each chunk's encrypt and the CPU sat idle during each PUT — a multi-part file transferred at the speed of a single connection.
How
SF_UPLOAD_CONCURRENCY(clamped 1..6).SF_UPLOAD_CONCURRENCY=1reproduces the exact old serial behaviour.b2_upload_partURL per concurrent connection. K parts draw from a small pool seeded with the session URL, grown on demand, and returned after each success for reuse. A URL whose part failed is dropped, not reused.Promise.raceover the in-flight set (which is provably non-empty when raced).chunkIndex, not finish order, and the finalize manifest is rebuilt densely in strict chunk order. Sob2_finish_large_file's ordered part list and the multipartciphertextHash(SHA-1 over the concatenated 20-byte part digests) are byte-identical to the serial version even though parts now land in network order.prefix||chunkIndex), first-chunk-only upload proof, and the ciphertext bytes.2 · K · chunkSize(K plaintext + K ciphertext chunks resident).Tests
40/40pass (node --test).SF_UPLOAD_CONCURRENCY=1(they assert arrival order + a single URL refresh — serial-only invariants).5,4,3,2,1, and asserts the finalizedpartSha1Arrayis back in chunk order (not completion order), theciphertextHashmatches the chunk-order digests, one URL is drawn per connection, and the reassembled/decrypted plaintext equals the original file.Adversarially reviewed (3 independent lenses: ordering integrity, data races/buffer aliasing, failure-abort) — zero confirmed findings. Verified that
readFileChunksyields independent copies (new Uint8Array(buffer.subarray(...))allocates a fresh backing buffer), so concurrent parts never alias the reused read buffer.Verify (post-merge)
Needs one real
sf pushof a > 5 MiB file (multi-part) against prod to confirm end-to-end, plus aSF_UPLOAD_CONCURRENCY=1run to confirm the serial path still works.