perf(upload): WebCrypto chunk encryption + resilient part retry - #1
Merged
Conversation
…noble encryptFileChunk used @noble/ciphers gcm() (~70 MiB/s), which caps upload throughput on any reasonably fast uplink since encryption is serialized with the part PUT. Switch the per-chunk encryption to Node WebCrypto (subtle.encrypt AES-GCM, ~5 GiB/s). The output is ciphertext||tag with the same 16-byte tag, so ciphertexts, the SHA-1 digests Backblaze checks, and the HMAC upload proof are byte-for-byte unchanged. @noble stays for the once-per-file key wrap and filename metadata, and for all decryption (which the server and apps also do with @noble). encryptFileChunk is now async; its only caller (encryptChunkWithDigest) was already async. Tests: the round-trip test now decrypts the WebCrypto output with @noble, plus a new test asserts WebCrypto and @noble produce identical bytes for the same key/nonce/plaintext, and an empty-chunk case. 38/38 pass.
… upload putPartToBackblaze retried only on an expired token (401/403); any 5xx, timeout, or network blip threw and aborted the entire upload with no resume, so a single transient failure forced a multi-gigabyte multipart restart from byte zero. Add a bounded retry (up to 3 attempts, exponential backoff, a fresh part URL per retry) for 5xx / 408 / 429 / network errors. Part uploads are idempotent by (part number, SHA-1), so a retry safely overwrites the same part. The existing expired-token refresh path is unchanged (it retries immediately without consuming a backoff slot). Tests: a new case drives a 503 on part 1 and asserts it is retried with a refreshed URL and the upload completes; the existing token-refresh flow test still passes. 38/38 pass.
This was referenced Jul 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent improvements to the
sf push/sf syncupload path, one commit each.1. Encrypt chunks with WebCrypto AES-GCM (~76× faster)
encryptFileChunkused@noble/ciphersgcm()at ~70 MiB/s. Because encryption is serialized with each part PUT, that caps upload throughput on any fast uplink. Switched the per-chunk file encryption to Node WebCrypto (subtle.encryptAES-GCM), measured at ~5.3 GiB/s.The output is
ciphertext||tagwith the same 16-byte tag, so ciphertexts, the SHA-1 digests Backblaze checks, and the HMAC upload proof are byte-for-byte identical.@noblestays for the once-per-file key wrap + filename metadata and for all decryption (which the server and apps also do with@noble).Proof: the round-trip test decrypts the WebCrypto output with
@noble, and a new test asserts WebCrypto and@nobleproduce identical bytes for the same key/nonce/plaintext (plus an empty-chunk case).2. Retry a failed part instead of aborting the whole upload
putPartToBackblazeretried only on an expired token (401/403); any 5xx, timeout, or network blip threw and aborted the entire upload — a multi-gigabyte multipart would restart from byte zero. Added a bounded retry (up to 3 attempts, exponential backoff, a fresh part URL per retry) for 5xx / 408 / 429 / network errors. Part uploads are idempotent by (part number, SHA-1), so a retry safely overwrites the same part. The existing expired-token refresh path is unchanged.Proof: a new test drives a 503 on part 1 and asserts it is retried with a refreshed URL and the upload completes; the existing token-refresh flow test still passes.
38/38 tests pass.
Not included (follow-up)
Pipelining/concurrent part uploads (2–4 parts in flight) is the remaining throughput lever but is a larger change — it needs multiple part URLs, interleaved retry handling, and a higher memory ceiling — on a multipart path that has not yet been run against production with a real large file. Better done deliberately with its own testing.