Speed up and stabilize local file upload - #96
Merged
Conversation
uploadChunk() was opening, writing, and closing the destination file on every single chunk - real per-chunk overhead (open/close syscalls, plus whatever flush happens on close), especially costly against the FAT-formatted USB target this writes to. For a ~1.2GB image at 3MB chunks, that's ~400 chunks each paying this cost. uploadMagicChunk already gets this right - it opens the file handle once (lazily, on the first chunk) and reuses it for the rest. This brings the regular upload path in line with that: uploadStart opens the file once and stores the handle on state.File, uploadChunk just writes to it, and uploadFinish (uploadCancel already did) closes it. Also fixes a pre-existing bug found along the way: the base64 decode error in uploadChunk was captured but never actually checked. One piece of a larger investigation into #75/#61 (slow/stalling uploads) - this addresses the server-side per-chunk overhead specifically; the client-side serialized, non-pipelined chunk loop and the base64/JSON wire overhead are separate, larger changes not included here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Drives upload_start -> ~22 upload_chunk calls -> upload_finish through the real handlers (not mocked), and checks the resulting file on disk matches the original payload byte-for-byte. Exercises the state.File handle kept open across chunks, so this would actually catch truncation, overwriting, or interleaving bugs - a build-only check can't. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
base64-encoding each chunk inflated the wire size by ~33% on an already WiFi-constrained upload path, and cost real CPU/JSON overhead on both ends. Chunks are now posted as raw octet-stream bodies instead - the client posts the File.slice() Blob directly, the server reads it straight off the request body. Separately, uploadLocalFile()'s request chain had no .catch() and no timeout at all, so a single dropped/hung request (this link has real multi-hundred-ms latency spikes and occasional dead spells) silently froze the whole upload forever with no feedback - see #61. Added a 20s per-chunk timeout and retry with exponential backoff (capped at 30s, up to 20 attempts) so a bad stretch gets ridden out instead. Chunk pipelining/concurrency was also tried as a further speed improvement, but reverted: this board's WiFi NIC and USB storage share a single USB hub with a single Transaction Translator (confirmed via the hub's own datasheet), so concurrent chunks caused real instability (a disk-write pileup requiring a board reboot, then persistent connection resets) rather than just being faster. Chunks stay serial, one at a time. Live-tested end to end: a full ~1.2GB image upload completed in 12m48s with no errors, faster than the pre-fix baseline (~20min) and with no hangs, on top of #91/#92/#93/#94. Closes #75
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
application/octet-stream) instead of base64-encoded inside JSON - removes ~33% wire-size inflation plus base64/JSON encode-decode overhead on both ends.uploadLocalFile()now has a 20s per-chunk timeout and retries with exponential backoff (up to 20 attempts, capped at 30s) instead of no timeout and no.catch()at all. Previously a single dropped/hung request silently froze the whole upload forever - see Sometimes the upload process stops during a file upload. #61.Investigated but reverted: chunk pipelining
Also tried sending several chunks concurrently (offset-based
WriteAtserver-side) to hide this link's real RTT. Reverted after it caused two separate live incidents:NET_RESET).Root cause: this board's USB hub (WiFi NIC + storage) shares a single Transaction Translator (confirmed via the CY7C65634 hub's datasheet), so it can't reliably handle simultaneous network-receive and disk-write transactions. Chunks stay serial, one at a time.
Testing
go test ./...-TestUploadChunkRoundTrippasses./dev/null-target diagnostic build - confirmed disk write was still a meaningful factor even on the newer/faster USB drive over a long sustained transfer, not just the network link.scpto/dev/null(~1.3 MB/s) - confirmed the remaining slowness is this link's real capacity, not protocol overhead from serial chunking.Closes #75