Commit cca773f
authored
feat(fs): add pipelined multipart upload (#2723)
* feat(multipart): add chunk reassembly window
- Reassemble concurrently uploaded chunks into a sequential stream through a ring file, bounding disk usage to slots*chunkSize per session
- Park writers up to a deadline when their slot is busy instead of rejecting instantly, so flow control does not surface as connection errors in browsers
- Record a per-chunk CRC32 table for re-fill verification and keep it readable after close
- Propagate cancellation to blocked readers via CloseWithError so drivers treat aborts like canceled requests
- Cover ordering, backpressure, idempotent resends and close/abort wake-ups with race-enabled tests
* feat(multipart): add pipelined upload session manager
- Start the driver upload at session init over a sequential stream backed by the window, so client-to-server and server-to-storage transfers run concurrently
- Attach client-provided hashes to the stream so drivers can attempt rapid upload before any chunk arrives, and absorb chunks racing pipeline completion idempotently
- Keep only metadata and chunk CRCs after a failed attempt: re-sending chunk 0 re-fills a fresh window, and content changes between attempts are rejected
- Resume receiving sessions only when client hashes prove the same file; failed_retriable sessions resume unconditionally
- Reclaim sessions with a sliding-TTL GC and sweep orphaned ring files at startup
- Cover the state machine with race-enabled tests over a stubbed storage layer
* feat(setting): add multipart upload settings
- Add multipart_enabled and multipart_chunk_size (MB) as public traffic settings
- Validate the chunk size on save (integer within 1-90) via the setting item hook
* feat(server): add multipart upload API
- Add /api/fs/multipart init/chunk/complete/status/abort endpoints with headers aligned with /fs/put
- Gate init behind the FsUp permission checks and reuse the client upload rate limiter for chunk uploads
- Drain the request body before answering chunk requests on every path, so browsers do not see early responses as network errors
- Start the multipart session GC when the router is initialized to reclaim ring files orphaned by a previous run
* refactor(multipart): remove unused overwrite session field
- The overwrite flag was stored on the session but never read: the handler
performs the pre-check and op.Put owns the overwrite semantics
* feat(setting): allow any positive multipart chunk size
- Validate the setting as a positive integer only; self-hosted admins decide
the ceiling themselves instead of an arbitrary 90MB cap
- Keep clamping the client-suggested X-Chunk-Size to the admin value: the
server buffers a window of 8 chunks per session, so an unbounded client
suggestion would translate directly into server-side disk usage
* refactor(server): simplify multipart chunk size clamp
- Fold the two-step clamp into one branch: the ceiling is already floored,
so a client suggestion just lowers the size with a 1MB floor1 parent a4ae2ac commit cca773f
9 files changed
Lines changed: 2418 additions & 0 deletions
File tree
- internal
- bootstrap/data
- conf
- multipart
- op
- server
- handles
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
249 | 249 | | |
250 | 250 | | |
251 | 251 | | |
| 252 | + | |
| 253 | + | |
252 | 254 | | |
253 | 255 | | |
254 | 256 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
164 | 164 | | |
165 | 165 | | |
166 | 166 | | |
| 167 | + | |
| 168 | + | |
167 | 169 | | |
168 | 170 | | |
169 | 171 | | |
| |||
0 commit comments