Skip to content

Commit cca773f

Browse files
authored
feat(fs): add pipelined multipart upload (#2723)
* feat(multipart): add chunk reassembly window - Reassemble concurrently uploaded chunks into a sequential stream through a ring file, bounding disk usage to slots*chunkSize per session - Park writers up to a deadline when their slot is busy instead of rejecting instantly, so flow control does not surface as connection errors in browsers - Record a per-chunk CRC32 table for re-fill verification and keep it readable after close - Propagate cancellation to blocked readers via CloseWithError so drivers treat aborts like canceled requests - Cover ordering, backpressure, idempotent resends and close/abort wake-ups with race-enabled tests * feat(multipart): add pipelined upload session manager - Start the driver upload at session init over a sequential stream backed by the window, so client-to-server and server-to-storage transfers run concurrently - Attach client-provided hashes to the stream so drivers can attempt rapid upload before any chunk arrives, and absorb chunks racing pipeline completion idempotently - Keep only metadata and chunk CRCs after a failed attempt: re-sending chunk 0 re-fills a fresh window, and content changes between attempts are rejected - Resume receiving sessions only when client hashes prove the same file; failed_retriable sessions resume unconditionally - Reclaim sessions with a sliding-TTL GC and sweep orphaned ring files at startup - Cover the state machine with race-enabled tests over a stubbed storage layer * feat(setting): add multipart upload settings - Add multipart_enabled and multipart_chunk_size (MB) as public traffic settings - Validate the chunk size on save (integer within 1-90) via the setting item hook * feat(server): add multipart upload API - Add /api/fs/multipart init/chunk/complete/status/abort endpoints with headers aligned with /fs/put - Gate init behind the FsUp permission checks and reuse the client upload rate limiter for chunk uploads - Drain the request body before answering chunk requests on every path, so browsers do not see early responses as network errors - Start the multipart session GC when the router is initialized to reclaim ring files orphaned by a previous run * refactor(multipart): remove unused overwrite session field - The overwrite flag was stored on the session but never read: the handler performs the pre-check and op.Put owns the overwrite semantics * feat(setting): allow any positive multipart chunk size - Validate the setting as a positive integer only; self-hosted admins decide the ceiling themselves instead of an arbitrary 90MB cap - Keep clamping the client-suggested X-Chunk-Size to the admin value: the server buffers a window of 8 chunks per session, so an unbounded client suggestion would translate directly into server-side disk usage * refactor(server): simplify multipart chunk size clamp - Fold the two-step clamp into one branch: the ceiling is already floored, so a client suggestion just lowers the size with a 1MB floor
1 parent a4ae2ac commit cca773f

9 files changed

Lines changed: 2418 additions & 0 deletions

File tree

internal/bootstrap/data/setting.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,8 @@ func InitialSettings() []model.SettingItem {
249249
{Key: conf.StreamMaxClientUploadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE},
250250
{Key: conf.StreamMaxServerDownloadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE},
251251
{Key: conf.StreamMaxServerUploadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE},
252+
{Key: conf.MultipartEnabled, Value: "true", Type: conf.TypeBool, Group: model.TRAFFIC, Flag: model.PUBLIC},
253+
{Key: conf.MultipartChunkSize, Value: "10", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PUBLIC, Help: `chunk size of multipart upload in MB (positive integer), keep it under your CDN's request body limit; each active session buffers up to 8 chunks on the server's disk`},
252254
}
253255
additionalSettingItems := tool.Tools.Items()
254256
// 固定顺序

internal/conf/const.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,8 @@ const (
164164
StreamMaxClientUploadSpeed = "max_client_upload_speed"
165165
StreamMaxServerDownloadSpeed = "max_server_download_speed"
166166
StreamMaxServerUploadSpeed = "max_server_upload_speed"
167+
MultipartEnabled = "multipart_enabled"
168+
MultipartChunkSize = "multipart_chunk_size"
167169
)
168170

169171
const (

0 commit comments

Comments
 (0)