feat(usage): streaming recursive directory usage API#346
Open
pengpeng wants to merge 3 commits into
Open
Conversation
Add pkg/hertz/biz/handler/stream with a small Writer that owns the NDJSON response framing (content-type, per-line flush, terminal _done / _error lines). Rewire archive EntriesMethod to use it so the upcoming dir-usage endpoint can reuse the same code and wire protocol. Co-authored-by: Cursor <cursoragent@cursor.com>
Add DirUsage(ctx, contextArgs, emit) to base.Execute and implement it across drivers, mirroring archive's synchronous walk-with-callback style: - posix: filepath.WalkDir summing regular files, throttled progress; cache/external delegate to posix. - cloud: rclone operations/size for a single server-side recursive call (new rclone GetFilesUsage returning count+bytes). - sync: repo root uses Seafile size/file_count fast path, files resolve size from the parent listing, subdirs walk level-by-level via getFiles. emit reports running totals so the HTTP layer can stream progress; an emit error (client gone) or ctx cancellation aborts the walk. Co-authored-by: Cursor <cursoragent@cursor.com>
Add the usage handler + router and register the route. UsageMethod
resolves the file param, gates on ActionRead, then streams DirUsage via
the shared NDJSON writer: {"count","size"} progress lines followed by a
terminal {"_done":true,"count","size"} (or {"_error","code"}). The wire
protocol matches archive entries.
Co-authored-by: Cursor <cursoragent@cursor.com>
68d8978 to
07aceec
Compare
Member
Author
|
Heads-up on the red
This PR's own packages (drivers / stream / usage handler+router) build and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a streaming
GET /api/usage/*pathendpoint that reports recursive file count and total byte size for a directory (and(1, size)for a single file), reusing the archive entries NDJSON streaming code and wire protocol.pkg/hertz/biz/handler/stream(Emit/Done/Fail, owning content-type, per-line flush, and the terminal{"_done":...}/{"_error":...,"code":...}framing). Rewired archiveEntriesMethodto use it so both endpoints share one code path.DirUsage(ctx, contextArgs, emit)tobase.Execute, mirroring archive's synchronous walk-with-callback style:filepath.WalkDirsumming regular files with throttled progress; cache/external delegate.GetFilesUsageviaoperations/size(single server-side recursive call returning count+bytes).size/file_countfast path, files resolve size from the parent listing, subdirs walk level-by-level viagetFiles.ResolveFileHandler,GateonActionRead, and the shared stream writer), registered inregister.go.Wire protocol: zero or more
{"count":N,"size":B}progress lines, then a terminal{"_done":true,"count":N,"size":B}(or{"_error":"...","code":"..."}). The client closing the connection cancels the walk viactx.The matching frontend (file attributes dialog showing live size + item count) lives in the Olares repo.
Test plan
du-style sum of regular filescount=1and correct sizeoperations/sizefile_count/size; subdir recurses correctlycode: "canceled")/entriesendpoint still streams unchanged (shared writer regression)Made with Cursor