Skip to content

feat(usage): streaming recursive directory usage API#346

Open
pengpeng wants to merge 3 commits into
mainfrom
feat/dir-usage-streaming-api
Open

feat(usage): streaming recursive directory usage API#346
pengpeng wants to merge 3 commits into
mainfrom
feat/dir-usage-streaming-api

Conversation

@pengpeng

@pengpeng pengpeng commented Jun 5, 2026

Copy link
Copy Markdown
Member

Summary

Adds a streaming GET /api/usage/*path endpoint that reports recursive file count and total byte size for a directory (and (1, size) for a single file), reusing the archive entries NDJSON streaming code and wire protocol.

  • Shared NDJSON writer: extracted pkg/hertz/biz/handler/stream (Emit / Done / Fail, owning content-type, per-line flush, and the terminal {"_done":...} / {"_error":...,"code":...} framing). Rewired archive EntriesMethod to use it so both endpoints share one code path.
  • Driver interface: added DirUsage(ctx, contextArgs, emit) to base.Execute, mirroring archive's synchronous walk-with-callback style:
    • posix: filepath.WalkDir summing regular files with throttled progress; cache/external delegate.
    • cloud: new rclone GetFilesUsage via operations/size (single server-side recursive call returning count+bytes).
    • sync: repo root uses Seafile size/file_count fast path, files resolve size from the parent listing, subdirs walk level-by-level via getFiles.
  • Endpoint: new usage handler + router (reusing ResolveFileHandler, Gate on ActionRead, and the shared stream writer), registered in register.go.

Wire protocol: zero or more {"count":N,"size":B} progress lines, then a terminal {"_done":true,"count":N,"size":B} (or {"_error":"...","code":"..."}). The client closing the connection cancels the walk via ctx.

The matching frontend (file attributes dialog showing live size + item count) lives in the Olares repo.

Test plan

  • posix dir: streams incrementing count/size, terminal totals match du-style sum of regular files
  • posix single file: returns count=1 and correct size
  • cloud (google/s3/tencent/dropbox) dir: returns count+bytes from operations/size
  • sync repo root: fast path returns repo file_count/size; subdir recurses correctly
  • client disconnect mid-stream cancels the walk (code: "canceled")
  • permission denied returns 403 before streaming starts
  • archive /entries endpoint still streams unchanged (shared writer regression)

Made with Cursor

pengpeng and others added 3 commits June 5, 2026 23:38
Add pkg/hertz/biz/handler/stream with a small Writer that owns the
NDJSON response framing (content-type, per-line flush, terminal
_done / _error lines). Rewire archive EntriesMethod to use it so the
upcoming dir-usage endpoint can reuse the same code and wire protocol.

Co-authored-by: Cursor <cursoragent@cursor.com>
Add DirUsage(ctx, contextArgs, emit) to base.Execute and implement it
across drivers, mirroring archive's synchronous walk-with-callback style:
- posix: filepath.WalkDir summing regular files, throttled progress;
  cache/external delegate to posix.
- cloud: rclone operations/size for a single server-side recursive call
  (new rclone GetFilesUsage returning count+bytes).
- sync: repo root uses Seafile size/file_count fast path, files resolve
  size from the parent listing, subdirs walk level-by-level via getFiles.

emit reports running totals so the HTTP layer can stream progress; an
emit error (client gone) or ctx cancellation aborts the walk.

Co-authored-by: Cursor <cursoragent@cursor.com>
Add the usage handler + router and register the route. UsageMethod
resolves the file param, gates on ActionRead, then streams DirUsage via
the shared NDJSON writer: {"count","size"} progress lines followed by a
terminal {"_done":true,"count","size"} (or {"_error","code"}). The wire
protocol matches archive entries.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pengpeng pengpeng force-pushed the feat/dir-usage-streaming-api branch from 68d8978 to 07aceec Compare June 5, 2026 15:38
@pengpeng

pengpeng commented Jun 5, 2026

Copy link
Copy Markdown
Member Author

Heads-up on the red test check: it is a pre-existing failure inherited from main, not caused by this PR.

  • Failing test: TestHydrateExternalRootItemMetadataMountedValid (pkg/drivers/posix/posix/posix_list_test.go:117), mod time mismatch: got=0001-01-01 ... expected=2026-06-01.
  • It goes through hydrateExternalRootItemMetadata -> statMountedExternalRootItem -> externalMountGuard.run(...), which is unrelated to the DirUsage / usage-streaming changes here.
  • The same test already fails on main: the CI run for the PR Decouple external mount probing from guarded operations #344 merge (Decouple external mount probing from guarded operations, commit ea294d4) failed with the identical test (run 27017151726). That change made the guard return "probe pending" for an un-probed mount, so statMountedExternalRootItem no longer runs the stat and ModTime stays zero, but posix_list_test.go was not updated to the new model.
  • This branch is already rebased on the latest main (including ea294d4); the failure persists because the test itself is stale.

This PR's own packages (drivers / stream / usage handler+router) build and go vet clean under GOOS=linux. Suggest fixing TestHydrateExternalRootItemMetadataMountedValid on main separately (owned by the #344 change); this PR will go green once that lands and we rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant