Skip to content

perf(pipeline): use streaming io.Copy for content passthrough file copying#606

Merged
zeroedin merged 1 commit into
mainfrom
spec/issue-586-streaming-passthrough
May 13, 2026
Merged

perf(pipeline): use streaming io.Copy for content passthrough file copying#606
zeroedin merged 1 commit into
mainfrom
spec/issue-586-streaming-passthrough

Conversation

@zeroedin

Copy link
Copy Markdown
Owner

Summary

  • Replace os.ReadFile/os.WriteFile with fileutil.CopyFile for content-colocated passthrough files
  • Eliminates heap pressure from large binary assets (images, videos, fonts) during build
  • fileutil.CopyFile already handles MkdirAll, streaming io.Copy with 32KB buffer, source permission preservation, and deferred close with error propagation

Verification

Existing test at build_test.go:1069 ("build succeeds with passthrough mappings") exercises this path with real file I/O via BuildWithContent. All 144 passing tests remain green; the 1 pre-existing failure is issue #581 (unrelated).

Closes #586

🤖 Generated with Claude Code

…pying

Replace os.ReadFile/os.WriteFile with fileutil.CopyFile for
content-colocated passthrough files. Avoids holding large binary
assets entirely in memory during build. (issue #586)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces memory usage during builds by switching content-colocated passthrough file copying (non-content files found under content/) from whole-file read/write to a streaming copy helper already used elsewhere in the pipeline.

Changes:

  • Replace os.ReadFile/os.WriteFile passthrough copying with fileutil.CopyFile in the build pipeline.
  • Add internal/fileutil import to support the new streaming copy path.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@zeroedin

Copy link
Copy Markdown
Owner Author

Code Review — 5 reviewers, 0 blocking findings

Verdict: Approve

Correctly replaces inline os.ReadFile/os.WriteFile with the existing fileutil.CopyFile utility. Net -8 lines. Eliminates heap allocation for large binary passthroughs. Matches the pattern already used by static copy and asset copy. Permission change from hardcoded 0644 to source-preserving srcInfo.Mode() is consistent with the rest of the codebase.

Advisory findings (no merge blockers)

# File Issue Reviewer Conf Pre-existing
1 internal/fileutil/copy.go CopyFile has 5 error branches + deferred close, zero direct unit tests testing 75 Yes
2 internal/pipeline/build_test.go Passthrough tests verify path list, never verify file content on disk testing 80 Yes
3 internal/pipeline/build.go:839 Sequential copy loop — codebase has copyDirConcurrent for same pattern performance 68 Yes
4 internal/pipeline/build.go:842 Permission change (0644 → srcInfo.Mode()) untested correctness, testing 55 No

All findings are advisory. #1-#3 are pre-existing on main. #4 is a behavioral change but aligns with how static and asset copies already work — consistent, just undocumented.

Clean passes

  • Correctness: Deferred close error propagation correct. Symlink behavior matches old code. Error wrapping improved (MkdirAll now includes relPath).
  • Maintainability: Clean extraction to existing utility. No remaining duplicate copy patterns in build.go.
  • Project standards: Conventional commits, feature branch, single-concern scope.

Reviewed by: correctness, maintainability, performance, testing, project-standards
Run ID: 20260512-212915-784adefd

@zeroedin
zeroedin merged commit 59883fc into main May 13, 2026
4 checks passed

@zeroedin zeroedin left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review #1 — 6 reviewers dispatched, 6 returned

Verdict: Ready to merge

Clean -8 net line reduction. Replaces a manual 10-line os.ReadFile/os.WriteFile pattern with the project's existing fileutil.CopyFile streaming utility. Eliminates heap allocation for large binary passthrough files. 0 actionable findings after synthesis.

Scope Note

3 of 6 reviewers explored beyond the PR diff (reading files on the branch not changed by this PR). 10 out-of-scope findings discarded. GitHub confirms 1 file changed (+2/-10).

Findings

0 actionable findings.

1 advisory (P2): Branch uses spec/ prefix (spec/issue-586-streaming-passthrough) but this is a perf implementation PR, not a specification PR.

Residual Risks

  1. Permission semantics change — Output passthrough files now inherit source file permissions instead of hardcoded 0644. This aligns with how static.CopyPassthroughWithValidation and assets.CopyAssets already work via the same CopyFile — making behavior more consistent across copy paths. PR description doesn't mention this change.

Testing Gaps (advisory)

  1. Permission preservation untested — No test distinguishes 0644 from source-preserving behavior. Existing passthrough test (build_test.go:1069) creates source files with 0644, making the change invisible.

Pre-Existing Issues (not introduced by this PR)

  • fileutil.CopyFile has no unit tests (5 error branches, used by 3 copy paths)
  • Sequential copy loop (static path already uses copyDirConcurrent with runtime.NumCPU() workers)
  • No fsync before close in CopyFile
  • Truncated destination on error (no atomic write pattern)

Correctness Verification

  • CopyFile already battle-tested by static and assets copy paths on main
  • Handles MkdirAll, streaming io.Copy, source permission preservation, deferred close with error propagation
  • Error wrapping correct — relPath now included in all error messages (improvement over old MkdirAll message)
  • Loop structure, path computation, and reportEndStage() on error all preserved

6 reviewers (correctness, testing, maintainability, project-standards, performance, reliability) · 10 findings discarded (out-of-scope) · 4 separated (pre-existing) · 2 demoted (mode-aware) · 0 suppressed (confidence gate)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(pipeline): use streaming io.Copy for content passthrough file copying

2 participants