Perf/bundle parallelization - #164
Conversation
Replace the two-pass file discovery (GetAllFiles followed by GetFilteredFiles) with GetFilteredFilesSingleWalk, which traverses the project tree once and prunes ignored directories (e.g. node_modules, .git) instead of walking the whole tree and glob-matching every file underneath them. That traversal was the dominant cost on real projects. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Process files with a pool of workers when creating a bundle. The per-file work (stat, read, hash) is I/O-bound and independent across files, so oversubscribing the CPUs keeps the disk busy while syscalls block — this matters most on Windows, where per-file syscalls are comparatively expensive. Results are merged under a mutex, and the lazy filter load is guarded so concurrent workers fetch filters exactly once. Each file is opened a single time and stat'd via the open handle to avoid a redundant filesystem lookup. Also deduplicate the UTF-8 conversion in BundleFileFrom: the content is now converted once and reused for both the hash and the bundle content, via the new util.HashContent helper, instead of converting twice. Add a benchmark covering bundle creation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The committed replace directive pointed at a nonexistent local path (/Users/ianzink/git/gaf-v0.3.2-local), which broke `go mod tidy`, build, unit tests, and the Snyk SCA/license checks in CI. The parallelization work also depends on `GetFilteredFilesSingleWalk`, which currently only exists on the companion GAF branch (snyk/go-application-framework#657). Repoint the replace directive at that GAF commit (github.com/z4ce/go-application-framework@5ac64af, pseudo-version v0.0.0-20260716054810-5ac64afcd596) and update go.sum so the module graph resolves and CI can build/test the change. NOTE: this is a temporary measure to unblock CI while both PRs are drafts. Before merge, GAF #657 should land and this should be repinned to a real snyk/go-application-framework version with the replace removed.
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
PR Reviewer Guide 🔍
|
Description
Provide description of this PR and changes, if linked Jira ticket doesn't cover it in full.
Checklist
🚨After having merged, please update the
snyk-lsand CLI go.mod to pull in latest client.