feat(cache): delta-based incremental SQLite analysis cache for Go by shivasurya · Pull Request #649 · shivasurya/code-pathfinder

shivasurya · 2026-04-07T15:42:13Z

Summary

Adds AnalysisCache backed by SQLite to speed up repeated Go call graph builds.

Cold → warm speedup: ~2m 39s → ~24s (6.6×) on 392 Go files / 39,743 call sites.

Enable with --enable-db-cache on resolution-report (scan/ci in PR-10).

Design

Which passes are cached:

Passes 2a/2b/3 (file-local): cached by content hash — always safe
Pass 4 (cross-file): delta-based — files are replayed from cache unless dirty
Pass 1: never cached — must run to detect what changed

Pass 4 delta invalidation (NeedsPass4Rerun): a file is dirty if:

No cache entry exists
Content hash changed
A resolved callee FQN was removed from the index
An unresolved call name matches a newly-added FQN

Warm files skip resolution entirely; cached edges replay directly into the call graph.

Per-table schema versioning — three constants (fileCacheVersion, functionIndexVersion, pass4Version) stored in the meta table. On version mismatch only the affected table is wiped; other tables stay warm. Project-root change wipes all.

Test coverage

analysis_cache_test.go: per-table version wipe, file-cache wipe, project-root change
go_builder_test.go: cold→warm round-trip, nil cache regression, dirty-file re-resolution

🤖 Generated with Claude Code

safedep · 2026-04-07T15:42:21Z

SafeDep Report Summary

Package Details

Package	Malware	Vulnerability	Risky License	Report
`github.com/dustin/go-humanize @ 1.0.1` _{sast-engine/go.mod}				🔗
`github.com/mattn/go-isatty @ 0.0.20` _{sast-engine/go.mod}				🔗
`github.com/ncruces/go-strftime @ 1.0.0` _{sast-engine/go.mod}				🔗
`github.com/remyoudompheng/bigfft @ 0.0.0-20230129092748-24d4a6f8daec` _{sast-engine/go.mod}				🔗
`modernc.org/libc @ 1.70.0` _{sast-engine/go.mod}				🔗
`modernc.org/mathutil @ 1.7.1` _{sast-engine/go.mod}				🔗
`modernc.org/memory @ 1.11.0` _{sast-engine/go.mod}				🔗
`modernc.org/sqlite @ 1.48.1` _{sast-engine/go.mod}				🔗

View complete scan results →

_{This report is generated by SafeDep Github App}

github-actions · 2026-04-07T15:42:55Z

Code Pathfinder Security Scan

No security issues detected.

Metric	Value
Files Scanned	26
Rules	205

_{Powered by Code Pathfinder}

codecov · 2026-04-07T15:48:14Z

Codecov Report

❌ Patch coverage is 81.67641% with 94 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.74%. Comparing base (40cc8e3) to head (e994560).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...t-engine/graph/callgraph/builder/analysis_cache.go	85.24%	19 Missing and 17 partials ⚠️
sast-engine/graph/callgraph/builder/go_builder.go	84.64%	25 Missing and 10 partials ⚠️
sast-engine/cmd/scan.go	7.69%	12 Missing ⚠️
sast-engine/cmd/resolution_report.go	28.57%	8 Missing and 2 partials ⚠️
sast-engine/cmd/serve.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #649      +/-   ##
==========================================
+ Coverage   84.45%   84.74%   +0.29%     
==========================================
  Files         163      165       +2     
  Lines       23594    24403     +809     
==========================================
+ Hits        19926    20680     +754     
- Misses       2920     2938      +18     
- Partials      748      785      +37

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

shivasurya · 2026-04-07T20:32:17Z

shivasurya · 2026-04-07T20:35:13Z

Merge activity

Apr 7, 8:35 PM UTC: A user started a stack merge that includes this pull request via Graphite.
Apr 7, 8:51 PM UTC: Graphite rebased this pull request as part of a merge.
Apr 7, 8:51 PM UTC: @shivasurya merged this pull request with Graphite.

…l graph Adds AnalysisCache backed by SQLite to speed up repeated scans. Cold run: ~2m 39s → warm run: ~24s (6.6×) on 392 Go files / 39,743 call sites. Key design: - Per-table schema versioning (fileCacheVersion, functionIndexVersion, pass4Version) so future upgrades wipe only the affected table, not the full cache - Pass 4 uses delta-based invalidation: after Pass 1 builds the FQN index, ComputeFunctionIndexDelta detects added/removed FQNs; NeedsPass4Rerun marks a file dirty if content hash changed, a resolved callee was removed, or an unresolved name matches a newly-added callee - Warm files skip resolution entirely; cached CachedPass4Edge rows are replayed directly into the call graph (Stage 2b) - All SQL calls use ...Context variants to satisfy noctx linter - 100% coverage on new logic paths Enable via --enable-db-cache flag on resolution-report command. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Exposes the incremental SQLite analysis cache (from PR-09) on the two most commonly used commands. The flag is experimental and opt-in. pathfinder scan --project . --rules ./rules --enable-db-cache pathfinder ci --project . --ruleset cpf/go --enable-db-cache Cache is opened per-project, errors are logged as warnings (non-fatal) so the flag degrades gracefully if the cache directory is not writable. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… wiring Add targeted error-path tests to bring analysis_cache.go and scan.go patch coverage above the codecov threshold: analysis_cache.go: - TestLoadFunctionIndex_ClosedDB: covers QueryContext error path - TestOpenAnalysisCache_UserCacheDirFallback: covers TempDir fallback - TestOpenAnalysisCache_MkdirError: covers MkdirAll failure path - TestSaveFunctionIndex_TableDropped: covers DELETE error in SaveFunctionIndex - TestSavePass4Results_TableDropped: covers PrepareContext error in SavePass4Results - TestOpenAnalysisCache_ProjectRootMismatch_WipesAll: full wipe on root change - TestOpenAnalysisCache_VersionMismatch_OnlyWipesAffectedTable: per-table wipe cmd/scan_test.go: - TestScanCmdEnableDBCacheFlag: verifies flag registered + GetBool path - TestScanCmdEnableDBCacheWithGoProject: exercises OpenAnalysisCache + warm path - TestScanCmdEnableDBCacheOpenError: exercises warning + graceful degradation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- TestCICmdEnableDBCacheFlag: verifies flag is registered with correct default - TestCICmdEnableDBCacheWithGoProject: exercises the full OpenAnalysisCache + BuildGoCallGraph with cache path using the existing setupCIIntegrationTest helper and a minimal Go project fixture Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- TestCICmdEnableDBCacheOpenError: blocks cache dir with a file to trigger the cacheErr != nil warning path in ci.go lines 289-291 - TestBuildGoCallGraph_Cache_WithArgsStdlibAndUnresolved: exercises argument serialisation loops (go_builder.go 542-544, 574-576), unresolved-call tracking (560-562), and warm stdlib edge replay (592-594) using a fixture with fmt.Println + local + unknown calls Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…l display, and inter-procedural enricher - TestDumpCallSitesJSON: exercises dumpCallSitesJSON with Go and non-Go call sites, verifying only Go functions are exported - TestDumpCallSitesJSON_WriteError: covers the file-create error path - TestTextFormatterTaintGlobalWithSourceSnippet: exercises the inter-procedural taint branch in writeDetailedFinding (lines 133-141) that shows Source then Sink headers with code snippets - TestEnrichDetection_InterproceduralSourceLocation: exercises enricher.go lines 62-65 where SourceLocation and SourceSnippet are populated when SourceFunctionFQN differs from FunctionFQN (cross-function taint) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Adds upload_go_thirdparty_to_r2.sh which runs the existing generate_go_thirdparty_registry.go tool and syncs output to R2 under registries/go-thirdparty/v1/. Adds go-thirdparty-r2-upload.yml GitHub Actions workflow that triggers on release, workflow_dispatch, and temporarily on pushes to shiva/pr-09-incremental-cache for end-to-end CDN pipeline validation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The test assumed CDN would return 404 (empty registry). Now that the CDN is live with 25 packages, the combined loader initializes non-nil even with no local go.mod deps. Fix: set CPF_CDN_URL to an invalid URL so the test is not coupled to live CDN availability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Run on release and workflow_dispatch only. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

shivasurya added enhancement New feature or request go Pull requests that update go code labels Apr 7, 2026

shivasurya self-assigned this Apr 7, 2026

shivasurya mentioned this pull request Apr 7, 2026

feat(cmd): wire --enable-db-cache into scan and ci commands #650

Closed

This was referenced Apr 7, 2026

feat: CDN registry generator for Go third-party packages #647

Merged

feat: GoThirdPartyCombinedLoader — CDN-first + local fallback routing #648

Merged

shivasurya changed the base branch from shiva/pr-08-golang-thirdparty to graphite-base/649 April 7, 2026 20:48

shivasurya changed the base branch from graphite-base/649 to main April 7, 2026 20:49

shivasurya and others added 11 commits April 7, 2026 20:50

fix(cdn): use actions/checkout@v6 and setup-go@v6

7817cc6

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(cdn): fix YAML syntax in workflow and remove inline Go heredoc

e1d6a4e

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(cdn): use go 1.26 to match go.work requirement

b488529

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore(cdn): remove PR branch trigger from go-thirdparty upload workflow

e994560

Run on release and workflow_dispatch only. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

shivasurya force-pushed the shiva/pr-09-incremental-cache branch from 1d06e1f to e994560 Compare April 7, 2026 20:50

shivasurya merged commit 0fbf2cd into main Apr 7, 2026
6 checks passed

shivasurya deleted the shiva/pr-09-incremental-cache branch April 7, 2026 20:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cache): delta-based incremental SQLite analysis cache for Go#649

feat(cache): delta-based incremental SQLite analysis cache for Go#649
shivasurya merged 12 commits intomainfrom
shiva/pr-09-incremental-cache

shivasurya commented Apr 7, 2026

Uh oh!

safedep bot commented Apr 7, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 7, 2026 •

edited

Loading

Uh oh!

codecov bot commented Apr 7, 2026 •

edited

Loading

Uh oh!

shivasurya commented Apr 7, 2026 •

edited

Loading

Uh oh!

shivasurya commented Apr 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

shivasurya commented Apr 7, 2026

Summary

Design

Test coverage

Uh oh!

safedep bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SafeDep Report Summary

Uh oh!

github-actions bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Pathfinder Security Scan

Uh oh!

codecov bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

shivasurya commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shivasurya commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

safedep bot commented Apr 7, 2026 •

edited

Loading

github-actions bot commented Apr 7, 2026 •

edited

Loading

codecov bot commented Apr 7, 2026 •

edited

Loading

shivasurya commented Apr 7, 2026 •

edited

Loading

shivasurya commented Apr 7, 2026 •

edited

Loading