Skip to content

docs: update build performance benchmarks (3.3.0)#524

Closed
github-actions[bot] wants to merge 1 commit intomainfrom
benchmark/build-v3.3.0-20260319-081421
Closed

docs: update build performance benchmarks (3.3.0)#524
github-actions[bot] wants to merge 1 commit intomainfrom
benchmark/build-v3.3.0-20260319-081421

Conversation

@github-actions
Copy link
Contributor

Automated build benchmark update for 3.3.0 from workflow run #381.

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Mar 19, 2026

Greptile Summary

This is an automated CI-generated PR that appends 3.3.0 benchmark data to generated/benchmarks/BUILD-BENCHMARKS.md and refreshes the performance metrics table in README.md. The new entries cover all four benchmark dimensions tracked in BUILD-BENCHMARKS.md: per-file build speed, raw totals, build-phase breakdown, incremental rebuild times, and query latency.

  • All percentage-change annotations (↑129%, ↑59%, etc.) were spot-checked and are arithmetically correct.
  • README metrics (8 ms/file, 7ms query, 422ms 1-file rebuild, ~400.0s 50k estimate) are consistent with the data appended to BUILD-BENCHMARKS.md.
  • The benchmark link list in README.md was previously flagged for dropping three links; that regression is being addressed in PR fix: stabilize benchmark targets across engines and preserve README links #527.
  • QUERY-BENCHMARKS.md and INCREMENTAL-BENCHMARKS.md not being updated here is confirmed expected behaviour — those are produced by separate workflow runs (PRs docs: update query benchmarks (3.3.0) #523 and docs: update incremental benchmarks (3.3.0) #526).
  • The version history skips 3.2.x entirely (3.1.4 on 2026-03-16 → 3.3.0 on 2026-03-19), with no entries in any benchmark table. If a 3.2.x release was tagged but the workflow wasn't triggered for it, this creates a gap that makes the large regressions in 3.3.0 harder to attribute. If the version jump was deliberate, a brief note in the ### Notes section would clarify this.

Confidence Score: 5/5

  • Safe to merge — purely additive documentation change with internally consistent data.
  • All metric values are internally consistent between the README table and BUILD-BENCHMARKS.md, percentage-change calculations are correct, and the change is documentation-only with no executable code modified. The only open question is the missing 3.2.x history, which is a documentation gap rather than a correctness issue.
  • No files require special attention beyond the version-history gap noted in BUILD-BENCHMARKS.md.

Important Files Changed

Filename Overview
README.md Updated performance metrics table to reflect 3.3.0 numbers; benchmark link list reduced from 5 to 2 links (previously flagged and fixed in PR #527).
generated/benchmarks/BUILD-BENCHMARKS.md Appended 3.3.0 rows to all benchmark tables (build, incremental, query latency) and updated raw-totals, phase-breakdown, and 50k-file estimate sections. Notable: version history jumps directly from 3.1.4 to 3.3.0 with no 3.2.x entries.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Release Tag: 3.3.0] --> B[CI Workflow Run #381]
    B --> C[Build Benchmark Step]
    B --> D[Query Benchmark Step - separate PR #523]
    B --> E[Incremental Benchmark Step - separate PR #526]
    C --> F[Runs codegraph on own codebase\n429 files, native + WASM]
    F --> G[Collects metrics:\nbuildTimeMs, queryTimeMs,\nnodes, edges, dbSize,\nphase breakdown, query latency]
    G --> H[github-actions bot updates\ngenerated/benchmarks/BUILD-BENCHMARKS.md\nand README.md metrics table]
    H --> I[PR #524 opened to main]
Loading

Last reviewed commit: "docs: update build p..."

## 📊 Performance

Self-measured on every release via CI ([build benchmarks](generated/benchmarks/BUILD-BENCHMARKS.md) | [embedding benchmarks](generated/benchmarks/EMBEDDING-BENCHMARKS.md) | [query benchmarks](generated/benchmarks/QUERY-BENCHMARKS.md) | [incremental benchmarks](generated/benchmarks/INCREMENTAL-BENCHMARKS.md) | [resolution precision/recall](tests/benchmarks/resolution/)):
Self-measured on every release via CI ([build benchmarks](generated/benchmarks/BUILD-BENCHMARKS.md) | [embedding benchmarks](generated/benchmarks/EMBEDDING-BENCHMARKS.md)):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Broken links to existing benchmark files removed from README

This line previously linked to four benchmark resources, but the automated update reduced it to only two:

- [query benchmarks](generated/benchmarks/QUERY-BENCHMARKS.md)
- [incremental benchmarks](generated/benchmarks/INCREMENTAL-BENCHMARKS.md)
- [resolution precision/recall](tests/benchmarks/resolution/)

All three still exist in the repository (generated/benchmarks/QUERY-BENCHMARKS.md, generated/benchmarks/INCREMENTAL-BENCHMARKS.md, and tests/benchmarks/resolution/resolution-benchmark.test.js). The QUERY-BENCHMARKS.md file in particular contains granular query-depth breakdowns (d1/d3/d5) and diffImpact latencies that are not present in BUILD-BENCHMARKS.md. Removing these links will make those resources undiscoverable from the README.

Suggested change
Self-measured on every release via CI ([build benchmarks](generated/benchmarks/BUILD-BENCHMARKS.md) | [embedding benchmarks](generated/benchmarks/EMBEDDING-BENCHMARKS.md)):
Self-measured on every release via CI ([build benchmarks](generated/benchmarks/BUILD-BENCHMARKS.md) | [embedding benchmarks](generated/benchmarks/EMBEDDING-BENCHMARKS.md) | [query benchmarks](generated/benchmarks/QUERY-BENCHMARKS.md) | [incremental benchmarks](generated/benchmarks/INCREMENTAL-BENCHMARKS.md) | [resolution precision/recall](tests/benchmarks/resolution/)):

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #527 — the regex now uses (.+?): to correctly capture markdown links with parentheses in URLs, preserving all existing benchmark links.

Comment on lines 7 to +9
|---------|--------|------|------:|----------------:|-----------:|-----------:|-----------:|----------------:|
| 3.3.0 | native | 2026-03-19 | 429 | 8 ↑129% | 6.8 ↑112% | 17.8 ↑33% | 36.8 ↑28% | 49247 ↑11% |
| 3.3.0 | wasm | 2026-03-19 | 429 | 11.5 ↑20% | 6.7 ↑46% | 17.7 ↑32% | 36.8 ↑27% | 47815 ↑12% |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 QUERY-BENCHMARKS.md and INCREMENTAL-BENCHMARKS.md not updated for 3.3.0

The automated workflow appended 3.3.0 rows to BUILD-BENCHMARKS.md but the sibling files generated/benchmarks/QUERY-BENCHMARKS.md and generated/benchmarks/INCREMENTAL-BENCHMARKS.md still only contain data through 3.1.4. This creates a gap in the historical record for those benchmark dimensions (e.g. fnDeps at varying depths, diffImpact, and native batch vs. JS import-resolution throughput).

If the CI workflow for generating query and incremental benchmarks is separate from the build benchmark workflow, this may be expected. However, if those files are supposed to be updated in the same pass, the workflow for this run (#381) may have only executed the build-benchmark step.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sibling benchmark files (query, incremental) are updated by separate workflow runs that produced PRs #523 and #526. This is expected — each benchmark dimension has its own workflow step and PR.

@carlos-alm
Copy link
Contributor

The issues identified in this PR (inconsistent mid targets, README link stripping, missing regression notes) are addressed in #527. That PR should be merged first, then this benchmark PR can be re-run or closed and re-triggered.

@carlos-alm
Copy link
Contributor

@greptileai

@carlos-alm carlos-alm closed this Mar 19, 2026
@github-actions github-actions bot locked and limited conversation to collaborators Mar 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant