Skip to content

docs: update query benchmarks (3.3.1)#548

Merged
carlos-alm merged 3 commits into
mainfrom
benchmark/query-v3.3.1-20260320-080835
Mar 20, 2026
Merged

docs: update query benchmarks (3.3.1)#548
carlos-alm merged 3 commits into
mainfrom
benchmark/query-v3.3.1-20260320-080835

Conversation

@github-actions
Copy link
Copy Markdown
Contributor

Automated query benchmark update for 3.3.1 from workflow run #397.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Mar 20, 2026

Greptile Summary

This is an automated documentation PR that records query benchmark results for version 3.3.1, adding new rows to the summary table and detailed result sections for both the native (Rust) and WASM engines.

Key changes recorded:

  • fnDeps and fnImpact query latencies are 2–3× higher than 3.3.0 for both engines (e.g., native fnDeps d1: 1.4ms → 3.6ms, +157%)
  • Hub target changed from src/types.tsbuildGraph, and WASM mid target changed from functionNodeIddb — suggesting the graph's most-connected node(s) shifted between versions
  • diffImpact latency improved significantly: native 8.8ms → 7ms (↓20%), WASM 8.5ms → 4.8ms (↓44%)
  • diffImpact affected functions and affected files both recorded as 0 (expected for a benchmark run with no staged diff)

The hub target change means the fnDeps/fnImpact regression is not a clean apples-to-apples comparisonbuildGraph may have substantially higher fan-out than src/types.ts. It is worth verifying whether the performance increase is a genuine regression or an artifact of the denser hub node.

Confidence Score: 4/5

  • Safe to merge — documentation-only automated PR with no code changes, though the recorded latency regression warrants a follow-up investigation.
  • The PR only modifies a generated benchmark markdown file. There is no risk to runtime behavior. The 2–3× latency increase for fnDeps/fnImpact is notable and could indicate a real performance regression, but it may also stem from the hub target changing to a denser node. This does not block merging the docs update itself, but the underlying performance question should be tracked.
  • generated/benchmarks/QUERY-BENCHMARKS.md — review the latency regression context (hub target change vs. genuine perf regression)

Important Files Changed

Filename Overview
generated/benchmarks/QUERY-BENCHMARKS.md Automated benchmark update for v3.3.1 — adds new rows to the summary table and detailed result sections for both native and WASM engines. Notable: hub target changed from src/types.ts to buildGraph, and fnDeps/fnImpact latencies increased ~2-3x vs 3.3.0, though diffImpact improved. The WASM mid target also changed from functionNodeId to db.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Version 3.3.1 Benchmarks] --> B[Native Engine]
    A --> C[WASM Engine]

    B --> B1["fnDeps d1: 3.6ms ↑157%\nfnDeps d3: 3.6ms ↑177%\nfnDeps d5: 3.5ms ↑169%"]
    B --> B2["fnImpact d1: 2.5ms ↑92%\nfnImpact d3: 2.6ms ↑100%\nfnImpact d5: 2.5ms ↑92%"]
    B --> B3["diffImpact: 7ms ↓20% ✅"]

    C --> C1["fnDeps d1: 3.5ms ↑169%\nfnDeps d3: 3.6ms ↑177%\nfnDeps d5: 3.5ms ↑192%"]
    C --> C2["fnImpact d1: 2.4ms ↑85%\nfnImpact d3: 2.4ms ↑85%\nfnImpact d5: 2.4ms ↑85%"]
    C --> C3["diffImpact: 4.8ms ↓44% ✅"]

    B1 & B2 --> W1["⚠️ Hub changed: src/types.ts → buildGraph\nLatency 2-3x vs 3.3.0"]
    C1 & C2 --> W1
    B3 & C3 --> OK["diffImpact improved ✅"]
Loading

Last reviewed commit: "docs: update query b..."

Comment on lines 7 to +8
|---------|--------|----------:|----------:|----------:|------------:|------------:|------------:|-----------:|
| 3.3.1 | native | 3.6 ↑157% | 3.6 ↑177% | 3.5 ↑169% | 2.5 ↑92% | 2.6 ↑100% | 2.5 ↑92% | 7ms ↓20% |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Significant query latency regression in 3.3.1

fnDeps and fnImpact query latencies have roughly doubled to tripled compared to 3.3.0:

Metric 3.3.0 (native) 3.3.1 (native) Δ
fnDeps d1 1.4ms 3.6ms +157%
fnDeps d3 1.3ms 3.6ms +177%
fnImpact d1 1.3ms 2.5ms +92%

Note that the hub target changed from src/types.tsbuildGraph between versions. If buildGraph is a significantly denser node in the graph, part of this increase may be expected due to traversing a larger subgraph. However, if the benchmark is meant to track consistent performance across versions, the change in hub target makes the comparison not fully apples-to-apples. It's worth confirming whether this regression is intentional (larger graph, new node as hub) or a genuine performance degradation introduced in 3.3.1.

The diffImpact latency did improve (8.8ms → 7ms native, 8.5ms → 4.8ms WASM), which is a positive signal.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged — the hub target changed from src/types.ts to buildGraph between versions, which is a significantly denser node (it orchestrates the full build pipeline). The latency increase is expected given the larger subgraph traversal. The benchmark script auto-selects the hub as the highest-degree node, so this is an artifact of the codebase growing rather than a genuine performance regression. The diffImpact improvement confirms the query engine itself is performing well.

@carlos-alm
Copy link
Copy Markdown
Contributor

Not a real regression — benchmark target changed

The ↑157-192% fnDeps/fnImpact deltas are misleading. PR #528 pinned the benchmark hub target from auto-selected src/types.ts (a type-barrel file with shallow call chains) to buildGraph (a deep orchestration function with 2-3x more edges to traverse). The two versions measured different targets.

Evidence there's no engine regression:

  • diffImpact improved: 8.8ms → 7ms native (-20%), 8.5ms → 4.8ms WASM (-44%)
  • Depth scaling is flat (depth 1/3/5 all similar) — the cost difference is in the target's connectivity, not traversal performance

The pinning was the right call — future version-to-version comparisons (3.3.1 vs 3.3.2+) will be stable and meaningful. But this specific 3.3.0 → 3.3.1 row is an apples-to-oranges artifact.

Suggestion: Add a footnote to the history table noting the target change, or back-fill by re-running the v3.3.0 benchmark with the current pinned-hub script for a true baseline.

@carlos-alm carlos-alm merged commit d6ac4ca into main Mar 20, 2026
20 checks passed
@carlos-alm carlos-alm deleted the benchmark/query-v3.3.1-20260320-080835 branch March 20, 2026 08:32
@github-actions github-actions Bot locked and limited conversation to collaborators Mar 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant