docs: update query benchmarks (3.3.1)#548
Conversation
Greptile SummaryThis is an automated documentation PR that records query benchmark results for version 3.3.1, adding new rows to the summary table and detailed result sections for both the native (Rust) and WASM engines. Key changes recorded:
The hub target change means the Confidence Score: 4/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Version 3.3.1 Benchmarks] --> B[Native Engine]
A --> C[WASM Engine]
B --> B1["fnDeps d1: 3.6ms ↑157%\nfnDeps d3: 3.6ms ↑177%\nfnDeps d5: 3.5ms ↑169%"]
B --> B2["fnImpact d1: 2.5ms ↑92%\nfnImpact d3: 2.6ms ↑100%\nfnImpact d5: 2.5ms ↑92%"]
B --> B3["diffImpact: 7ms ↓20% ✅"]
C --> C1["fnDeps d1: 3.5ms ↑169%\nfnDeps d3: 3.6ms ↑177%\nfnDeps d5: 3.5ms ↑192%"]
C --> C2["fnImpact d1: 2.4ms ↑85%\nfnImpact d3: 2.4ms ↑85%\nfnImpact d5: 2.4ms ↑85%"]
C --> C3["diffImpact: 4.8ms ↓44% ✅"]
B1 & B2 --> W1["⚠️ Hub changed: src/types.ts → buildGraph\nLatency 2-3x vs 3.3.0"]
C1 & C2 --> W1
B3 & C3 --> OK["diffImpact improved ✅"]
Last reviewed commit: "docs: update query b..." |
| |---------|--------|----------:|----------:|----------:|------------:|------------:|------------:|-----------:| | ||
| | 3.3.1 | native | 3.6 ↑157% | 3.6 ↑177% | 3.5 ↑169% | 2.5 ↑92% | 2.6 ↑100% | 2.5 ↑92% | 7ms ↓20% | |
There was a problem hiding this comment.
Significant query latency regression in 3.3.1
fnDeps and fnImpact query latencies have roughly doubled to tripled compared to 3.3.0:
| Metric | 3.3.0 (native) | 3.3.1 (native) | Δ |
|---|---|---|---|
fnDeps d1 |
1.4ms | 3.6ms | +157% |
fnDeps d3 |
1.3ms | 3.6ms | +177% |
fnImpact d1 |
1.3ms | 2.5ms | +92% |
Note that the hub target changed from src/types.ts → buildGraph between versions. If buildGraph is a significantly denser node in the graph, part of this increase may be expected due to traversing a larger subgraph. However, if the benchmark is meant to track consistent performance across versions, the change in hub target makes the comparison not fully apples-to-apples. It's worth confirming whether this regression is intentional (larger graph, new node as hub) or a genuine performance degradation introduced in 3.3.1.
The diffImpact latency did improve (8.8ms → 7ms native, 8.5ms → 4.8ms WASM), which is a positive signal.
There was a problem hiding this comment.
Acknowledged — the hub target changed from src/types.ts to buildGraph between versions, which is a significantly denser node (it orchestrates the full build pipeline). The latency increase is expected given the larger subgraph traversal. The benchmark script auto-selects the hub as the highest-degree node, so this is an artifact of the codebase growing rather than a genuine performance regression. The diffImpact improvement confirms the query engine itself is performing well.
Not a real regression — benchmark target changedThe ↑157-192% fnDeps/fnImpact deltas are misleading. PR #528 pinned the benchmark hub target from auto-selected Evidence there's no engine regression:
The pinning was the right call — future version-to-version comparisons (3.3.1 vs 3.3.2+) will be stable and meaningful. But this specific 3.3.0 → 3.3.1 row is an apples-to-oranges artifact. Suggestion: Add a footnote to the history table noting the target change, or back-fill by re-running the v3.3.0 benchmark with the current pinned-hub script for a true baseline. |
Automated query benchmark update for 3.3.1 from workflow run #397.