perf: Optimize query hot paths from CPU profiling#1296
Merged
Conversation
build_fallback_prefixes iterates the entire DB namespace table (per-namespace prefix derivation plus a sort) on every IriCompactor::new. With the default MostGranular split a dataset can register thousands of namespaces, making this the dominant per-query result-formatting cost. Yet the fallback table is only consumed by the CLI and commit-builder display paths (compact_for_display / effective_prefixes); the server query formatters (SPARQL XML/JSON, TSV/CSV) never touch it. Defer it behind a OnceLock so it is built only on first display use. The other parts of new() scale with the small per-query context, not the namespace table. Also add namespace_prefix(), a zero-alloc prefix lookup for streaming writers.
escape_text_into / escape_attr_into decoded and re-pushed every char one at a time, even for the common case (IRIs and most literals) where nothing needs escaping. Replace with a byte scan that bulk-copies clean runs with a single push_str and only handles the ASCII metacharacters and forbidden control points individually; multibyte characters are copied verbatim without decoding. Output is byte-identical to the previous is_xml_char gate: C0 controls other than tab/LF/CR are stripped, tab/LF/CR and DEL and C1 controls are kept, and the U+FFFE/U+FFFF non-characters are stripped. Covered by a parity test against the original char-scan plus multibyte/C1/non-character regression tests. Used by both the SPARQL Results XML and RDF/XML serializers.
Rewrite the SPARQL Results XML serializer to write directly into one pre-sized
buffer. The previous version allocated a fresh String per cell, built an
FxHashMap per row to reorder columns, and cloned every binding through an
unconditional disaggregation pass even when no column was grouped.
The new path precomputes each batch's column indices once, streams the common
(non-grouped) row straight into the buffer, and falls back to cartesian
expansion only when a row actually contains a Grouped binding. Registered-
namespace refs are written as prefix+name without decode_sid's format! alloc,
encoded subject/predicate refs resolve inline (no materialize re-encode
round-trip), and a lang-tagged literal no longer decodes its datatype IRI.
Blank nodes: a blank-node Sid carries the registered BLANK_NODE namespace
("_:" prefix), so namespace_prefix returns Some("_:"); frame it as <bnode>,
not <uri>. Document the hazard on namespace_prefix and cover it with a test
for the registered-code path (the prior test only exercised the EMPTY code).
Output stays byte-equivalent to the previous serializer for SELECT/ASK results,
with one intentional fix: select_one now scans past an empty leading batch
instead of returning empty. Adds unit coverage for datatype omission, xml:lang,
special doubles, grouped cartesian order, unbound omission, head sorting, and
select_one.
The batched NestedLoopJoin accumulator was grouped through std::collections::HashMap<u64, Vec<usize>> built fresh on every flush: the default SipHash hasher over raw u64 subject/object IDs, with no pre-sizing so the table rehashed as it grew. On star-join workloads (BSBM Explore) this grouping hashes one key per accumulated row and was a top self-time cost (the hash_one / reserve_rehash hotspots). Switch group_accumulator_by_subject / _by_object and the star-predicate lookup to FxHashMap and pre-size to the accumulator length. FxHash over a u64 key is near-free versus SipHash, and pre-sizing removes the per-flush rehash growth and its allocation churn. Result-equivalent: keys, grouping, and the sorted unique-id list are unchanged.
The leaflet batch cache decoded all seven columns on every miss (ColumnProjection::all), overriding the cursor's projection so a current-time scan still decoded and allocated the `t` column it never reads. On scan-bound workloads (BSBM Explore) that is wasted decode plus inflated cache weight (fewer leaflets fit, more misses). Decode only what is needed. BinaryCursor computes a per-leaflet decode set from its projection, widened defensively: filter columns when a filter runs, `t` when time-travel replay runs, and ALL when an overlay merge runs (it reads every base column). The V3 batch cache key now carries the column-set bitmask, so a narrow batch is never served to a query that needs an omitted column; the narrowed entries also weigh less. The batched-subject join probe now routes through the inserting cached loader (previously it decoded on miss without caching). Bundle the loader plumbing into LeafletDecodeSpec (leaf_id, leaflet_idx, order, decode_set) so the cached-load signatures stay small. Add v3 batch cache hit/miss counters (LeafletCache::v3_hit_miss) plus a periodic log, to tell whether a scan is thrashing the cache (capacity problem) versus paying per-decode — kept as a standing diagnostic. Result-equivalent: same columns reach every consumer; only unused column decodes are dropped. Fast-path/count scanners pass ALL (unchanged).
…batch BinaryCursor::next_batch dominates Explore self-time (~34%) but the flamegraph gives no informative child frames under it, so inference can't say what the loop spends its time on. Every per-leaflet operation read individually is cheap (max_t is a field load, dir() a thin virtual call, moka get ~0.8%), so the cost is either huge leaflet breadth or a partial unwind artifact — undecidable from the profile alone. Add process-cumulative counters (relaxed atomics) at each loop decision: calls, leaflets visited / skipped / empty / returned, rows returned, and filter invocations. Logged on a doubling-`visited` schedule under target `fluree_db_binary_index::scan`, and snapshot() for programmatic read. The ratios localize the 34% directly: visited/returned (breadth / over-scan), rows/returned (volume), filtered/returned (per-row filter), returned/call (per-call overhead). Diagnostic only — no behavior change; the atomics add minor hot-loop overhead so absolute throughput on this build dips slightly, but the ratios are exact.
Make the binary-scan leaflet-loop counters readable without driving load to trip the periodic log: - GET /debug/scan-stats[?reset=true] returns the counters + derived ratios (visited_per_returned, rows_per_returned, filtered_per_returned, returned_per_call) as JSON; reset zeroes them to scope a workload. - scan_stats::reset() added; promote fluree-db-binary-index to a regular server dependency (was dev-only) so the handler can read the counters. - Drop the initial log threshold 1<<26 -> 1<<18 so it surfaces quickly on small/local runs.
BinaryCursor::next_batch linearly visited every leaflet in each matched leaf and the only pre-decode skip (skip_leaflet) compared just p_const / o_type_const. Leaflets whose key range cannot contain the filter's bound subject/object were still decoded and per-row filtered to empty — the inlined filter_batch loop that dominates Explore's next_batch self-time. Measured: ~14 leaflets visited per useful one, 92% of decoded leaflets filtered to zero rows. Add BinaryFilter::leaflet_out_of_range: sound pruning on the order-leading contiguous bound prefix. It walks the order's sort columns (o_type precedes o_key in every order) and, for each bound column, requires the bound value to lie within the leaflet's [first_key, last_key] range while higher columns are constant, stopping at the first unbound column. Generalizes the OPST star-probe's existing first_key/last_key reject to all four orders. Gated on the same !overlay && !history-replay guards as skip_leaflet. A bound o_key with a wild o_type (untyped strings) correctly prunes nothing beyond skip_leaflet, since o_type leads o_key. BSBM Explore @1m: leaflets decoded+filtered 73,110 -> 7,636 (-90%), filtered-to-empty 67,643 -> 2,169 (-97%), rows returned unchanged (7,129,816). Correctness-exact: W3C eval 250/327 (identical baseline), indexed-scan / time-travel / history / aggregate suites all green.
fast_eq_ne_for_iri_bindings resolved an EncodedSid binding to its IRI string (resolve_subject_iri) and the other side to an IRI string too, then compared strings — per row. On the BSBM Explore join-bound profile that resolve_subject_iri is ~6.4% of total CPU (the dominant cost of this fn), driven entirely by these comparisons (both sides are index subjects). Within a single ledger the binary index's internal s_ids are directly comparable, so when both sides are EncodedSid bindings, compare the raw u64 s_ids and skip resolving either side to an IRI string (and skip materializing the other side via eval_to_comparable). Gated on !is_multi_ledger — cross-ledger s_ids are not comparable, so those fall through to the unchanged IRI-string path. Correctness-neutral: W3C eval 250/327 (identical baseline), multi-ledger (it_query_dataset), sparql, misc, connection, and the full query lib suites all green.
…lters 1b55ecc only reduced the EncodedSid == EncodedSid (var = var) shape to a raw s_id compare. BSBM Explore query5's filter is var = const-IRI (`FILTER (<product> != ?product)`), which fell through to the EncodedSid arm and still resolved the bound side to an IRI string via resolve_subject_iri (~6.4% of total CPU on the join-bound profile). When the bound side is an EncodedSid and the other side is a constant IRI/Sid, reduce that constant to its internal s_id via subject_ref_to_s_id and compare u64s — skipping resolve_subject_iri entirely. Gated on !is_multi_ledger; a None reduction (subject not indexed, or an IRI-normalization mismatch) falls through to the unchanged IRI-string path, so it stays conservative. Correctness-neutral: W3C eval 250/327 (identical baseline), query lib 1086, multi-ledger it_query_dataset + it_query_sparql green.
…mbined row The batched subject-star join built a row-wise combined Vec<Binding> per match (cloning every left column into it), pushed it into a Vec<Vec<Vec<Binding>>> scatter arena keyed by accumulator slot, then transposed the whole arena back into columnar output batches. On the BSBM Explore join-bound profile that row assembly is the bulk of flush_batched_accumulator's self time plus the per-row Binding clone/drop churn. Replace it, for the common case, with a right-only flat scatter: scatter[slot] holds just the right-side tails (right_width bindings per match) concatenated. Inline FILTERs run against a CombinedRowView — a zero-copy view over the stored left row plus the right tail — so no combined row is materialized for filtering. emit_right_scatter_to_output then writes left columns (cloned once, straight from the left batch) and right tails directly into the output columns, in accumulator order. The leaf-scan + object-decode loop is factored into scan_matches, shared by both paths via a per-match closure. Gated on right_width >= 1 and no Bind inline op; Binds (which append/clobber a column) and the object/exists probe paths keep the materialized combined-row path unchanged. right_scan_inline_ops are Filters only by construction, so the right tail width is exactly right_width. Correctness-neutral: query lib 1086, it_query_jsonld/sparql/dataset/aggregates/ construct 253, W3C eval 250/327 (identical baseline). Left-row ordering and OPTIONAL semantics preserved.
scan_matches collected every matching (row, s_id) for a leaflet into a temporary matches: Vec<(usize, u64)> and then replayed it to decode the object binding and scatter. Fuse the two passes: iterate the matched subjects and rows directly into the object-decode + on_match path, so there is no per-leaflet match vector to allocate, grow, and re-read. Also resolve the subject's accumulator indices once per subject instead of once per matched row. Pure restructuring of the scan loop — same PSOT (p_id, s_id, row) order, same matched set. Correctness-neutral: query lib 1086, it_query_jsonld/ sparql/dataset/aggregates 243, W3C eval 250/327 (identical baseline).
…ng it GraphDb::binary_graph built the BinaryGraphView's namespace_codes_fallback with `Arc::new(self.snapshot.namespaces().clone())`, which dereferences the snapshot's already-Arc namespace table and deep-copies the entire map on every query. On a 100M BSBM ledger the namespace table is large (a distinct code per producer/vendor/rating-site data graph), so this per-query copy was ~15% of total CPU on the Explore join-bound profile — second only to the join itself, and entirely in per-query setup. Use the existing `namespaces_arc()` accessor (a refcount bump on the snapshot's `Arc<HashMap<u16, String>>`) — the same fix already applied to the result-formatting IriCompactor. binary_graph was simply missed. Correctness-neutral: the fallback is read-only and the shared Arc holds the same contents. it_query_jsonld/sparql/dataset/construct/aggregates 253, W3C eval 250/327 (identical baseline).
The single-ledger `?var =/!= <const>` fast path resolved the CONSTANT side to its internal s_id via subject_ref_to_s_id on every row. For a Sid/IRI constant that is a dictionary reverse-lookup (find_subject_id_by_parts -> DictTreeReader::reverse_lookup), and on the BSBM Explore @100m join-bound profile that per-row lookup of a loop-invariant constant was ~12% of total CPU — the single largest remaining frame after the namespace-table fix. Add a per-query memo on ExecutionContext (ConstSidKey -> Option<s_id>) and consult it before resolving. The constant resolves once and every subsequent row is a HashMap hit. The context owns exactly one store/snapshot, so the memo is correctly scoped (no cross-ledger aliasing); Arc<Mutex<..>> keeps the context Send+Sync and lets derived per-graph contexts share it. A None resolution is cached too, so an unindexed constant still avoids the per-row reverse lookup before falling through to the IRI-string path. Correctness-neutral: query lib 1086, it_query_jsonld/sparql/dataset/ aggregates/construct 253, W3C eval 250/327 (identical baseline).
The per-query const→s_id memo added in f24abb9 was shared into every derived context, including with_graph_ref. But with_graph_ref switches to the target graph's own store/snapshot AND clears multi_ledger, so the single-ledger const→s_id fast path DOES run inside it — against a different store than the parent. Because the memo key is the constant value alone (no store identity), a dataset query could reuse an s_id resolved in one graph/store while filtering another, mis-filtering composed/multi-default-graph queries where the same IRI maps to a different internal id (or is absent) in another graph. Give with_graph_ref a fresh memo. with_active_graph and with_default_graph keep the same store (only g_id changes), so they correctly continue to share the parent's memo. Correctness-neutral for single-store queries: query lib 1086, it_query_dataset + it_query_sparql 171.
Both are loop-invariant work that was being redone on every row of a scan. 1. Constant filter operand re-encoded per row. `?var =/!= <const>` evaluated the constant operand with eval_to_comparable every row, and for an IRI that re-ran encode_iri_strict -> canonical_split (split + namespace lookup) each time. The operand is variable-free, so its s_id is invariant: resolve it once and memoize by the operand expression's identity (ConstSidKey::ExprPtr). The memo lives exactly as long as the query and the expression tree it points into, so the pointer key is stable across rows and never reused across queries. The memo is consulted by pointer first, so the variable-free check only runs on the first row per operand. 2. Bool predicate re-analyzed (SipHash) per row. Expression::eval_to_bool calls analyze_cacheable_bool_predicate every row, which SipHashes the whole expression tree — but for a multi-variable predicate (e.g. BSBM's `?a < ?b + k`) it then immediately returns None as uncacheable. Add a cheap, hash-free variable-usage pre-check that bails before the hash walk for any non-single-variable predicate. On the BSBM Explore @100m profile these were ~2% (canonical_split under fast_eq_ne) and ~1.6% (SipHash under analyze_bool_cache_inner) of total CPU. Correctness-neutral: query lib 1086, it_query_jsonld/sparql/dataset/aggregates/ construct 253, W3C eval 250/327 (identical baseline).
The batched-join / binary-scan object decode emitted EncodedLit only for the dict/arena kinds (string/json/vector/num-big) and materialized every inline kind, including xsd:integer / xsd:long / xsd:double / xsd:float. Materialized Binding::Lit values hash via FlakeValue (BigInt conversion) and clone/drop the heap payload, so DISTINCT and join dedup over numeric columns paid that cost per row — the to_bitwise_digits_le frame under Binding::hash on the BSBM Explore @100m profile. Emit these four inline numerics as EncodedLit(NUM_INT / NUM_F64) with the matching reserved DatatypeDictId, exactly as the novelty (DictOverlay) path already does — so base and novelty now agree on the representation. The dt_id resolves back to the original o_type through the OType registry (resolve(o_kind, dt_id, 0) == o_type), so DATATYPE() and terminal materialization reconstruct the correct datatype. Other inline subtypes (xsd:int, xsd:short, temporals, …) have no reserved dict id and stay on the materialized path unchanged. EncodedLit hashes/compares/clones as cheap ints, so these values flow through DISTINCT and joins without bigint conversion or heap churn; materialization is deferred to projection. Correctness-neutral: query lib 1086, it_query_jsonld/sparql/dataset/aggregates/ datatype/construct 278, W3C eval 250/327 (identical baseline).
…ATATYPE Covers commit 2f45f10 (inline xsd:integer/double emitted as EncodedLit on the binary-scan path). Drives the INDEXED path via rebuild_and_publish_index + fluree.db(), so it exercises the binary-scan / batched-join decode rather than the novelty path, and asserts the encoded values still round-trip through projection, DISTINCT (deduping a duplicate integer), numeric FILTER, SUM, and DATATYPE — plus inline doubles (NUM_F64) deduping under DISTINCT.
eval_to_comparable on an EncodedLit Var routes through ExecutionContext::decode_encoded_value, which built a fresh BinaryGraphView (graph_with_novelty + namespace/tracker clones) on every call. After inline integers/doubles became EncodedLit (2f45f10), BSBM's numeric FILTERs hit this per row — ~1.6% of total CPU on the Explore @100m profile, all in graph_view. NUM_INT / NUM_F64 values are self-contained in o_key, so decode them directly (ObjKey::decode_i64 / decode_f64) — the same shortcut the aggregate path already uses — and skip graph_view entirely. Gated on the standard integer/double dt_ids; other shapes (e.g. an integer-valued double stored as NUM_INT by novelty) fall through to the view path unchanged. Correctness-neutral: query lib 1086, it_query_sparql/aggregates/datatype/jsonld 231 (incl. the indexed inline-numeric round-trip), W3C eval 250/327 (identical).
# Conflicts: # fluree-db-api/src/view/types.rs
Strip the temporary instrumentation added while profiling BSBM Explore: - scan_stats module + per-leaflet fetch_add counters in BinaryCursor::next_batch (the only one with hot-path overhead) and the GET /debug/scan-stats endpoint that read it; drop the now-unused regular fluree-db-binary-index server dep (the dev-dependency is kept for tests). - log_fastpath_hit_once + its 7 call sites in compare.rs. - LeafletCache v3 hit/miss counters and the once-per-1M-misses tracing::info! "leaflet v3 batch cache stats" log. No behavior change. binary-index + query 1362 tests, it_query_datatype/sparql 164, all green.
Serialize JSON-LD, SPARQL JSON, Typed JSON, and Agent JSON results directly into the output String instead of building a serde_json::Value tree and walking it a second time to serialize. Output is byte-identical to the DOM path (enforced by parity tests), which is retained as the reference for the Value API, pretty mode, and the select_one/ASK/CONSTRUCT/hydration fallbacks. - New format/json_write.rs holds the shared streaming primitives (string escaping, integer/float formatting) with parity tests against serde_json. Floats reuse serde's CompactFormatter so output matches serde exactly (e.g. 1e+30), not raw ryu; serde_json is built with preserve_order, so keys stream in insertion order. - format_results_string* route eligible SELECT results through the streaming path via json_stream_eligible; ineligible cases fall back to the DOM path unchanged. - Agent JSON streams rows into a buffer and measures the byte budget from buffer-length deltas, dropping the per-row second serialization; the buffer capacity is capped at the budget so truncated responses no longer preallocate for the full row count. Large result sets format ~7-9x faster for SPARQL JSON and ~4x for Typed JSON; JSON-LD and Agent JSON see smaller gains and no regressions. Updates the output-formats performance notes accordingly.
zonotope
approved these changes
Jun 9, 2026
| ) -> bool { | ||
| // Sort-column sequence per order (`o_type` precedes `o_key`; `t` and the | ||
| // identity tiebreaks are never bound, so they are not consulted). | ||
| let fields: &[SortField] = match order { |
Contributor
There was a problem hiding this comment.
How are the extra o fields like lang and i taken into account? That probably doesn't matter for spot and psot, but they could change the predicted order of the s binding for post and opst.
Contributor
Author
There was a problem hiding this comment.
OType encodes language now so it is not a missing field (for this reason, one less thing to have to look up) -- langString has its own OType range. This is new with our latest index version from several months ago, it wasn't always this way.
i was a gap and addressed in 978daa1 in a way that it should not affect non-list data at all for performance but data containing lists might prune a bit less but now will be accurate.
Include `o_i` in V3 leaflet range-pruning sort fields so POST/OPST scans do not incorrectly prune on later fields when list indices vary. Preserve the non-list fast path by allowing descent past unbound fields that are constant across the leaflet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch improves query throughput by using CPU-profile-guided optimizations across serialization, binary index scanning, batched joins, filter evaluation, and late materialization.
The largest wins come from removing repeated loop-invariant work from hot paths, reducing binary scan work before decoding, and keeping values encoded longer so joins and DISTINCT process cheaper integer-backed bindings instead of materialized values.
Key Changes
Performance Notes
CPU profiling showed repeated low-level costs were often caused by loop-invariant work happening per row or per call. This branch addresses those cases directly: namespace cloning, constant IRI-to-subject-ID lookup, graph-view reconstruction for inline numeric decode, avoidable binary leaf scans, and materializing numeric values too early.
On the benchmark workload used during development, these changes produced substantial throughput improvements across single-client and high-concurrency runs.