perf: query read scaling, object→subject hash join, planner fixes, and EXPLAIN physical-plan tooling#1293
Merged
Merged
Conversation
Concurrent read throughput against a single ledger barely scaled with cores (BSBM-Explore-class workloads): an in-process bench showed ~1.96x at 2 cores, then a collapse to ~1.3x that stayed flat through 16 cores. Every read query took an EXCLUSIVE tokio::sync::Mutex in LedgerHandle::snapshot() and, while holding it, deep-cloned the whole LedgerSnapshot (namespace maps, stats, schema, graph registry, watermarks). Reads on one ledger therefore serialized on that mutex, and each paid a deep clone. Two changes, both required: - LedgerState.snapshot / LedgerView.snapshot become Arc<LedgerSnapshot>, propagated through GraphDb so deriving a query view is a refcount bump rather than a deep copy (eliminates two per-query deep clones). Snapshot mutations (commit / load / reindex / reload / staging / rebase / push) use Arc::make_mut (copy-on-write); writers are rare and already serialized, so the CoW clone is off the read hot path. - LedgerHandle's state and binary_store guards become RwLock, so concurrent reads take a shared read() (cloning the now-cheap Arc snapshot) instead of serializing on an exclusive Mutex; transactions take write(). The Arc change is the necessary complement to the RwLock: under a shared read lock, N readers clone the snapshot concurrently, and Arc makes that a refcount bump instead of N contending deep copies. In-process bench (single ledger, trivial lookup queries): flat ~1.3x / ~16k QPS peak before -> 3.75x @4c, 6.54x @8c, 8.05x @16c, ~107k QPS peak; +12% single-core. Adds it_scaling_bench.rs (#[ignore]) as the harness.
Adds HashJoinOperator for the BSBM-BI "small + large" object->subject path join (e.g. `?review rev:reviewer ?reviewer . ?reviewer bsbm:country <US>`). The planner correctly drives from the selective bound-object side, but the NestedLoop then resolves the large predicate via per-driving-object OPST seeks into the scattered object-major index, which degrades superlinearly (~47s at 100M). The hash join builds a table from the small driving side and scans the large predicate's contiguous partition once + probes, turning N scattered seeks into one sequential scan. Selection is cost-based in build_scan_or_join: the shape must match AND the probe predicate must be large enough (probe_count >= 250k) but not too large relative to the driving set (probe_count <= driving_est * 64). driving_est is a running per-triple cardinality estimate threaded through build_triple_operators and build_sequential_join_block. FLUREE_HASH_JOIN is a force-override only (On/Off/unset=Auto). Correctness: unbound/poisoned build-side join vars are kept as wildcard rows that match every probe row (the join var takes the probe value), matching the nested-loop "take the right side" semantics rather than dropping rows. On a true multi-graph dataset the operator falls back to a NestedLoopJoinOperator at open() rather than erroring, since the ledger-local key normalisation is single-graph. Verified at 10M: auto-selects (~1.45x faster), correct results; VALUES UNDEF gives byte-identical results vs the nested loop; 205 query tests pass with the hash join forced on. Follow-up: fold the shared planning state into a HashJoinPlanner.
Fold the hash-join selection state — StatsView, the FLUREE_HASH_JOIN force mode, and the running driving-chain cardinality — into a single HashJoinPlanner threaded through a WHERE join block. build_triple_operators and build_sequential_join_block now share one planner via before_step() instead of each maintaining its own driving_est running-product loop, and build_scan_or_join takes &HashJoinPlanner in place of the (stats, driving_est) parameter pair. Move all of the selection logic (force enum, cost constants, cost model, shape eligibility, planner) from where_plan.rs into hash_join.rs, co-locating "when to use the hash join" with the operator itself and trimming the where_plan god-file. Behavior-preserving: the cost decision is byte-identical to before.
The object→subject hash join's cost gate rejected the very query it exists for — BSBM BI-1's `?review rev:reviewer ?reviewer . ?reviewer bsbm:country <US>` COUNT(*) stayed ~48s. The driving estimate for a bound-object pattern is the average object size (count/ndv ≈ 28K for country), but US is skewed-popular (actual ~61.8K), so the estimated probe/driving ratio (2.85M/28K = 101.7×) exceeded the 64× cap even though the actual ratio (46×) sits under it. The scattered-OPST alternative costs ~760 µs/seek vs ~26 ns per sequential probe row, putting the true break-even ratio near 29,000×, so 64× was far too tight. Raise it to 1024× — comfortably absorbs the average-vs-skewed estimate error while still guarding the pathological tiny-driving × huge-probe case.
The eligibility check required `tp.p.is_sid()` and the cost lookup used `tp.p.as_sid()`, but only reasoning queries are pre-encoded to SIDs (runner.rs); a plain SPARBM/JSON-LD query reaches planning with `Ref::Iri` predicates. So the hash join was silently ineligible for the exact BSBM BI-1 join it targets — `?review rev:reviewer ?reviewer . ?reviewer bsbm:country <US>` — regardless of the cost gate, which is why raising the scan-ratio cap alone changed nothing. Accept any fixed predicate (SID or IRI) for the probe, and look probe cardinality up by IRI via `get_property_by_iri` when it isn't a SID — mirroring the planner's `property_stats` fallback so the cost model and reorder agree. The cap raise + this together are what let the hash join replace the scattered-OPST nested loop.
…ogical plan Explain previously re-derived join order with a hand-rolled greedy reorder (reorder_for_explain) that diverged from the executor's planner::reorder_patterns — so the "optimized" order shown to users could differ from what actually runs. Route explain_patterns through reorder_patterns (wrap triples as Pattern::Triple, reorder, unwrap) and delete the duplicate, including its all-fallback/all-equal short-circuits. Add a compound-aware `plan.logical` view that preserves OPTIONAL/UNION/MINUS/ EXISTS/subquery structure and renders each node with kind + category + estimate in the planner's chosen order, with user-facing IRIs. It is present even without statistics (heuristic estimates). The flat original/optimized arrays are unchanged for back-compat. Reconcile docs/query/explain.md with the emitted JSON (the old text-format example was produced by no endpoint) and lock the logical/execution-hints surfaces with tests.
Add an Operator::describe() introspection method (provided; default leaf via op_name/plan_children/plan_details) so the real operator tree can be rendered without executing. The hot path is untouched — describe() is only ever called by explain. Explain now builds the real tree via build_operator_tree (build-only: no open(), no I/O — so no scans/joins run) and walks it into a new plan.physical field. Because fast-path / count-planner / fold selection happens at build time, plan.physical shows the chosen physical operators (PropertyJoin vs HashJoin vs NestedLoop, count fast paths, etc.) that the pattern-level views cannot. Build errors are surfaced in-band rather than failing the explain. Edges carry a relationship kind (child / fallback / conditional) so a fast-path operator's correctness fallback never reads as a co-executing child. plan_children is implemented on the modifier chain, grouping, the three joins, compound operators, and count_rows; remaining fast-path operators render as named leaves (the "which fast path fired" signal is still shown). Document plan.physical and the planned-vs-actual boundary; add tests asserting the tree is connected and operators resolve to concrete names.
…more operators Cover the rest of the operator tree in plan.physical: - FastPathOperator (the wrapper used by the whole count/metadata fast-path family, including the generic count planner) now renders as "FastPath:<label>" so explain names WHICH fast path fired, with the generic pipeline attached on a `fallback` edge (not a child — only one runs). - fast_group_count_firsts operators and UnionStarCountAll expose their fallback edge likewise. - Graph/Service/Values/GeoSearch/S2Search/PropertyPath connect their child so the tree no longer truncates at them. - DatasetOperator exposes scan details (predicate + planned index hint when the planner set one) via a new DatasetBuilder::plan_details hook. Add a test asserting a COUNT(*) explain surfaces a fast-path operator.
Add HashJoinPlanner::explain_object_hash_join, which computes the annotated decision (join var, probe_count, driving_est, scan_ratio, chosen, and a HashJoinReason: forced-on/off, cost-wins, probe-too-small, scan-ratio-too-high, no-probe-stats). build_scan_or_join now computes the decision once, applies `chosen` to pick HashJoin vs NestedLoop, and stashes it on the chosen operator (plan-only — never read on the hot path). The prior choose_object_hash_join is replaced (its single caller switched over). plan.physical now renders hash-join-chosen / hash-join-reason and the cost inputs on join nodes, so a NestedLoop that lost the hash join shows why — the diagnostic the object-drive perf work needs. Also fill in the last bare leaves: PropertyJoinOperator lists its fused predicates; count-plan already rendered as a labeled FastPathOperator. Add object->subject join tests: NestedLoop with the rejected reason by default, HashJoin under FLUREE_HASH_JOIN=1.
…h join In a "bowtie" join — two equally-selective bound-object filters connected through a chain (BSBM-BI: producer→country=DE and reviewer→country=US) — reorder tied the two filters on estimate and broke the tie by original query index. Since the BI queries are written producer/DE-first, that drove from the DE side, which leaves the 2.85M-row rev:reviewer join as a subject-driven forward join over a large intermediate (NestedLoop, ~110s on BI-1's F2 at 100M). Add a tie-break before original index: among equally-selective starts, prefer the one whose chain turns the LARGER predicate into an object→subject hash join (one contiguous probe scan) instead of a forward join. Driving from US makes the 2.85M rev:reviewer join a hash join — F2 110s -> 2.38s (46x), identical result. The signal is 0 without stats, so it only ever breaks exact row-count ties.
A forward join — the right pattern's subject is bound from the left and its object is new — is not an object→subject hash candidate, so the decision helper returned None and EXPLAIN left the NestedLoop opaque (no reason, no driving size). This is exactly the BSBM-BI F2 case: the reviewer joins land subject-driven and the plan gave no hint why they weren't hash joins. Classify the join shape into eligible / subject-driven / not-a-candidate, and for the subject-driven case return a decision carrying reason="subject-driven-forward- join", the driving-set estimate, and the probe predicate count. The NestedLoop already renders a stashed decision, so these now show up — making "reorder to drive the other end" legible from the plan alone. join_var becomes Option (a rejected or forward join has none).
The SubqueryOperator builds its subplan lazily (it holds the SubqueryPattern + stats + planning, not an operator field), so the default plan_children walk truncates at the subquery node — leaving the inner joins, where BSBM-BI time actually goes, opaque in plan.physical. Override describe() to rebuild the inner tree on demand for explain: build-only (no open()/exec), via the same build_where_operators_seeded + apply_solution_modifiers path the runtime uses, seeded with a schema-only SeedOperator carrying the correlation vars so the inner reorder sees the same initial bound set. The inner plan renders under a SubqueryBody node (the first child remains the outer correlated input); node details report join-mode and correlation-vars. Add a sub-SELECT test asserting the inner aggregation is visible under SubqueryBody.
build_inner_plan_for_explain always seeded the rebuilt inner plan with SeedOperator(correlation_vars), but execution seeds two ways: join-mode evaluates the body once with an EmptyOperator (materialize), while a genuine per-row subquery seeds the correlation vars. The seed's schema is the inner reorder_patterns' initial bound set AND the child every nested subquery sees, and a SeedOperator reports estimated_rows()=Some(1) — so nested subqueries' cardinality guard evaluated 1 >= 8 as false and EXPLAIN rendered them join-mode=false even when they actually materialize, misreporting both the inner join order and nested join-mode (e.g. the BSBM BI-8 plan). Seed EmptyOperator for join-mode / uncorrelated subqueries and only SeedOperator(correlation_vars) for the per-row path, so plan.physical reflects the plan that runs.
…orld scan estimate_pattern scored every property path at DEFAULT_PROPERTY_SCAN_SELECTIVITY (1,000,000) regardless of whether an endpoint was bound. A transitive path with a constant (or already-bound) subject/object — e.g. `<concept/912> skos:broader+ ?b` — enumerates the reachable set from a fixed node, typically a handful of rows, but the 1M estimate made reorder rank it above a joined high-cardinality predicate. So `<c912> skos:broader+ ?b . ?b skos:prefLabel ?lbl` drove a full scan of all 150k prefLabel triples instead of probing prefLabel by the ~5 nodes the path produces: 88s vs ~1ms for the VALUES-materialised equivalent (issue #1287). Estimate a path anchored at a bound endpoint as a small bounded closure so the planner drives it first and the joined predicate becomes a per-subject probe. An unanchored path (both endpoints free) is genuinely unbounded and keeps the large estimate.
… test env Address three review findings on the hash-join/explain work. The object→subject hash join collapsed both Unbound and Poisoned join keys to a single wildcard that matched every probe row. That is correct for Unbound, but a Poisoned binding (a failed OPTIONAL) must BLOCK matching — the nested loop skips such rows — so a failed OPTIONAL feeding the hash join could fan out to every probe row instead of producing no match. join_key now returns a 3-way class: Unbound is a wildcard, Poisoned is dropped, everything else is keyed. Adds an operator-level test that a Poisoned build row produces no fan-out, plus a classification test. plan.logical re-estimated every rendered node with an empty bound-var set, so later nodes showed full-scan estimates as if no earlier variable were bound, diverging from the planner's context-aware estimates. The ordered logical plan now threads the evolving bound set (and inner compound pattern lists thread their own), with a regression test that the second triple of a chain estimates bound-subject rather than a full predicate scan. The explain test mutated FLUREE_HASH_JOIN process-wide without isolation; since HashJoinPlanner reads it at plan time, a parallel plan-building test could go flaky. A shared mutex + RAII guard now serializes the file's plan-building tests and restores the prior value on drop (including on panic).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR improves query scaling and planner diagnostics for large read/query workloads.
Arc<LedgerSnapshot>and switches cached ledger state toRwLock, allowing concurrent read snapshots to proceed in parallel while preserving exclusive write access.HashJoinOperatorfor large path joins, including planner tie-breaks that choose the driving side most likely to unlock a large hashable predicate scan./fluree/explainwith compound-aware logical plans and planned physical operator trees, including fast-path labels, scan details, hash-join decisions, and expanded subquery bodies.Closes: #1287
Performance Notes
8.05xat 16 cores, with peak QPS reported around107k.110s -> 2.38simprovement for the BI F2 shape.<concept/912> skos:broader+ ?bare estimated as bounded closures rather than million-row scans.Explain Changes
plan.logicalnow preserves compound query structure while reflecting the same planner order used by execution.plan.physicalrenders the planned operator tree without executing the query, including:HashJoinOperatorvsNestedLoopJoinOperatorselectionSubqueryBody