feature: cooperative query cancellation primitives#1303
Merged
Conversation
Add a targeted CyclicBgpOperator for fixed-predicate triangle and 4-cycle BGPs to avoid left-deep nested-loop intermediate blowups. The operator keeps the existing sequential plan as a fallback, exposes EXPLAIN details, and can defer broad cycle edges into exact membership probes for LIMIT-heavy WGPB cases like S2-31/S2-11. Document the new EXPLAIN fields and tuning env vars: FLUREE_CYCLIC_BGP, FLUREE_CYCLIC_BGP_MAX_ROWS, and FLUREE_CYCLIC_BGP_DEFER_ROWS.
Extend CyclicBgpOperator so object-only cycle variables can join on late-materialized encoded object values instead of requiring every object to be an IRI ref. Keep the IRI-only guard for variables that must later act as subjects, preserving correctness for path-style cycles. Also expose object-only encoded support in EXPLAIN and add focused tests for literal object-only squares and subject-bridge rejection.
Restore the original raw SID path for ref-only cyclic joins while keeping encoded object-key support for object-only cycle variables. This preserves the S2 holdout fixes without forcing path-style cycles and triangles through the more expensive Binding-based join path. Expose the selected cyclic mode in EXPLAIN via object-only-values (`iri-ref` or `encoded`) and add tests for mode selection.
Lower the object-to-subject hash join probe floor so TI object in-stars use a single predicate scan instead of many scattered OPST seeks. Guard auto hash joins from wide intermediates to avoid draining multiway star cross-products before LIMIT can stop execution.
Extend the object-to-subject hash join cost gate so small driving sets can hash-join below the generic probe floor. This lets WGPB TI anchors avoid thousands of scattered OPST seeks while preserving the guard against draining wide multiway intermediates before LIMIT can stop execution.
Relax the object hash-join width guard so multi-edge TI in-stars can still hash when the intermediate is wide but estimated small.
Introduce a runtime-agnostic cancellation handle and thread server-configured query deadlines through API builders so long-running queries can stop at execution checkpoints.
External timeout monitors now signal cancellation through the shared handle, while long-running query operators check that handle at direct scan and merge boundaries.
Share query execution cancellation wiring across HTTP and MCP requests, including timeout-triggered cancellation and client-disconnect cancellation when request futures are dropped. Add MCP-specific query timeout configuration with a 5 minute default via --mcp-query-timeout-ms, FLUREE_MCP_QUERY_TIMEOUT_MS, and [server.mcp] query_timeout_ms.
Run server-owned query execution in spawned tasks so request-drop can signal ClientDisconnected while the query task is still alive to observe cooperative checkpoints. Share cancellation controls across HTTP, multi-query, and MCP sparql_query, including configurable server/MCP timeouts and request-scoped execution options.
Add focused query_control tests for disconnect-driven cancellation, normal completion disarming, and timeout signaling so the spawned query task behavior is covered.
…ruct Group the 7 query-level planning fields (required_where_vars, var_counts, protected_vars, group_by, distinct_query, planning, stats) that were threaded identically through build_triple_operators, build_sequential_triple_chain, and build_sequential_join_block into a single TriplePlanContext<'_> struct. Removes the #[allow(clippy::too_many_arguments)] annotations from all three functions; the test wrapper constructs the struct with its known defaults.
Resolved conflicts: - fluree-db-query/src/cyclic_bgp.rs: took perf/star-joins (superset; HEAD had no cancellation checkpoints here) - fluree-db-query/src/hash_join.rs: took perf/star-joins (HASH_JOIN_MEDIUM_BUILD_MAX + de-benchmarked test names) - fluree-db-query/src/execute/where_plan.rs: kept HEAD TriplePlanContext refactor, took perf/star-joins comment cleanup - fluree-db-query/src/context.rs: merged imports (QueryCancellation from HEAD, FIRST_USER_GRAPH_ID from perf/star-joins) - fluree-db-api/src/server_defaults.rs: kept HEAD query_timeout_ms config comment - docs/operations/configuration.md: kept HEAD query_timeout_ms example - docs/query/explain.md: took perf/star-joins expanded CyclicBgp explain fields
Three boundary-level checkpoints — no per-row overhead: - next_batch: checked once per batch before enumeration - open_ref_fast_path: checked once per edge during relation loading - open_encoded_fast_path: same for the encoded-object path
…truct Eight `*_with_r2rml*` methods had been pushed past the clippy too_many_arguments threshold (7+self) by the `options: QueryExecutionOptions` parameter added for query cancellation. The `r2rml_provider` / `r2rml_table_provider` pair already travels as a unit — ExecuteConfig stores them as a 2-tuple — so bundle them into a `pub(crate) R2rmlProviders<'a>` struct defined in fluree-db-api. Also fix: - `merge_modifier_intersect_range` in count_plan_exec: no natural grouping, add targeted `#[allow(clippy::too_many_arguments)]` - Duplicate test name introduced by perf/star-joins merge: rename the regression variant to `indexed_inline_type_star_aggregate_overlay_undercount_regression` - `query_execution_options(&state)` needless-borrow in server routes where `state: &AppState`
…timeout-cancellation
The CyclicBgpOperator relation loaders treated a predicate absent from the base index dictionary as an empty relation. With an active overlay the predicate's facts can live in novelty, so the empty relation zeroed the whole BGP via the empty-driver early exit while the fallback operator tree (whose cursors merge the overlay) found the matches. All four loaders now decline (bail reason "novelty-only-predicate") when the overlay may hold facts, routing to the overlay-correct fallback. The empty-relation shortcut is kept when no overlay is active, where it is genuinely correct. Regression test covers both join modes (RefOnly directed triangle, EncodedObject shortcut) against the fallback ground truth; verified to fail without the fix.
… by predicate Every collect_resolved_overlay_ops call re-walked the graph's entire novelty and re-translated matching flakes via dict probes; count plans and cursor builders repeat this several times per query for the same predicate. Three changes: - Per-execution OverlayOpsCache on ExecutionContext, keyed (GraphId, order, Sid) with translation failure memoized. The cache lazily binds to a fingerprint of its inputs (overlay identity+epoch, to_t, dict novelty, runtime small dicts, store) and re-validates on every access, so a context derivation that shares it while changing any bound input computes uncached instead of reading stale ops (and trips a debug_assert). Lifecycle mirrors const_sid_cache: fresh on root constructors and with_graph_ref, shared on graph switches, reset by the binding-relevant builder setters. Misses compute outside the map lock. - Predicate-bounded overlay walks: Psot/Post collections pass superset-safe boundary flakes so Novelty's slice_for_range binary- searches to the predicate's range instead of walking all novelty. The callback-side predicate filter remains the correctness backstop. - BinaryCursor::set_overlay_ops takes Arc<[OverlayOp]>, so cached ops reach any number of cursors without copying; the no-overlay default is a shared empty slice (no per-cursor allocation). Rewired through the cache: build_overlay_cursor_for_predicate (covers cyclic full scans and count cursor builds), count_plan_exec, fast_union_star_count_all, fast_post_order_limit. BinaryScanOperator's own all-predicate translation path is unchanged (follow-up).
parallel_overlay_psot_filter_count and count_predicate_overlay_delta still called collect_resolved_overlay_ops directly, so a declined parallel lane or union/count fallback re-walked a predicate's novelty that the cached cursor builder would fetch again later in the same query. Both now share the per-execution cache.
…lane The NestedLoopJoin batched lane reads base leaflets directly, so any novelty on the graph forced the per-row fallback — a fresh right scan per left row, each re-translating overlay state. The subject-probe lane now stays live under a single-graph overlay by merging the right predicate's resolved ops per probed subject. ProbeOps (fast_path_common) mirrors merge_overlay_into_batch's case analysis for set-of-matches consumers: an identity-matching retract suppresses the base row, an identity-matching assert keeps it (re-asserted fact), and unconsumed asserts are injected per probed subject after the base scan — through the same bounds checks and on_match closures as base rows, so inline filters and both scatter shapes apply unchanged. Consumed-state is scoped to one flush: probed keys recur across flushes with new left rows. The gate (BatchedOverlayMode, recomputed per next_batch from the per-execution ops cache) declines to per-row scans before any accumulation when ops fail translation, the predicate exists only in novelty, or the PSOT branch is absent while novelty exists. The object (OPST) and exists lanes still require an overlay-free graph. Injected values resolve through DictOverlay (novelty-minted subject and string ids); the object-binding construction is factored out of scan_matches and shared by base rows and injected asserts so both produce identical representations. Differential test: retract-drop, cross-commit retract+re-assert netting, assert injection, novelty-only subjects, dict-novelty strings, and FILTER bounds on injected rows — against a reindexed ground truth, with span capture proving the batched lane engaged under novelty.
The strided per-group ticker still perturbed codegen of the fused merge loops (+5-15% on wikidata-scale COUNT benchmarks); compiling checks out entirely (cancel-checks-off) recovered baseline. Replace in-loop checks with checkpoints on boundaries that already exist: - leaflet/cursor-batch refill in PsotSubjectCountIter, PsotSubjectSeek, PsotSubjectWeightedSumIter, CursorSubjectCountStream (opt-in via with_cancellation; checked every few thousand rows, ~sub-ms of work) - partition start in parallel_partition_count - chain lane threads the handle through execute_chain Hot loop bodies are bit-identical to pre-cancellation builds; cancellation latency stays well under client-visible thresholds.
Accumulated keys from Sid/IriMatch bindings resolved only through the persisted reverse dict, so a left subject minted after the last index (e.g. bound via VALUES) could never enter the batched lane even though the overlay merge can serve its facts — each such row paid a per-row fallback scan. Resolution now falls back to DictNovelty, which yields the same s_id space the overlay ops are translated into. The differential test asserts engagement per query and, for the VALUES-with-novelty-subject case, that both keys reach the accumulator (flush span accum_len) — verified to fail without the fix.
Extends the subject-probe overlay merge to every consumer of the shared leaflet-probe helpers: the batched EXISTS lane, single and grouped OPTIONAL builders, and property-join's chunked probes and SPOT star walk. All previously bailed to slower paths whenever the graph carried any novelty. ProbeOps generalizes to multi-predicate ops: p_id joins the fact identity, so one reconciler serves both the single-predicate PSOT probes and SPOT star probes over ops merged across the star's predicates (both sort subject-major). A shared ProbeLanePlan (Clean/Merge/Decline) centralizes the decline analysis — translation failure, novelty-only predicate, branch absent with novelty, eager materialization (Sid-space reasoning overlays keep the per-row path), multi-graph — so every lane gates identically before accumulating. Both helpers fate-check base rows per probed subject and inject unconsumed asserts through the same bounds and emit paths as base rows (filter decode is dict-overlay-aware for novelty-minted values). The exists lane gains the dict overlay it previously lacked; OPTIONAL subject resolution falls back to DictNovelty; the star walk gains cancellation checks. OPTIONAL retract semantics fall out of merge-before-ingest: a subject whose only match is novelty-retracted yields an unbound optional row, never a dropped one. property_join's subject keys normalize Sid to encoded id (persisted dict, then DictNovelty) outside eager mode, collapsing the double-keyed entries that kept the driver-id capture — and with it both batched walks — from ever engaging when a scan fallback produced Sid rows. Differential tests cover the exists lane (injected and retracted matches), OPTIONAL retract-to-unbound, grouped OPTIONAL, and an anchored property-join star whose subject exists only in novelty, each against a reindexed ground truth with per-lane engagement assertions (the test support layer now captures events as well as spans).
Split the property-join engagement assertion into one query per fast lane: an anchored star proving the SPOT star walk engaged, and a star containing a novelty-only predicate (which declines the whole-star plan) proving the remaining base predicate took the chunked batched probes. Both OPTIONAL builders now emit a completion debug event so the single and grouped lanes each assert engagement directly instead of relying on fallback-event absence (which passes vacuously when the builder is never invoked — the grouped query was actually taking the correlated per-row chain and is now shaped as consecutive single-triple OPTIONALs, the form the grouped builder handles). Also fixes a probe-coverage gap the new assertion exposed: property join re-captured driver subject ids on every scan iteration, which erases them after the first remaining predicate (the capture helper returns None off the driver position) — so in an N-predicate star only one predicate could ever take the batched probe path. Capturing only at the driver position keeps later predicates probing.
…be lanes Completes the overlay-aware probe conversion: the last two leaflet-probe lanes that bailed (or silently read stale base state) under novelty now merge the predicate's resolved ops per probed key. Bound-object (OPST) lane: an object-major reconciler owns the predicate's IRI_REF op subset re-sorted by (o_key, s_id, o_i) — served from the already-cached PSOT ops, no second overlay walk — and reconciles per probed object: retracts suppress matched base rows (discriminating multi-entry @list refs by o_i, which required widening the lane's projection beyond CORE; neither cache configuration decoded OI before), unconsumed asserts inject through the same emit path. The duplicated per-arm emit block is extracted so base and injected matches share one construction, inline filters included. Cyclic probes: per-edge ops are planned once at open (any unmergeable edge disables probing for the cascade; full scans, which merge through the overlay cursor, are unaffected) and merged inside both scan-for-subjects loops via a new identity-yielding PsotSubjectSeek variant that decodes OI. Injected asserts respect the ref-mode split exactly like base rows. Since stats exclude novelty, each edge's op count is folded into the probe-vs-scan estimate so novelty-heavy edges don't look cheaper to scan than they are. Also closes a policy leak introduced with the property-join subject-key normalization: an all-Id key set enables the batched walks, which read raw leaflets and bypass the per-leaf filter_flakes policy filtering — under a restrictive policy the walk exposed rows the scans had hidden. All probe lane plans now decline under a non-root policy enforcer and the key normalization is skipped, restoring the scan path those views always used. Tests: bound-object differential under novelty (retract, inject, novelty-only subjects, and a duplicate-@list retract where exactly one o_i entry dies); the cyclic differential gains a novelty phase across all three execution configs with a probe-engagement assertion; the policy regression is covered by the existing it_policy_indexed suite.
…the overlay-free return The non-root-policy decline sat after the overlay_free_single_graph() short-circuit, so index-only (novelty-free) policy views still received ProbeLanePlan::Clean — permission for the batched raw-leaflet lanes, which never run the per-leaf filter_flakes policy filtering. The policy check now precedes the overlay-free return in all three plans, making the protection independent of novelty state. Adds an index-only restrictive-policy tripwire test for the two-pattern join shape: current routing keeps such queries off the batched lanes upstream, so the assertion guards against any future routing change exposing the raw path.
…s-off The four per-visited-node checks from the original cancellation commit are the only never-relaxed checkpoints on the transitive-path execution route. Routing them through bail_if_cancelled keeps default behavior identical while letting the cancel-checks-off control build rule them in or out of the isolated transitive-path regression (8.5s vs 6.7s published).
…w-flake log storm Typed xsd:decimal and overflow xsd:integer values are NumBig-arena-keyed (per graph + predicate, handles minted at index build), so overlay translation rejected every such flake as Unsupported. The consequences compounded: on the range path each query re-warned per untranslatable flake (the reported log storm — one ledger emitted two WARNs per decimal novelty value on every update), and on the batched probe lanes a single decimal in novelty declined the predicate's whole lane. Translation now resolves big numerics read-only against the arena (find_bigint/find_bigdec — the read-side arena rebuilds its dedup map, so lookups work from the query path), mirroring the resolver's split: i64-fitting integers encode inline, everything else by arena handle. This covers every retract and every re-assert of an indexed value — the dominant update pattern — keeping the batched lanes live. A value the index has never seen still has no handle (minting one would need the ephemeral-handle/decode side-table follow-up) and falls back as before. The range path now logs one summary per translation call instead of one line per flake: a debug summary for Unsupported-only outcomes (a handled condition — the raw-flake merge is correct), warn when unexpected errors occurred. Per-flake detail moves to debug. Generic xsd:duration stays untranslated deliberately: its binary-path decode is a stub (returns Null), so the raw flake preserves the value where a translated row would not. The unused translate_overlay_flakes (superseded by the _with_untranslated variant) is removed.
aaj3f
approved these changes
Jun 11, 2026
aaj3f
left a comment
aaj3f
left a comment
Contributor
There was a problem hiding this comment.
I'm quite excited about this -- I think very large datasets where LLM behavior degraded because timed-out queries were still running on fluree resources will now improve drastically.
Claude found a few direct-leaf-read operators (i.e. not child operators that would inherit checkpoints) that have zero check_cancelled() sites:
fluree-db-query/src/fast_predicate_scalar_agg.rs(e.g.:357-393,:486-492
— full leaf/leaflet/row walk of a predicate for SUM/MIN/MAX)fluree-db-query/src/fast_min_max_string.rsfluree-db-query/src/fast_post_order_limit.rsfluree-db-query/src/fast_count.rsfluree-db-query/src/binary_history.rs(history replay walk)fluree-db-query/src/fast_label_regex_type.rs,fast_exists_join_count_distinct_object.rs,fast_sum_strlen_group_concat.rs
I'm not sure if any of them deserve cancellation checks within this PR or if we just want to file an issue to consider them additively in future PRs
Contributor
Author
I'd like to take another pass at fine tuning this as we go forward, I had some super high performing queries in benchmarks degrade up to 15% because there were too many checks so already moved some things 'up' so they would not been the hot path as much. The goal here is to, I'd say, fully cancel within say 1/4 of a second, not instantly. Some of these hot paths give instant but at a cost. |
Reword the feature comment (the experiment branch it referenced is gone) and document the checkpoint placement policy + cancellation latency bound in the query-timeout configuration docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Query timeout and cooperative cancellation
Long-running queries had no way to stop early: a slow property-path traversal, a large star-join, or a multi-query bundle would hold server resources until it finished even if the client had already gone away or a wall-clock limit had been exceeded. This PR adds a complete cooperative cancellation system and wires it end-to-end.
Cancellation primitive (
fluree-db-core)QueryCancellationis a runtime-agnostic, atomic cancel handle. Three reasons are tracked —Cancelled,Timeout, andClientDisconnected— and the first one signalled wins (compare-exchange). A disabled handle is a singleNonepointer, so callers that do not opt in pay only a branch at each checkpoint (~1 ns).Query operator checkpoints (
fluree-db-query)check_cancelled()calls are placed at the hot iteration boundaries in:A cancelled query returns
QueryError::Cancelled { reason }and unwinds normally — no panics, no leaked state.Builder and execution wiring (
fluree-db-api)QueryExecutionOptionscarries the cancellation handle and any lifecycle guards.ViewQueryBuilderandDatasetQueryBuildereach gain.execution_options()and.cancellation()methods. All internalquery/query_tracked/query_view_with_r2rmlcall sites are updated to forward these options so the handle flows from server route → builder → execution context → checkpoints.Server task isolation and disconnect detection (
fluree-db-server)run_query_taskspawns query work in a separate tokio task. A drop guard on the awaiting side signalsClientDisconnectedif the HTTP or MCP handler future is dropped before the query task completes — catching both graceful client closes and aborted requests without polling. A lightweight timeout task firesTimeoutafter the configured duration and then exits; its abort handle is kept in a second drop guard so it is cleaned up when the query finishes normally.current_query_execution_options()uses a task-local to make already-installed options available to route helpers without threading them through every call site.Configuration
FLUREE_QUERY_TIMEOUT_MSsparql_querytimeoutFLUREE_MCP_QUERY_TIMEOUT_MSSet to
0to disable. Both are also settable via the config file.Tests
cancellation.rsverify disabled/cloned/first-wins semantics.query_control.rsverify disconnect signalling (waiter abort →ClientDisconnected), clean-completion no-signal, and timeout-fires-before-query-ends.it_query_sparql_indexed.rscover end-to-end cancellation through query execution.