feat(sql): pushdown lane M1/M2 — MIN/MAX, BIND, widening filters, partitioning, caps, block cache, class policies - #1785
Conversation
…ment Each extreme now reads its column through its own materialization alias (AccessInfo.output_names names the statement outputs feeding an access), so MIN and MAX of one column, or of a key, are distinct outputs instead of declining as "MIN/MAX column also a key or another extreme". Pinned ungrouped and per-group, and live on SQLite, Postgres and MySQL.
A BIND no longer throws the block back to the per-scan lane. The statement returns the columns the expression reads and the engine computes the value per row before the residual filters, so a filter over the bound variable still works. Admission is positional: the BIND reads only variables the block bound before it, nothing the statement joins or filters on reads the bound variable, and a BIND inside an OPTIONAL or a UNION branch is left to the engine. DISTINCT narrowing keeps the columns the expressions read; the grouped statement ignores binds its aggregates do not read. Pinned with a per-scan differential and live on SQLite, Postgres and MySQL.
…ment cannot run exactly A filter the plan cannot evaluate exactly no longer costs the whole filter: STRSTARTS, STRENDS and CONTAINS of a constant against a string column, and a REGEX anchored on a literal prefix with no flags, push a LIKE with its own escape character while the exact filter stays in the engine over the returned rows. A collation can only match more strings than a byte prefix, so the pushed predicate is a superset. Inside a conjunction the parts that cannot widen are dropped from the pushed predicate; a disjunction needs every part. The filter remains a residual, so LIMIT stays in the engine and the grouped statement declines. Live on SQLite and MySQL the case-insensitive LIKE returns the extra row and the engine narrows it back.
…ring triples maps An entity whose members come from several triples maps sharing its subject template used to answer no rows and send no statement on the pushdown lane, and no rows on the per-scan lane either (F10 in the virtual-dataset audit). The lane now resolves such an entity to one access per distinct table, joined on the subject's key columns, with maps over the same table and subject sharing one access; a class from one map and a member from another share the access too, and a foreign key into the entity joins whichever access shares the parent's row. Every member must have exactly one providing map when no map provides them all, or the block is left to the engine. Object terms now name the triples map their object map belongs to and the materializer keeps an encoder per map, so a second map's datatypes encode correctly. The per-scan lane's star formation now refuses to fuse a star no single map covers, emitting the members as separate scans joined on the subject, so the differential oracle answers these shapes correctly. Pinned with a partitioned mapping on the fake endpoint and live on SQLite, Postgres and MySQL.
…e key-set caps A VALUES block or an IN list inside the block above the provider's key-set cap is no longer rendered into one statement: the block still runs on the lane, the VALUES in the engine and the IN as a residual. Such a statement would have been rejected by the provider as a hard query error rather than a fallback, since the block's own key set is not chunked the way outer bindings are. One outer key row over the byte budget runs the block unseeded. Pinned: 2001 outer keys go out as two statements, and a UNION above eight branch combinations declines with its tracked reason.
…statement-size caps
The literal encoder kept xsd:dateTime, xsd:date and xsd:time values as string-backed typed literals. Ordering coerces a string against a temporal literal, so FILTER(?t > ...) worked, but RDFterm-equal treats the pair as a type error, so FILTER(?t = ...) and != matched nothing on any graph source and on either lane. Coerce temporal datatypes like the numeric ones; a fractional lexical now renders canonically (.5Z, not .500000Z).
…tamps An xsd:dateTime literal against a timestamp column with no zone was never pushed, so the whole table came back for the engine to filter. Whatever zone such a column was written in, its values lie within 14 hours of the instant they denote, so a window that wide around the literal keeps every row the exact comparison can. lower_superset now pushes it for =, <, <=, > and >= as a naive TIMESTAMP literal (the database converts nothing) and the exact filter stays a residual; != is not widened. SQLite stores timestamps as text, where a time-of-day bound orders wrong across the two separators the bridge decodes, so a new timestamp_is_text capability renders the bounds as whole days (>= '2024-01-09'), which a date literal against a naive timestamp column now renders on SQLite only. Pinned on the fake and live on SQLite, Postgres and MySQL.
…y set Every outer batch became a seeded statement, so 50,000 ledger subjects against a 100,000-row table meant 25 round trips carrying key sets, and a block nothing seeds re-ran once per batch. Once the outer side has passed one key set (2000 rows) the block source now counts the branch with one COUNT(*) and, when it holds at most 100,000 rows and no more than four rows per outer row seen so far, fetches it whole with no modifiers and hash-joins every later batch to it in memory. A larger block, or one still far larger than the outer side, stays seeded exactly as before; the row cap is FLUREE_SQL_PUSHDOWN_CACHE_ROWS (0 keeps every batch seeded). On the 1M-row Postgres probe, 50,000 VALUES against customers (100k rows) drops from 380 ms to 177 ms; 5,000 stays seeded at 39 ms (was 38); the orders table (1M rows) is counted and stays seeded, 43 ms against 25. The decision is measured rather than estimated: the count is one statement, and the outer size is what has already streamed in. Pinned on the fake for the three paths (cached, above the cap, caching off) and two new probe shapes for each table.
…edicate
A view policy targeting a class, over a map whose rdf:type comes from a
column, made the whole block decline ("policy not static"): for a
deployment with policy always on, the lane was off for that map. The gate
now decides such a (map, predicate) verdict once per class any view
restriction is selected by, against the no-class baseline, and the lane
renders the result on the class column: rows of a denied class drop out
(kind IS NULL OR NOT kind IN (...), a row without a class keeping the
default) or only rows of an allowed class stay (kind IN (...)). A
subject-targeted policy, a map deriving classes from several columns or
maps, and a policy on an OPTIONAL member still decline.
The same reversal lets a class constraint (?p a <kind/staff>) on such a
map become kind = 'staff' instead of declining, and a class outside the
template empties the block without a round trip. The per-scan lane
answers neither shape (it matches a constant class against rr:class
alone, #1780), so that pin is lane-only; the policy pins compare with the
per-scan lane, which derives row classes for policy.
Pinned on the fake and live on SQLite, Postgres and MySQL.
bplatz
added
enhancement
…e decision, and the scan lane's derived-class gap
aaj3f
left a comment
aaj3f
left a comment
There was a problem hiding this comment.
@bplatz the improvement/additions on top of #1777 make sense, and most of what I'd have (as a human) said about this are already described in the review on 1777. I'll provide Claude's detailed review below. If 1777 gets any significant rework and you want an additional review on this as a result, just let me know.
This delta is careful work, @bplatz, and most of what I went after held up under adversarial probing: the LIKE escaping (%, _, ! — the ! mutation goes red), the four class-policy polarities against the native evaluator (flipping IS NULL reddens the oracle), positional BIND admission, the F10 resolution rules with same_row sharing and Empty on differing templates, the 2001-key chunking, the UNION cap, MIN+MAX through separate aliases — and, the thing I checked hardest, the per-scan star split in rewrite.rs fires only where fusing already returned zero rows, so no previously-fused Iceberg star splits and it_graph_source_r2rml stays 66/66. The block cache never double-emits: each child batch is wholly seeded or wholly cached per branch, and the join key is the same function the seeded path uses.
The one thing I have to block on is small: the REGEX prefix extraction's metacharacter set omits the backslash, so REGEX(?n, "^A\\w") pushes LIKE 'A\w%' and silently drops every matching row on Postgres, SQLite and Trino — reproduced on the fake with the scan lane as oracle (lane [], scan ["n=Ada"]); MySQL is only saved because sql_string refuses backslashes. It's the one shape in the PR where the lane narrows the answer, and it's one character plus a pin.
Fold-ins alongside: the VALUES-above-cap story (the code declines the whole block; the test doc, commit body and sql.md:225 say it stays on the lane — one of them has to change); count_block/fetch_block errors aborting a query that #1777 answered seeded (map to Seeded, as the size-unreadable path already does); the cache cap being rows rather than bytes; and the vertical-partition join edges skipping the type check — the same class as #1777's join finding, which the fix there covers. Two are genuinely your call and I've named them as such inline: the unbounded COUNT(*) probe is your own #1784 (needs an inner-LIMIT node — a real scope decision), and the ±14h window buys nothing under the engine's naive-as-UTC semantics (temporal.rs:38-45; the docs page says so) while costing the most common timestamp shape its LIMIT and its grouped statement — either push exactly and make naive-as-UTC the contract, or document the hedge.
Also inherited: #1777's review asks for changes that rebase underneath this branch (the native-path planner and batch-copy regressions, the folded-OPTIONAL and join-type declines), and nothing ran in CI for this head — three levels deep in the stack — so every gate above is local. One pre-existing wrinkle to know about when running clippy locally: without --all-features, fluree-db-iceberg trips a dead-code error (env_switch_enabled, aws-only caller) that stops the changed crates from being linted at all; with -A dead_code the 23 pedantic denies are clean.
Adherence to repo commitments:
- Patterns/abstractions: ✔ extends #1777's lowering, plan and materializer structures (
output_names,lower_superset,resolve_parts,plan_for_cache,policy_pred); widening predicates correctly stay residual. - Performance (speed first, memory second):
⚠️ lane net positive (cache: 380 → 177 ms at 50k keys); unbounded COUNT (#1784) and the window's over-fetch are real costs; cache unbounded by bytes; Iceberg per-scan lane no regression; native untouched (the native regressions live in #1777). - Testing:
⚠️ 14 lane tests with fake goldens, routing stamps and the scan-lane oracle, mutation-verified; the REGEX backslash shape is unpinned; one case enshrines a known-wrong per-scan answer; the fake is two-valued onNOT; nothing ran in CI on this head. - Conventions: ✔ ten focused multi-line commits; docs updated; follow-ups #1780/#1784 filed by the author; fmt clean; clippy clean on the changed crates once the pre-existing iceberg dead-code item is allowed.
Verified locally at branch HEAD 235259613: it_sql_pushdown_lane 14/14, it_graph_source_r2rml 66/66 (+2 ignored), it_sql_graph_source 6/6 (--features sql,native,iceberg; live_bridge_* pass trivially with no bridge here); fluree-db-query/sql/tabular unit suites green; cargo fmt --all -- --check clean; cargo clippy -p fluree-db-api … --tests --examples -- -D warnings clean; throwaway ^A\\w case red at HEAD, removed; two mutations red then restored byte-for-byte; worktree clean.
Just be sure to get the backslash into the metacharacter set (with its pin) before this merges, and let's settle the two decisions inline — happy to talk through either.
| return None; | ||
| } | ||
| let (col, pattern) = self.string_column_and_literal(args)?; | ||
| let prefix = pattern.strip_prefix('^')?; |
There was a problem hiding this comment.
Blocking (correctness — the widened LIKE is not a superset). The metacharacter set here is r".^$*+?()[]{}|"; the backslash is missing. So REGEX(?n, "^A\\w") yields the literal prefix A\w and pushes LIKE 'A\w%' ESCAPE '!', which requires a literal backslash in the data. Same for ^\d, ^a\.b, ^ab\b, ^\pL.
Reproduced on the fake with the per-scan lane as oracle (case inserted into cases(), since removed): SELECT ?n WHERE { ?c ex:name ?n FILTER(REGEX(?n, "^A\\w")) } — statement carries "t0"."name" LIKE 'A\w%' ESCAPE '!', lane rows [], scan lane ["n=Ada"]. MySQL is saved by accident (sql_string declines any backslash, so the shape falls back); Postgres with standard_conforming_strings on, SQLite and Trino render the literal backslash and silently drop rows. This is the one shape in the PR where the lane narrows the answer.
Fix: add \\ to the set (or stop the literal prefix at the first metacharacter including backslash) and pin it with a ^A\\w-style case.
There was a problem hiding this comment.
Fixed in ccaa032: the backslash ends the prefix like the other metacharacters, so ^A\\w stays a plain residual. Pinned against the per-scan lane; with the character removed from the set the lane returns no rows for n=Ada.
| @@ -1249,6 +1510,13 @@ impl<'a> Lowerer<'a> { | |||
| if out_rows.is_empty() { | |||
There was a problem hiding this comment.
Should-fix (the prose and the code disagree — fold in now). An oversized in-block VALUES declines the whole block: this decline("VALUES too large to push") is propagated at :444-446 and fails the block in lower_block, so it falls back to the per-scan lane. The test that pins it says exactly that ("an oversized VALUES in the block declines the lane"), but its own doc comment, commit b71bfc4a9 ("the block still runs on the lane, the VALUES in the engine") and docs/graph-sources/sql.md:225 all say the block stays on the lane with the VALUES in the engine.
Either implement it — drop the key set from the statement and join in the engine the way outer bindings are — or fix the three pieces of prose. The IN-above-cap residual claim is correct (:1568).
There was a problem hiding this comment.
Prose fixed in db88f19: the block declines when its own VALUES exceeds the cap, the IN list above the cap stays a residual. The test doc and the docs page now say that.
| let mut joins = Vec::with_capacity(self.resolved.branches.len()); | ||
| for branch in 0..self.resolved.branches.len() { | ||
| if outgrown && matches!(self.caches[branch], BlockCache::Untried) { | ||
| self.caches[branch] = self.count_block(branch, &graph_ctx).await?; |
There was a problem hiding this comment.
Should-fix (fold in now — the fallback rule applied to the cache). count_block and fetch_block are optimization statements, but both sites here use ?, so a provider that cannot run that shape, or a transient error, now fails a query #1777 answered seeded. The "size unreadable" path already degrades to BlockCache::Seeded for a bad result shape; the error path should do the same with a debug log.
There was a problem hiding this comment.
Fixed in a16ba10: a failed count or fetch statement, or a failed page, leaves the branch Seeded with a debug log. Materializer errors still propagate since they would recur on the seeded path.
| /// seeded for the outer rows seen, is fetched once and joined in memory | ||
| /// in place of a seeded statement per outer batch (or, for a branch | ||
| /// nothing seeds, a re-run per batch). | ||
| async fn count_block(&self, branch: usize, ctx: &ExecutionContext<'_>) -> Result<BlockCache> { |
There was a problem hiding this comment.
Should-fix (performance) — already tracked as #1784, so naming the decision rather than re-raising it. Once the outer side passes one key set, every Untried branch is counted with a full SELECT COUNT(*) over the branch root with no cap, and the commit's own probe shows the cost: the 1M-row orders shape goes 25 ms → 43 ms (+72%) and then stays seeded, so the count bought nothing. The body says a bounded probe needs a derived-table node in the plan IR — that's a real scope call and #1784 is the right place for it. Two things worth knowing meanwhile: the docs' "an index-only scan on most tables" is optimistic, since the root carries the IS NOT NULL predicates from require(), so the count needs a covering index; and SELECT COUNT(*) FROM (SELECT 1 FROM … LIMIT cap+1) x is the shape that bounds it once an inner LIMIT node exists.
There was a problem hiding this comment.
Deferred to #1784, on the m3 branch: the derived-table node it needs exists there for sub-SELECT, so the probe becomes a count over a LIMIT cap+1 subquery. Agreed on the covering-index caveat; the docs wording will change with that fix.
| } | ||
|
|
||
| /// A branch's rows, fetched once and joined to every outer batch in memory. | ||
| struct CachedBlock { |
There was a problem hiding this comment.
Should-fix (memory, fold in now). The cache cap is rows, not bytes: rows: Vec<Vec<(VarId, Binding)>> plus a second copy of every join key in index. 100k rows × wide TEXT columns is unbounded, where the seeded path never held more than one page plus one child batch. A byte budget beside cache_max_rows, checked in the fetch loop that already checks rows.len() > max_rows and bailing to Seeded, bounds it. (FLUREE_SQL_PUSHDOWN_CACHE_ROWS is read once per query in resolve_block — that part is fine.)
There was a problem hiding this comment.
Fixed in a16ba10: a byte estimate from the materialized bindings (binding plus owned string heap) is checked beside the row cap in the fetch loop, 64 MiB by default, FLUREE_SQL_PUSHDOWN_CACHE_BYTES overrides. The join index still adds a copy of every key on top of that estimate.
| declined: None, | ||
| }, | ||
| Case { | ||
| name: "a predicate two maps provide, with no map providing the entity, declines", |
There was a problem hiding this comment.
Optional. This case pins the per-scan lane's duplicated answer (e=ada@example.org l=Ada twice, e=cy@example.org l=Cy twice) as the expected rows. A graph is a set of triples, so one solution per subject is the right answer; the duplication is a pre-existing per-scan bug, but pinning it here enshrines it as expected output. Maybe mark the oracle as known-wrong for this shape (and point at the per-scan fix) rather than asserting the duplicates — minor, but I'd rather it not become the thing a future fix has to "break".
There was a problem hiding this comment.
Leaving the pin as is on this branch. The m3 branch establishes set semantics in both lanes (a triple two maps mint alike is held once), and re-pins this shape to one solution per subject there.
| @@ -741,6 +749,15 @@ fn parse_pred(text: &str) -> Result<Pred, String> { | |||
| if let Some(c) = text.strip_suffix(" IS NULL") { | |||
There was a problem hiding this comment.
Optional (the fake is two-valued on NOT). Pred::Not(p) => !p.eval(t, r) at :722 (not in this diff), and Pred::In returns false on NULL, so NOT (NULL IN (…)) is TRUE in the fake but NULL — row dropped — in real SQL. The shipped policy predicate sidesteps it with IS NULL OR, so nothing is wrong today, but the fake cannot catch a 3VL mistake: flipping the IsNull branch to IsNotNull let the NULL-kind row through the fake where every real dialect would drop it (the scan-lane oracle caught it, the fake didn't). A three-valued eval would make the goldens honest on their own. Commenting here because :722 isn't in the diff.
There was a problem hiding this comment.
Done in b6110fa: Pred::eval is three-valued, NOT preserves NULL, AND/OR follow the truth tables, only TRUE keeps a row. Re-ran your flip and the fake now drops the NULL-kind row on its own.
| @@ -348,7 +348,11 @@ pub(crate) fn render_literal(lit: &Literal, ty: FieldType, dialect: SqlDialect) | |||
| Literal::Int(i) => is_numeric(ty).then(|| i.to_string()), | |||
There was a problem hiding this comment.
Optional, suspected rather than verified. MySQL's LIKE matches per collation element, and its docs note 'ch' LIKE 'c%' can be false under Czech-style contraction collations (ch is one element) — if so, LIKE 'c%' is not a superset of the byte prefix c there and the "a collation can only match more" argument fails. Not verified live (the words fixtures use the default collation). Cheap hardening: render the widened LIKE with BINARY on MySQL, as the lane already does for = and keys, which makes it exact there at no cost.
There was a problem hiding this comment.
Done in 9c35b0f: the pattern is marked BINARY on MySQL, pinned in the renderer tests and the live MySQL cases.
| continue; | ||
| } | ||
|
|
||
| if let Some(m) = mapping { |
There was a problem hiding this comment.
Praise, and the thing I checked hardest. This is the hunk that touches the Iceberg per-scan lane, so I traced whether any previously-fused star now splits: covered is false only when no single map has constant-predicate poms for every star member, rdf:type members never enter the coverage list (they're returned early as class_filter/type_var), and previously such uncovered stars materialized zero rows (F10) — so the split fires only where fusing was already wrong, covered stars still fuse, and it_graph_source_r2rml is 66/66. No Iceberg regression.
| @@ -144,19 +168,23 @@ impl R2rmlPolicyGate { | |||
| /// materialize each row's classes. | |||
There was a problem hiding this comment.
Praise. I went looking for a bypass in the class-policy reversal and didn't find one: static_verdicts decides per policy_classes (subclass-expanded upstream in build_policy_set) against a no-derived-class baseline that already includes the map's static rr:class, allows() merges tm_classes with row classes so a policy denying a map's constant class still yields Deny, and all four polarities (kind IS NULL OR NOT kind IN (…) / kind IN (…) / Empty / no predicate) match the native evaluator — flipping the IsNull branch turns the pinned oracle red.
…lane-m1 # Conflicts: # fluree-db-api/tests/it_sql_pushdown_lane.rs # fluree-db-query/src/r2rml/policy.rs # fluree-db-sql/src/plan_render.rs
The anchored-literal extraction stopped at `. ^ $ * + ? ( ) [ ] { } |` but
not at `\`, so `REGEX(?n, "^A\\w")` read `A\w` as a literal prefix and pushed
`LIKE 'A\w%'`, which needs a backslash in the data. Postgres, SQLite and Trino
rendered it and dropped every matching row; MySQL only escaped because
`sql_string` refuses any backslash. The one shape where the lane narrowed the
answer.
The backslash now ends the prefix like any other metacharacter, and the
pattern stays a plain residual. Pinned against the per-scan lane; with the
character removed from the set the lane returns no rows for `n=Ada`.
Two maps whose subject templates differ only in the placeholder's column name
— `order/{id}` over orders and `order/{order_no}` over shipments — mint the
same IRI whenever the values agree, and the per-scan lane joins them on it.
The lowering compared the template text, so it took the two for different
subjects and, since no row could carry every member, resolved the block to
Empty: zero rows, no round trip, where the per-scan lane returns the join.
The same text comparison declined the shape in `bind_var`'s template arm.
Subjects now compare by skeleton, the template with its placeholders
anonymized: one skeleton means one IRI space, and the placeholder columns
join pairwise. Each pair is vetted with `same_class` where the edge is
pushed, so a bigint key against a text key declines before `open` commits to
the lane rather than failing in the renderer. `same_row` keeps the exact
comparison it needs: one relation, one subject map column for column.
Two templates whose literal prefixes cannot begin one another (`order/…`
against `customer/…`) are still provably apart and keep the Empty shortcut;
templates the lane cannot relate either way, and a ref-object-map parent in
that position, now decline instead of answering empty.
The fixture gains `shipments` (a bigint key sharing the order template) and a
memo map minting order subjects from the text `order_ref`. Both shapes are
pinned against the per-scan lane and replayed live on SQLite, Postgres and
MySQL; with the text comparison restored the join shape declines.
… by bytes `count_block` and `fetch_block` are optimizations, but both propagated the provider's error, so a backend that cannot run the count, or a transient failure, failed a query the seeded path answers. A failed statement or page now leaves the branch `Seeded` with a debug log, as an unreadable count already did. Materializer errors still propagate: they would recur seeded. The cache was capped by rows only; a hundred thousand wide text rows is unbounded. A byte estimate from the materialized bindings — the binding plus the heap its string owns — is now checked alongside the row cap in the fetch loop, 64 MiB by default with `FLUREE_SQL_PUSHDOWN_CACHE_BYTES` to override. The join index adds a copy of every key on top of that estimate.
The lowering declines the whole block when the block's own VALUES exceeds the key-set cap, and the pinning test asserts exactly that; its doc comment and the docs page said the block stays on the lane with the VALUES in the engine. An IN list above the cap does stay a residual on the lane. The prose now says what the code does.
|
Merged main (5f1ff9e) and folded in the review's fix-now items:
Left as named: the unbounded COUNT probe stays on #1784; the naive-timestamp window is a product call I'll take separately. |
…ctly The lane reads a `timestamp` column with no zone as UTC when it builds the term, and the docs say so. The ±14h window pushed for a comparison against such a column was therefore pure over-fetch: it kept the exact filter in the engine, which cost the most common timestamp shape (Postgres `timestamp`, MySQL `DATETIME`) its pushed LIMIT and its grouped statement, and hedged against a naive-means-local reading the engine does not have. Naive-as-UTC is now the contract on both sides of the comparison. A dateTime literal is exact against a zoneless column, and the renderer writes it naive (`TIMESTAMP '2024-01-10 00:00:00.000000'`) where the column carries no zone, so `=`, `!=` and the orderings push as they do against a zoned column, and a LIMIT above them pushes too. SQLite keeps such a column as text in whatever format its writer used, which no rendering compares exactly, so the renderer still refuses a timestamp literal there and the lowering keeps the whole-day bounds, now around the literal's own day rather than the widened span. Letting the literal render on SQLite turned the per-scan lane's own pushed equality into no rows for a value stored without fractional seconds, which is what pins the refusal. The five naive-column cases are re-pinned against the per-scan lane and replayed live on all three backends.
A pushed LIKE is a superset of the engine's byte-prefix test only if the pattern matches at least those strings. MySQL matches LIKE per collation element, and under a contraction collation (`ch` as one element) `'ch' LIKE 'c%'` is false, so the widening could drop a row the engine would keep. The pattern is now marked `BINARY`, as string equality and key sets already are, which makes the LIKE exact there at no cost.
`Pred::eval` was two-valued, so `NOT (NULL IN (…))` was TRUE and a NULL operand under `NOT` kept a row that every real dialect drops. The shipped policy predicate sidesteps that with `IS NULL OR`, but the fake could not catch a mistake in it: flipping that `IS NULL` to `IS NOT NULL` let the NULL-kind row through the fake, and only the per-scan oracle noticed. A NULL operand now yields NULL, `NOT` preserves it, `AND` and `OR` follow SQL's truth tables, and only TRUE keeps a row. With the same flip the fake now drops the NULL-kind row as the real backends do.
|
Settled the two decisions and the optional items:
The duplicate-answer pin stays as is here; the m3 branch's set semantics replaces it. |
…lane-m3 Carries #1785 and the review round on it. Beyond the textual conflicts: the subject-skeleton rule is ported into `parts_of`, so a union-entity resolution over templates that provably never meet is skipped and one the lane cannot relate declines; `collect_col_eqs` descends into derived tables and UNION ALL branches so the column-class vet reaches every join the statement renders; the fake's new predicate arms (`AsciiOnly`, `ExprCmp`) are three-valued like the rest; the SUM-over-nullable top-k rule reads the aggregates `group_plan` now receives directly.
2 participants
Builds on #1777, now merged; the branch carries main. Everything here keeps the lane's contract: a statement is built only where the answer is exact or provably a superset with the exact check kept in the engine, terms are built in-engine, and anything else declines to the per-scan lane.
What lands
M1 leftovers
MIN/MAXof one variable in a grouped statement, each extreme read through its own materialization alias.BINDinside the block: the statement returns the columns the expression reads and the engine computes the value per row before residual filters. Positional admission; aBINDin aUNIONbranch orOPTIONALis not admitted.STRSTARTS/STRENDS/CONTAINSof a constant andREGEXanchored on a literal prefix push aLIKE … ESCAPE '!'; the filter stays a residual, soLIMITstays in the engine and a grouped query over it declines.rewrite.rsso the oracle is right.INlist above the key-set cap stays a residual on the lane, and a blockVALUESabove it declines the block to the engine (an oversized statement used to be a hardexecute_planerror); outer chunking and theUNIONcap are pinned.M2
timestampcolumn without a zone (every MySQLDATETIME, default Postgrestimestamp) is read as UTC when its term is built, and that is now the stated contract for comparisons too: anxsd:dateTimeliteral pushes exactly against it, rendered naive, soLIMITand grouped statements push on that shape. SQLite stores timestamps as text in whatever format was written, so it keeps whole-day bounds around the literal (>= '2024-01-10') with the exact check in the engine. (An earlier revision pushed a ±14h window instead; see the review thread.)FLUREE_SQL_PUSHDOWN_CACHE_ROWSandFLUREE_SQL_PUSHDOWN_CACHE_BYTESoverride the caps (0rows keeps every batch seeded). A count or fetch the provider cannot run leaves the branch seeded.rdf:type: the gate decides each(map, predicate)verdict per targeted class and the lane renders it on the class column (kind IS NULL OR NOT kind IN ('staff'), orkind IN ('guest')), where the whole block used to decline. A class constraint on such a map (?p a <kind/staff>) becomeskind = 'staff'.Found on the way
xsd:dateTime/date/timeas string-backed literals, soFILTER(?t = "…"^^xsd:dateTime)matched nothing on any graph source and either lane (ordering coerces, RDFterm-equal is a type error). Fixed inoperator.rs; pinned init_graph_source_r2rml.rr:classonly,?s a ?tnever emits a derived class). The class-constraint pin is lane-only for that reason.Review round
REGEXprefix ending in a backslash is no longer pushed as a literalLIKEprefix.order/{id}andorder/{order_no}mint one IRI space and join on their key columns, with the column-class vet at the edge. The text comparison had resolved that block to Empty, zero rows where the per-scan lane returns the join. Templates with disjoint literal prefixes keep the Empty shortcut; anything else declines.LIKEis markedBINARYon MySQL; the fake endpoint's predicates are three-valued.Numbers
sql_pushdown_lane_probe, 1M-row Postgres shop, median of 5, lane only:customers(100k rows)customersorders(1M rows)ordersThe last row is the cost of the unbounded count; bounding it needs a derived-table node in the plan IR.
Verification
kind = BINARY 'staff'on MySQL), the shared-subject join and the temporal equality.Follow-up: #1780 (per-scan lane column-derived classes)
Follow-up: #1784 (bounded block-size probe)