Take the undirected close on the intersection and lift the predicate that blocked it - #101
Merged
Conversation
Filter placement puts a predicate at the first point its slots are bound. That is the right default almost everywhere, and on a closing expand it is also what keeps the intersection out: the physical compiler fuses a close into the expand below it only when the two are adjacent, so one filter in between costs the fusion and the plan falls back to a storage probe per candidate row. The ordered triangle is the shape that pays for it. `a.id < b.id` binds at the second hop, so the comparison lands between that hop and the close, and the close then walks every 2-path the comparison was about to reject. The close keeps a few hundred rows out of tens of thousands, so it is by far the cheaper of the two filters and the predicate belongs above it. Only a marked close moves anything. An unmarked one runs the binary probe either way, and there the predicate first is the better order. A lifted predicate cannot name the rel the close binds, since it was placed before that rel was bound, and it keeps its own optional group, so this is a reorder of two row filters and nothing else. The ldbc bench grows a phase for the shape, since nothing else in it puts a predicate in that position. SF1 p50 before and after, recorded run on each host: local M series 452.948 to 34.195 ms, gamingpc 373.309 to 30.307, server1 1312.932 to 131.796.
An undirected end over a rel whose two sides are the same node table reads two stored lists, and both engines took that as a reason to keep the close off the intersection: the optimizer never marked it and the fusion checked the direction and declined. So the undirected triangle built every 2-path and then asked storage whether each one closed, one probe per candidate row. Two lists per end is not a reason to decline. The probe side only answers whether an edge exists, so its two lists merge into one sorted set once per vector and the walk stays a single leapfrog. The seed side walks each list in turn, which is what the row by row expand it replaces would have counted: a pair joined both ways is two edges and closes the wedge twice. The mark now allows an undirected close when the rel is self referencing, and both executors carry the two sides through. The old engine's MultiwayIntersect grew a probe cache holding the union and the forward list, so an emitted rel still knows which way it was stored. The pipeline executor's Intersect holds Dirs rather than one Dir on each side, and a close the intersection cannot take now falls back to the semijoin in place rather than sending the whole query back to the old engine. On the 1k ER graph the undirected triangle drops from 27.59 to 10.65 ms in explain_analyze: the close stops producing 217664 rows for the predicate above it to throw away and produces 4188.
The ordered triangle phase the filter lift added had no budget line, so it was printing and nothing else. It gets one now, set the way the close above it is: high enough that server1 gates under its own service load, low enough that a plan falling back to the probe fails. The undirected close ceiling comes down 1200 to 400 on the back of the intersection. server1 measured 147 and 182 on two passes against 761 on main, so 400 leaves it the room its vCPU spread needs and still catches the old plan by a wide margin. Both hosts pass at these ceilings: gamingpc 30.659 ordered and 39.170 close, server1 114.445 and 181.965.
17 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An undirected triangle used to be the worst shape in the engine. Both halves of why are here.
The predicate between the expand and the close
Filter placement puts a predicate at the first point its slots are bound. On the ordered triangle that puts
a.id < b.idbetween the second hop and the close, and the close only fuses into the expand below it when the two are adjacent, so one filter in between costs the intersection and the plan goes back to a storage probe per candidate row. The close keeps a few hundred rows out of tens of thousands, so it is the far cheaper of the two filters and the predicate moves above it. Only a marked close moves anything, and a lifted predicate cannot name the rel the close binds because it was placed before that rel was bound.The undirected close
An undirected end over a rel whose two sides are the same node table reads two stored lists, and both engines took that as a reason to keep the close off the intersection: the optimizer never marked it and the fusion checked the direction and declined. Two lists per end is not a reason to decline. The probe side only answers whether an edge exists, so its two lists merge into one sorted set once per vector and the walk stays a single leapfrog. The seed side walks each list in turn, which is what the row by row expand it replaces counted, so a pair joined both ways still closes the wedge twice.
Both executors carry the two sides through. The old engine's MultiwayIntersect grew a probe cache holding the union and the forward list, so an emitted rel still knows which way it was stored. The pipeline executor's Intersect holds Dirs rather than one Dir on each side, and a close the intersection cannot take now falls back to the semijoin in place instead of sending the whole query back to the old engine.
Numbers
LDBC SF1, p50 in ms, second run on each host since the run right after a build is never the recorded one.
Ordered triangle, before and after the lift. This phase is new, nothing else in the bench puts a predicate in that position.
Undirected close, main and the branch interleaved on the same host in the same session.
The full ldbc gate passes on both gate machines with the new ceilings: gamingpc ordered 30.659 and close 39.170, server1 114.445 and 181.965. server1 is the box that sets the ceilings and it swings, 147 on one pass and 182 with a 218 max on the next, so 400 is the spread and not slack.
The plain directed triangle is the one number that did not move the same way everywhere. Locally the branch runs 8.740 against main's 11.174, but on gamingpc the interleave read 6.708 and 6.775 on the branch against 6.422 and 6.425 on main, a few percent the other way that reproduced on both passes. That query does not go through either change, so this is build to build layout on a 6 ms measurement rather than the plan, but it is what the runs said.
In graph-bench,
micro-triangle-undirectedon the 1k ER graph was the only query where zu lost to ladybug. In-process p50, same run of the harness: 5.72 ms against ladybug's 14.56, where it was 22.45 against 13.69. The same query on the power law graph goes from 2.17 against 6.32 to 0.69 against 6.58.Gates
Full ldbc gate green on gamingpc and server1, and
cargo test --workspace, fmt and clippy are clean.