rewrite phrase SQL by remove JOIN with group by (doc_id, anchor) by cpegeric · Pull Request #24761 · matrixorigin/matrixone

cpegeric · 2026-06-01T12:20:29Z

What type of PR is this?

Which issue(s) this PR fixes:

What this PR does / why we need it:

In fulltext search, a boolean-mode query with many optional (OR) words
— e.g. 25 words becomes very slow. Tracing the path explains why:

 - With the ngram parser, a bare word is not a single word = '...' lookup. GenTextSql
 (pkg/fulltext/sql.go:242-260) re-tokenizes each word into overlapping 3-grams and routes it
 through SqlPhrase, which reconstructs the word with a positional self-join of its grams
 (a ⌈L/3⌉-way join for an L-char word).
 - In all-optional boolean mode, SqlBoolean emits one such phrase search per OR term,
 UNION ALL'd. So 25 OR words = 25 positional gram-reconstruction joins. The "many JOINs"
 are the per-OR phrase searches, not joins between words.

 The multi-pattern branch of SqlPhrase (pkg/fulltext/sql.go:569-613) builds an N-way join:

 WITH kw0 AS (SELECT doc_id,pos FROM idx WHERE word='g0'),
      kw1 AS (...), ... kwN AS (...)
 SELECT kw0.doc_id [,CAST(0 as int)] FROM kw0,...,kwN
 WHERE kw0.doc_id=kw1.doc_id AND kw1.pos-kw0.pos=off1 AND ...   -- residual positional filter
 [GROUP BY kw0.doc_id]   -- boolean only

 The planner can only treat doc_id as a clean equi-key; the positional part is a residual
 filter applied after a per-doc cross-product of each gram's positions. Cost is multiplicative
 in per-gram occurrences and grows with both gram count and word count.

 The same N-way-join generator (SqlPhrase) backs all phrase paths: NL/default phrase,
 boolean quoted phrase, and (via GenTextSql) every ngram OR term. So one rewrite fixes all of
 them. The index table is CLUSTER BY (word) (pkg/sql/plan/build_ddl.go:~1994), so the
 per-gram word= / prefix_eq lookups are already efficient sorted-range scans — the join is
 the problem, not the lookups.

 Intended outcome: make each phrase reconstruction linear, removing the per-OR join blow-up,
 with identical result semantics so the downstream consumer is untouched.

 The fix: replace the N-way positional join with anchor grouping

 A phrase occurrence is exactly a set of positions where every gram i sits at
 pos = anchor + offset_i for one common anchor (offset_i = the gram's distance from the
 first gram). Normalize each gram hit to its anchor (anchor = pos − offset_i), UNION ALL the
 per-gram hits, then group by (doc_id, anchor): a group containing all N grams is one phrase
 occurrence. Linear in the sum of per-gram posting lists; no cross-product; no N-relation
 join-order decision.

 This is provably equivalent to the current predicate: all slots share an anchor iff
 pos_i − pos_0 = offset_i for every i, which is the existing kwi.pos − kw0.pos = off_i.
 COUNT(*) = N is correct even with repeated words/grams (N = slot count): the group key
 includes anchor (not just doc_id), so multiple occurrences of the same word in one sentence
 fall into different anchor buckets, and each slot contributes at most one row per anchor (a
 word can't sit at two positions both equal to anchor + off_i). Worked example confirming this:
 query "a b a" over sentence "a b a b a" yields exactly two occurrences (anchors 0 and 2),
 matching the old join. (A COUNT(DISTINCT slot)=N variant with a per-slot tag is available if a
 provably-duplicate-immune form is ever wanted, but is not needed here.)

 Rewrite only the len(ps) > 1 branch of SqlPhrase (pkg/fulltext/sql.go:569-613). Build,
 for each pattern slot i (base = ps[0].Position, offset_i = ps[i].Position − base):

 SELECT doc_id, pos - <offset_i> AS anchor FROM <idxtbl> WHERE <cond_i>
 where cond_i is word = '<esc>' (TEXT) or prefix_eq(word,'<esc>') (STAR) — reusing the
 existing escape + *-suffix handling.

 Let innerUnion = <slot selects joined by " UNION ALL "> and proj = withIndex ? "doc_id, CAST(0 as int)" : "doc_id".

 - NL / default mode (one row per occurrence → preserves tf):
 SELECT <proj> FROM (<innerUnion>) anchors GROUP BY doc_id, anchor HAVING COUNT(*) = <N>
 - Boolean mode (collapse to one row per doc, matching today's GROUP BY kw0.doc_id):
 SELECT <proj> FROM (
   SELECT doc_id FROM (<innerUnion>) anchors GROUP BY doc_id, anchor HAVING COUNT(*) = <N>
 ) phrase GROUP BY doc_id

 The single-pattern branch (len(ps) == 1) and all boolean set-algebra
 (GenJoinSql/GenJoinPlusSql/GenSql OR/MINUS assembly) are unchanged — in particular we
 do not apply HAVING COUNT=N across OR terms, which would wrongly require all words.

qodo-code-review · 2026-06-01T12:20:34Z

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

mergify · 2026-06-02T08:17:27Z

Merge Queue Status

✅ Entered queue — 2026-06-02 08:17 UTC · Rule: main
✅ Checks started · in-place
❌ Checks failed
🚫 Left the queue — 2026-06-02 10:12 UTC · at c4babda4297f48d6b21247791f3d8c082c7d7118

This pull request spent 1 hour 55 minutes 27 seconds in the queue, with no time running CI.

Waiting for

All conditions

Reason

The merge conditions cannot be satisfied due to failing checks

Hint

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

mergify · 2026-06-02T11:29:19Z

rewrite phrase SQL by remove JOIN with group by (doc_id, anchor)

45897d7

cpegeric requested review from XuPeng-SH and fengttt as code owners June 1, 2026 12:20

cpegeric had a problem deploying to ci June 1, 2026 12:20 — with GitHub Actions Error

cpegeric temporarily deployed to ci June 1, 2026 12:20 — with GitHub Actions Inactive

cpegeric had a problem deploying to ci June 1, 2026 12:20 — with GitHub Actions Error

cpegeric temporarily deployed to ci June 1, 2026 12:20 — with GitHub Actions Inactive

matrix-meow added the size/M Denotes a PR that changes [100,499] lines label Jun 1, 2026

Merge branch 'main' into fulltext_pharse_rewrite

9cbc4c1

mergify Bot added the kind/bug Something isn't working label Jun 1, 2026

mergify Bot temporarily deployed to ci June 1, 2026 12:21 Inactive

mergify Bot had a problem deploying to ci June 1, 2026 12:21 Failure

mergify Bot temporarily deployed to ci June 1, 2026 12:21 Inactive

mergify Bot temporarily deployed to ci June 1, 2026 12:22 Inactive

cpegeric requested a review from iamlinjunhong June 1, 2026 12:34

Merge branch 'main' into fulltext_pharse_rewrite

c4babda

mergify Bot had a problem deploying to ci June 2, 2026 08:22 Failure

mergify Bot temporarily deployed to ci June 2, 2026 08:22 Inactive

iamlinjunhong approved these changes Jun 2, 2026

View reviewed changes

mergify Bot temporarily deployed to ci June 2, 2026 08:58 Inactive

mergify Bot added dequeued and removed queued labels Jun 2, 2026

mergify Bot temporarily deployed to ci June 2, 2026 10:24 Inactive

mergify Bot added the queued label Jun 2, 2026

mergify Bot merged commit fd6d9f1 into matrixorigin:main Jun 2, 2026
42 of 44 checks passed

mergify Bot temporarily deployed to ci June 2, 2026 11:29 Inactive

mergify Bot had a problem deploying to ci June 2, 2026 11:29 Failure

mergify Bot removed queued dequeued labels Jun 2, 2026

mergify Bot temporarily deployed to ci June 2, 2026 11:37 Inactive

mergify Bot temporarily deployed to ci June 2, 2026 11:55 Inactive

mergify Bot temporarily deployed to ci June 2, 2026 12:38 Inactive

mergify Bot temporarily deployed to ci June 2, 2026 13:01 Inactive

Navigation Menu

Uh oh!

rewrite phrase SQL by remove JOIN with group by (doc_id, anchor) - #24761

rewrite phrase SQL by remove JOIN with group by (doc_id, anchor)#24761
mergify[bot] merged 3 commits into
matrixorigin:mainfrom
cpegeric:fulltext_pharse_rewrite

cpegeric commented Jun 1, 2026 •

edited

Loading

Uh oh!

qodo-code-review Bot commented Jun 1, 2026

Uh oh!

mergify Bot commented Jun 2, 2026 •

edited

Loading

Uh oh!

mergify Bot commented Jun 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

cpegeric commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

Which issue(s) this PR fixes:

What this PR does / why we need it:

Uh oh!

qodo-code-review Bot commented Jun 1, 2026

Qodo reviews are paused for this user.

Uh oh!

mergify Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Reason

Hint

Uh oh!

mergify Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cpegeric commented Jun 1, 2026 •

edited

Loading

mergify Bot commented Jun 2, 2026 •

edited

Loading

mergify Bot commented Jun 2, 2026 •

edited

Loading