fix(platform): stop a search returning one passage twice

[Israeltheminer]

A search no longer spends two of its result slots on the same passage.

Closes #3061.

Why

The same text can sit in the corpus twice — a file uploaded as two documents, or a paragraph two documents share. Both copies match, and a bounded result set then says one thing twice while a different answer falls off the end. A caller asking for eight passages can receive seven distinct ones.

Observed as duplication on a deployment: two pairs of corpus rows with matching content hashes, chunk counts and character counts under different refs. The retrieval symptom is reasoned from the code rather than seen there, because both copies of each pair were orphaned and already hidden.

What changed

searchKnowledge drops a passage the caller has already been given, keeping the best-scoring one. Fusion has already sorted by score, so the first occurrence is the best.

Keyed on the passage, not the document. The issue proposed deduping by content_hash, which the corpus query does not select — plumbing it through the SQL, the row type and fusion would be a lot of moving parts for the same outcome. A result slot holds a passage, so that is what is compared. It catches the whole-document case too, since identical documents produce identical chunks, and it also catches a shared paragraph between otherwise different files.

Whitespace is normalized, so two copies that differ only in how their source wrapped lines still count as one.

Corpus is part of the key: a web page and a document saying the same thing are two findings, and only one is citable by URL.

Risk

It runs after the retrievability filter, deliberately. Deduping first could keep a copy the caller cannot read and drop the readable one — the gate would then remove what was kept, and the passage would vanish entirely. That is worse than showing it twice, and it is the case a reader of this code is most likely to get wrong. A test pins it.

A second chunk of the same document survives, because it is a different answer. Deduping per document would have lost it.

Tests

Five cases: a duplicate pair collapsed to the higher-scoring ref, two distinct passages from one document both kept, copies differing only in whitespace collapsed, the readable copy kept when the better-scoring one is denied, and identical text across different corpora kept separate.

Four deliberate breakages, all caught — including moving the dedupe ahead of the gate, which is the ordering the test exists for.

Scope

Does not stop duplicates entering the corpus. Identical content still becomes two rows, still costs storage, and the corpus writer still clones chunks for it rather than re-embedding. This only stops the duplication reaching a reader.

Gate: repo-wide typecheck, oxlint --type-aware, oxfmt, knip, SAST 0 findings, platform suite 75,817 passing.

[Israeltheminer]

Superseded by #3172, which carries the same fix against the current tree.

Both files this branch touches were deleted by #3125; convex/knowledge/search.ts is now backend/core/knowledge/search.ts. The fix transplants unchanged — same helper, same key, same placement after the retrievability filter.

Three assertions were added beyond this branch's two, because the mutation pass showed the original pair did not cover all four properties: the corpus is part of the key (so the same wording in a document and on a crawled page stays two answers), and the after-the-gate ordering needs its own case, since deduping first only bites when one copy is unreadable.

#3061 stays open until #3172 lands.

[Israeltheminer]

Closing so the queue reflects reality — #3176 replaces this and is open. Both of this branch's files were deleted by #3125, so it cannot be rebased. #3061 stays open and is closed by #3176.