feat(search): add --lexical-gate to okfctl-search, preserving lexical recall

[cwest]

Closes #66.

Adds --lexical-gate to okfctl-search (off by default): gate embedding-only semantic results by a term-wise lexical match so an exact-identifier-shaped query surfaces a verbatim token the embedding blurs, while preserving question-shaped recall.

What the gate does

1. Run the semantic query wide (top-N=50, ≥ --k).
2. Intersect against a term-wise lexical match set (stopwords dropped, plurals/inflections stemmed), emitted in semantic order.
3. Append the lexical hits the semantic band missed, in lexical order — step 4 of the issue, the part that keeps this a win rather than a regression: a pure gate would discard a correct lexical hit outside the semantic top band.
4. Cut to --k.

Degrades to pure semantic (a no-op) when the query has no content terms (all-stopword) or a term matches more than 60% of the bundle (over-broad → no discriminating signal). The 60% fraction is calibrated against the 234-node reference corpus where the canonical over-broad term agent matches 172/234 = 73%; the rationale is co-located with the constant in cmd/okfctl-search/main.go.

Term matching resolves against the live bundle title+body at query time (mirroring the scoping filters and recency decay) — the vector index carries no prose. Composes with --path/--type/--tag (which constrain both the band and the appended tail) and with --half-life.

Reproduced mechanisms (fresh main @ ac8a7a5, real corpus, 234 nodes)

$ okfctl search --field body "how should an agent decide when to delegate work" .
   -> 0 hits          # phrase-wise body match; gate must be TERM-wise

body substring hit counts:  agent 172 | agents 100 | hash 18 | hashes 0
                            # raw substring is asymmetric; gate must stem

Acceptance evidence

Positive (real corpus, --embedder model2vec, q="chezmoi dotfiles"): the gate replaces semantic near-misses that do NOT contain the token with the nodes that actually do.

-- OFF --
0.4648  research/hermes-backup-restore-golden-path.md
0.3752  casey/hermes-backup-restore-golden-path-design.md
0.3461  research/go-vs-rust-okfctl-cli-spike.md        <- no "chezmoi"
0.3365  design/color-palette-collections-and-application.md  <- no "chezmoi"
0.3222  design/typography.md                            <- no "chezmoi"
-- ON --
0.4648  research/hermes-backup-restore-golden-path.md
0.3752  casey/hermes-backup-restore-golden-path-design.md
0.3032  infra/hermes-os-multi-service-production-reference.md  <- contains "chezmoi"
0.2967  casey/all-local-except-llm-hermes-stack.md            <- contains "chezmoi"
0.2922  casey/ai-design-tool-adoption.md                      <- contains "chezmoi"

Negative control (load-bearing) — the full question-shaped gold set (11 queries) scores no worse with the gate on than off, on both embedders:

[hash]      gate OFF: MRR=0.227 recall@5=0.273 | gate ON: MRR=0.227 recall@5=0.273
[model2vec] gate OFF: MRR=0.909 recall@5=0.909 | gate ON: MRR=0.909 recall@5=0.909

Zero movement — the degrade-to-semantic rule holds. (internal/search/eval_test.go TestEval_LexicalGate, gated on OKFCTL_EVAL_CORPUS + OKFCTL_TEST_MODEL_DIR.)

Second negative — gate off (the default) is byte-identical to main across every query shape tested (how should an agent…, colima docker desktop, chezmoi dotfiles, agent), compared binary-vs-binary on the real corpus.

Empty-term degrade — "how should the" with the gate on == gate off (real corpus, IDENTICAL).

Over-broad degrade — agent (73% of the corpus) with the gate on == gate off (real corpus, IDENTICAL).

Stemming symmetry — hash and hashes gate to overlapping sets (TestLexicalMatchSet_StemSymmetry, TestStem_PluralSymmetry).

Interaction — --lexical-gate with --path and with --half-life, and all three together: no panic, sane output (TestPlugin_LexicalGateInteractionSmoke); a lexical-only tail hit failing --path is not appended (TestPlugin_LexicalGatePathConstrainsTail).

Deliberate edge — a real content term matching zero nodes gates to empty (consistent with core lexical search), documented in TestGate_ZeroLexicalMatchIsEmpty rather than resolved silently.

Conformance gate (AGENTS.md, all three layers)

# Layer 1 — spec-conformance suite
go test ./internal/okf/ ./cmd/ -run Conformance -race     -> ok

# Layer 2 — full suite under -race; gofmt -l empty; go vet clean
gofmt -l .   -> (empty)
go vet ./... -> clean
go test ./... -race -count=1 -> all ok

# Layer 3 — real corpus (~/src/knowledge-base/bundles/knowledge), before/after
validate:            OK -> OK                (unchanged)
lint --strict:  1 finding -> 1 finding       (unchanged; same spec-version note)

This PR is a ranking overlay behind an opt-in flag; it touches no OKF-defined behavior, so the validate/lint counts are the control proving no drift — they did not move.

Ordering note

The issue preferred landing after #63 (scope filters) and #65 (recency decay). Both are already on main (ac8a7a5); the gate composes with them, no rebase needed.

Spec: docs/specs/2026-08-03-lexical-gate.md. Plan: docs/plans/2026-08-03-lexical-gate.md. Spec cited at v0.2 (§4.1 frontmatter).

[cwest]

The gate itself is in good shape. The term-wise stemmed matcher, the degrade-to-semantic rules for all-stopword and over-broad queries, and the lexical-tail preservation all read correctly, and the whole suite is green on this branch as it stands. The problem is that the branch no longer applies to main.

Main moved while this was in flight. #68 landed and reworked the scope filters into repeatable, negating flags, and it rewrote the same option-building region this PR edits. The result is a merge conflict in cmd/okfctl-search/main.go, and the conflict is the smaller half of the issue: main.go here is written against the old Filter shape. Detail inline.

What this needs before it can land:

1. Rebase onto current main. The Filter struct is now PathPrefixes/Types/Tags plus NotPathPrefixes/NotTypes/NotTags, all []string, and the scope flags are repeatable (StringArrayVar). The scalar path/type/tag vars and the Filter{PathPrefix:..., Type:..., Tag:...} literal in this PR won't compile against that. The resolution isn't just picking a side of the conflict marker; the surrounding filter-construction code has to move to the new API.

2. Re-run the combined-flags interaction check against the filters that actually exist now. The card asked for a gate-plus-scope smoke test, and this ran it against --path only. After the rebase, --not-path / --not-type / --not-tag exist too, and the gate composing with exclusion filters is genuinely new behavior worth a line of test.

3. Re-confirm the negative control and the gate-off byte-identical check on the rebased tree, since the pipeline underneath the gate changed.

The design work holds up; this is about landing it on top of what shipped since the branch was cut.

[cwest]

The rework itself is sound — I verified all of it at head 0d2cbb8. The rebase onto #68's Filter API landed correctly (main.go builds PathPrefixes/Types/Tags plus the NotPathPrefixes/NotTypes/NotTags exclusions through nonEmpty over the repeatable flags), the gate's step-4 lexical-tail preservation and degrade-to-no-op rules are correct, the new negating-filter coverage is real (NotPathConstrainsTail asserts the excluded node is gone and the wine hits remain), and I re-ran the load-bearing negative control against the 234-node corpus on both embedders: gate-on equals gate-off exactly (hash MRR 0.227 / recall@5 0.273; model2vec MRR 0.909 / recall@5 0.909). Full suite green under -race, gofmt/vet clean, real-corpus validate OK with lint --strict at one pre-existing finding.

The problem is external: main advanced again while this sat in review. #67 (bound recency decay) and #69 landed after the rebase, and #67 reworks the same decay-wiring region of cmd/okfctl-search/main.go this PR touches. The PR is now CONFLICTING against main — a test-merge of current main fails with conflicts in cmd/okfctl-search/main.go and cmd/okfctl-search/main_test.go (query.go auto-merges cleanly). It can't merge as-is.

This needs one more rebase onto current main, resolving the two conflicting files against #67's decay changes, then a re-run of the suite and the negative control on the rebased head. The gate logic doesn't need to change — only the flag/decay wiring in main.go and its test file where they collide with #67.

[cwest]

No changes needed.

The rebase onto current main is clean and the merge state is stable (mergeable, checks green, and main didn't move under the review this time). The earlier collision — #67's decay wiring overlapping this command's option-building block — is resolved by union: the var block, the DecayOptions construction, and the flag registrations now carry #67's --half-life/--decay-floor/--min-relevance alongside --lexical-gate, and all four show up in --help. The stale round-1 thread on the pre-#68 Filter literal is obsolete after the migration and is resolved.

The gate itself is unchanged from the last verified head and holds up: term-wise matching with stopwords dropped and a light stem so hash and hashes collapse; the intersection emitted in semantic order; the lexical tail the semantic band missed appended in path order, each hit carrying its real score; and a no-op degrade when the query has no content terms or a term matches more than 60% of the bundle. With no gate set the pipeline is byte-identical to a plain query. Filters run pre-ranking, so the appended tail is drawn only from already-filtered results — the exclusion filters constrain it too.

The test coverage is real, not decorative. The positive case asserts the gold node is semantically buried before checking the gate promotes it to rank 1; the step-4 tail test forces a match outside the band with WideN=1 and asserts it survives while the non-matcher is excluded; the degrade tests assert byte-identical output off-vs-on; the zero-lexical-match case documents the deliberate empty-result decision distinct from the empty-term no-op.

Verified on the rebased head: full suite green under -race, gofmt and vet clean, CGO_ENABLED=0 builds. The load-bearing negative control reproduces on the hash embedder against the real 234-node corpus — MRR 0.227 and recall@5 0.273, identical gate-on and gate-off across the 11-query question-shaped set — matching the pinned numbers. Spec citation is correct at v0.2, §4.1.