✨ feat(api): expose the lexical gate on /api/v1/search - #81
Conversation
GET /api/v1/search could not turn on the #66 lexical gate: the handler read q, k, path/type/tag (+ not-*), half_life, decay_floor and min_relevance but never set opts.LexicalGate, so every spelling of a gate param returned the byte-identical ungated body with HTTP 200. A caller who tuned a query with --lexical-gate on the CLI and moved it to the fast HTTP path silently got a different ranked list than the one they validated, with nothing in the response to say so. On the 234-node real corpus the gate alters the ranked list on 6 of 10 sweep queries via the CLI but 0 of 10 via the API — the same merge-order divergence shape as the #74/#76 decay-floor gap. - Promote the gate's shared surface into internal/search so both the CLI and the HTTP handler build identical options by construction, not by duplicated literals: LexicalGateWideN (50), LexicalGateOverBroadFraction (0.60), BundleTexts, and a BuildLexicalGate(bundle, query) constructor. The CLI drops its private constants and bundleTexts and calls the shared builder. - Accept a boolean gate param on the endpoint, both separator spellings (lexical_gate canonical, lexical-gate) since the separator-convention card has not merged; a non-boolean value is a 400 in the house wording, never a silently-ignored 200. Add the gate term to the needBundle guard so an ungated request stays byte-for-byte Query(). - Retain the resident live bundle alongside the store/meta cache, invalidated on the same index-modtime signal, so a gated request reads it from cache and the single-disk-load invariant still holds (N gated requests, one load). Adds a table-driven cross-surface parity test (the durable fix) driving both the HTTP handler and search.QueryWith through gate on/off, filters, half_life, decay_floor, min_relevance and combinations, asserting identical ranked paths+scores+snippets. This is what would have caught this gap and the #74/#76 decay-floor divergence. Env-guarded, it also runs over the real 234-node corpus and asserts the gate moves "hormesis training stress" the same way on both surfaces (dropping the no-overlap node, promoting the lexically-matched one into the tail). Real-corpus counts, bundle bundles/knowledge (234 nodes), hash embedder: gate-alters-ranking CLI 6/10, API 0/10 before -> API 6/10 after; validate 0->0, lint --strict 1->1 (a pre-existing per-node okf_spec_version finding, untouched). Resident-index latency for the worked query at k=5: ungated ~0.6 ms, gated ~56 ms — the gate re-tokenises the live prose per request as the CLI does, so gated queries are NOT free on the fast path. §4.1 (semantic query is a consumer concern; the gate is a permitted result-shaping choice, so the floor rejects no bundle for its absence). Closes #78
cwest
left a comment
There was a problem hiding this comment.
No changes needed.
The gate now reaches the HTTP surface, and the way it gets there is the right shape: both surfaces build their options from one search.BuildLexicalGate constructor over the shared LexicalGateWideN/LexicalGateOverBroadFraction constants, so the CLI and the API can't drift by a duplicated literal. The handler feeds the same q to the engine and to the gate builder, matching the CLI's semantic, and the needBundle guard picks up || lexicalGate so an ungated request stays byte-for-byte the old Query() path. Retaining the live bundle alongside the store/meta cache under the same modtime invalidation keeps the single-disk-load invariant intact under gating.
I ran the worked example against the live 234-node corpus over real HTTP, not just the test harness: ungated, rank 2 is casey/child-first-account-and-ipad-parental-controls.md (0.3826, no query-term overlap); with the gate on it's dropped and research/global-workspace-j-space-language-models.md is promoted into the tail. Non-boolean is a 400 in the house wording on both separators, gate-omitted and gate-false are byte-identical, both separator spellings produce identical rankings, and the underscore form wins when both are present. validate is clean and lint --strict holds at its one pre-existing per-node okf_spec_version finding, untouched.
The cross-surface parity test is the durable part — a table over gate on/off, filters, half_life, decay_floor, min_relevance and combinations, driven through both the handler and the CLI oracle from the same shared builders. That's the check that would have caught this gap and the earlier decay-floor divergence, and it can't pass while a literal on one side drifts from the other.
Closes #78.
Summary
GET /api/v1/searchcould not turn on the #66 lexical gate — the handlernever set
opts.LexicalGate, so every spelling of a gate param returned thebyte-identical ungated body with HTTP 200. This wires a boolean gate param to
the same
LexicalGateOptionsthe CLI builds, promotes the gate's sharedsurface into
internal/searchso the two surfaces agree by construction,and adds the durable cross-surface parity test that would have caught this
(and the same-shape #74/#76 decay-floor divergence).
What changed
internal/search—LexicalGateWideN(50),LexicalGateOverBroadFraction(0.60),BundleTexts, and aBuildLexicalGate(bundle, query)constructor. The CLI drops its privateconstants +
bundleTextsand calls the shared builder; the HTTP handlercalls the identical builder. No duplicated literals to drift.
internal/apiserver/search.go— acceptlexical_gate(canonical) andlexical-gate(both separators, since the separator-convention card 🐛 fix(api): reject unknown query parameters in /api/v1/search instead of ignoring them #77 hasnot merged). Non-boolean value → 400
"lexical_gate" must be a boolean.Add the gate term to the
needBundleguard so an ungated request staysbyte-for-byte
Query(). Retain the resident live bundle alongside thestore/meta cache, invalidated on the same index-modtime signal, so a gated
request reads it from cache (single-disk-load invariant preserved).
handler and
search.QueryWiththrough gate on/off, filters, half_life,decay_floor, min_relevance and combinations, asserting identical
paths+scores+snippets. Env-guarded real-corpus variant asserts the gate
moves the worked query identically on both surfaces.
Conformance run output (not a claim)
Real corpus (
bundles/knowledge, 234 nodes, hash embedder)Layer 3 — the layer fixtures cannot substitute for.
Gate-alters-ranking count (10-query sweep, k=5):
Worked example
q=hormesis training stress&k=5, over HTTP:validate / lint --strict, before → after:
Per-query gated latency, resident index, q above, k=5 (mean of 50):
The gate re-tokenises the live bundle prose per request (as the CLI does), so
gated queries are NOT free on the resident-index fast path — recorded here
rather than documented away. The single-disk-load invariant still holds (the
cost is CPU tokenisation, not an extra index load).
Spec
§4.1 semantic query, upstream v0.2 SPEC.md. The gate is a permitted
result-shaping choice a consumer may offer; the floor rejects no bundle for
its absence. No spec change.
Notes
separator spellings are accepted from the start.
counts above are the measured current corpus.