Skip to content

✨ feat(api): expose the lexical gate on /api/v1/search - #81

Merged
cwest merged 1 commit into
mainfrom
wt/t_a693d5bd
Aug 3, 2026
Merged

✨ feat(api): expose the lexical gate on /api/v1/search#81
cwest merged 1 commit into
mainfrom
wt/t_a693d5bd

Conversation

@cwest

@cwest cwest commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Closes #78.

Summary

GET /api/v1/search could not turn on the #66 lexical gate — the handler
never set opts.LexicalGate, so every spelling of a gate param returned the
byte-identical ungated body with HTTP 200. This wires a boolean gate param to
the same LexicalGateOptions the CLI builds, promotes the gate's shared
surface into internal/search so the two surfaces agree by construction,
and adds the durable cross-surface parity test that would have caught this
(and the same-shape #74/#76 decay-floor divergence).

What changed

  • Shared gate surface in internal/searchLexicalGateWideN (50),
    LexicalGateOverBroadFraction (0.60), BundleTexts, and a
    BuildLexicalGate(bundle, query) constructor. The CLI drops its private
    constants + bundleTexts and calls the shared builder; the HTTP handler
    calls the identical builder. No duplicated literals to drift.
  • internal/apiserver/search.go — accept lexical_gate (canonical) and
    lexical-gate (both separators, since the separator-convention card 🐛 fix(api): reject unknown query parameters in /api/v1/search instead of ignoring them #77 has
    not merged). Non-boolean value → 400 "lexical_gate" must be a boolean.
    Add the gate term to the needBundle guard so an ungated request stays
    byte-for-byte Query(). Retain the resident live bundle alongside the
    store/meta cache, invalidated on the same index-modtime signal, so a gated
    request reads it from cache (single-disk-load invariant preserved).
  • Parity test (the durable fix) — table-driven, drives BOTH the HTTP
    handler and search.QueryWith through gate on/off, filters, half_life,
    decay_floor, min_relevance and combinations, asserting identical
    paths+scores+snippets. Env-guarded real-corpus variant asserts the gate
    moves the worked query identically on both surfaces.

Conformance run output (not a claim)

$ gofmt -l .            # (empty)
$ go vet ./...          # (clean)
$ go test ./... -race
ok  cmd  cmd/okfctl-api  cmd/okfctl-search  internal/apiserver
    internal/okf  internal/okfconfig  internal/plugin  internal/search   # all ok

$ go test ./internal/okf/ ./cmd/ -run Conformance -race
ok  github.com/cwest/okfctl/internal/okf
ok  github.com/cwest/okfctl/cmd

Real corpus (bundles/knowledge, 234 nodes, hash embedder)

Layer 3 — the layer fixtures cannot substitute for.

Gate-alters-ranking count (10-query sweep, k=5):

surface count
CLI (after) 6 / 10
API (before) 0 / 10 — gate unreachable, the bug
API (after) 6 / 10 — surfaces now agree

Worked example q=hormesis training stress&k=5, over HTTP:

ungated rank 2 = casey/child-first-account-and-ipad-parental-controls.md  (no query-term overlap)
gated          = casey/... DROPPED,
                 research/global-workspace-j-space-language-models.md PROMOTED into the tail

validate / lint --strict, before → after:

validate      0 -> 0
lint --strict 1 -> 1   (pre-existing per-node okf_spec_version finding on
                        infra/serverless-gpu-compute-modal-vs-gke.md; untouched)

Per-query gated latency, resident index, q above, k=5 (mean of 50):

ungated  ~0.6 ms
gated    ~56 ms

The gate re-tokenises the live bundle prose per request (as the CLI does), so
gated queries are NOT free on the resident-index fast path — recorded here
rather than documented away. The single-disk-load invariant still holds (the
cost is CPU tokenisation, not an extra index load).

Spec

§4.1 semantic query, upstream v0.2 SPEC.md. The gate is a permitted
result-shaping choice a consumer may offer; the floor rejects no bundle for
its absence. No spec change.

Notes

GET /api/v1/search could not turn on the #66 lexical gate: the handler read
q, k, path/type/tag (+ not-*), half_life, decay_floor and min_relevance but
never set opts.LexicalGate, so every spelling of a gate param returned the
byte-identical ungated body with HTTP 200. A caller who tuned a query with
--lexical-gate on the CLI and moved it to the fast HTTP path silently got a
different ranked list than the one they validated, with nothing in the
response to say so. On the 234-node real corpus the gate alters the ranked
list on 6 of 10 sweep queries via the CLI but 0 of 10 via the API — the same
merge-order divergence shape as the #74/#76 decay-floor gap.

- Promote the gate's shared surface into internal/search so both the CLI and
  the HTTP handler build identical options by construction, not by duplicated
  literals: LexicalGateWideN (50), LexicalGateOverBroadFraction (0.60),
  BundleTexts, and a BuildLexicalGate(bundle, query) constructor. The CLI
  drops its private constants and bundleTexts and calls the shared builder.
- Accept a boolean gate param on the endpoint, both separator spellings
  (lexical_gate canonical, lexical-gate) since the separator-convention card
  has not merged; a non-boolean value is a 400 in the house wording, never a
  silently-ignored 200. Add the gate term to the needBundle guard so an
  ungated request stays byte-for-byte Query().
- Retain the resident live bundle alongside the store/meta cache, invalidated
  on the same index-modtime signal, so a gated request reads it from cache and
  the single-disk-load invariant still holds (N gated requests, one load).

Adds a table-driven cross-surface parity test (the durable fix) driving both
the HTTP handler and search.QueryWith through gate on/off, filters, half_life,
decay_floor, min_relevance and combinations, asserting identical ranked
paths+scores+snippets. This is what would have caught this gap and the
#74/#76 decay-floor divergence. Env-guarded, it also runs over the real
234-node corpus and asserts the gate moves "hormesis training stress" the
same way on both surfaces (dropping the no-overlap node, promoting the
lexically-matched one into the tail).

Real-corpus counts, bundle bundles/knowledge (234 nodes), hash embedder:
gate-alters-ranking CLI 6/10, API 0/10 before -> API 6/10 after; validate
0->0, lint --strict 1->1 (a pre-existing per-node okf_spec_version finding,
untouched). Resident-index latency for the worked query at k=5: ungated
~0.6 ms, gated ~56 ms — the gate re-tokenises the live prose per request as
the CLI does, so gated queries are NOT free on the fast path.

§4.1 (semantic query is a consumer concern; the gate is a permitted
result-shaping choice, so the floor rejects no bundle for its absence).

Closes #78
@cwest
cwest marked this pull request as ready for review August 3, 2026 14:43

@cwest cwest left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No changes needed.

The gate now reaches the HTTP surface, and the way it gets there is the right shape: both surfaces build their options from one search.BuildLexicalGate constructor over the shared LexicalGateWideN/LexicalGateOverBroadFraction constants, so the CLI and the API can't drift by a duplicated literal. The handler feeds the same q to the engine and to the gate builder, matching the CLI's semantic, and the needBundle guard picks up || lexicalGate so an ungated request stays byte-for-byte the old Query() path. Retaining the live bundle alongside the store/meta cache under the same modtime invalidation keeps the single-disk-load invariant intact under gating.

I ran the worked example against the live 234-node corpus over real HTTP, not just the test harness: ungated, rank 2 is casey/child-first-account-and-ipad-parental-controls.md (0.3826, no query-term overlap); with the gate on it's dropped and research/global-workspace-j-space-language-models.md is promoted into the tail. Non-boolean is a 400 in the house wording on both separators, gate-omitted and gate-false are byte-identical, both separator spellings produce identical rankings, and the underscore form wins when both are present. validate is clean and lint --strict holds at its one pre-existing per-node okf_spec_version finding, untouched.

The cross-surface parity test is the durable part — a table over gate on/off, filters, half_life, decay_floor, min_relevance and combinations, driven through both the handler and the CLI oracle from the same shared builders. That's the check that would have caught this gap and the earlier decay-floor divergence, and it can't pass while a literal on one side drifts from the other.

@cwest
cwest merged commit 69c3673 into main Aug 3, 2026
1 check passed
@cwest
cwest deleted the wt/t_a693d5bd branch August 3, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ feat(api): expose the lexical gate on /api/v1/search — --lexical-gate is unreachable over HTTP

1 participant