fix(cognition): drop stopwords in knowledge-pack relevance scoring (recall quality)

[rysweet]

Standing cognition goal — recall quality

Durable improvement toward the perpetual cognition goal (graph memory, recall
quality, distillation fact-yield, reasoner reliability). Code-only change; no
snapshot docs.

Problem

Planning-context knowledge-pack selection (knowledge_context::relevance_score,
consumed by enrich_planning_context → base_type_turn) scored packs by
DISTINCT WHOLE-WORD objective-token overlap after #4241 — but still counted
generic English stopwords (the, with, how, from, this, …) as
matches.

The episodic-recall tokenizer (memory_consolidation::tokenize_objective, PR-C
#2281) has always dropped these stopwords because they carry zero topical
signal. The two recall seams disagreed. Consequence: an objective that merely
shared a stopword with an off-topic pack's name/description inflated that pack's
relevance, creating spurious ties that could crowd a genuinely relevant pack out
of the top-MAX_PACKS_PER_OBJECTIVE (3) cut and inject off-topic knowledge into
the planning prompt — the SAME recall-precision crowd-out failure mode #4241
fixed, reached by a different route.

Fix

Align both recall seams on ONE stopword source of truth:

• Expose memory_consolidation::is_recall_stopword (the tokenizer now routes
  through it too — pure refactor, behaviour identical).
• Filter stopwords out of the objective token set in relevance_score.

Public surface unchanged (enrich_planning_context signature/contract intact);
this refines a private helper only.

Changed surfaces

• src/knowledge_context.rs — relevance_score (private fn) now drops
  stopwords; doc updated; 2 new unit tests.
• src/memory_consolidation/mod.rs — new pub(crate) is_recall_stopword
  predicate (single source of truth); tokenize_objective routes through it.
• tests/qa-scenarios/knowledge-context-stopword-relevance.yaml — new
  qa-team scenario (outside-in).

No user-facing surface changed (internal planning-context helper only), so no
end-user docs update is required; the shared-seam behaviour is documented in the
rustdoc of is_recall_stopword / relevance_score.

Evidence

Criterion 1 — qa-team scenario (gadugi-test):

• Wrote tests/qa-scenarios/knowledge-context-stopword-relevance.yaml.
• gadugi-test validate → ✓ Scenario "knowledge-context-stopword-relevance" is valid (1 valid, 0 invalid).
• gadugi-test run -s knowledge-context-stopword-relevance → ✓ Passed: 1 ✗ Failed: 0. Drives knowledge_context (11 tests), memory_consolidation (146), and base_type_turn (16), each test result: ok.

Criterion 2 — docs: No user-facing surface changed (private planning-context
helper); changed surfaces listed above with internal-only justification. Rustdoc
updated on the touched items.

Criterion 3 — quality-audit (≥3 SEEK→VALIDATE→FIX cycles, clean final):

• Cycle 1 (lowercasing correctness): relevance_score lowercases before the
  stopword filter; tokenize_objective lowercases up front. VALIDATE: targeted
  tests pass; clippy clean.
• Cycle 2 (single-source-of-truth refactor, no dangling refs): grep confirms
  TOKEN_STOPWORDS is referenced only by is_recall_stopword; both seams call
  the predicate. VALIDATE: memory_consolidation 146 tests pass (behaviour
  identical). FIX applied in an earlier cycle: clippy doc_lazy_continuation
  warning on the new doc list → fixed by adding blank lines / indentation.
• Cycle 3 (doc accuracy + CI-gate parity): rustdoc links resolve; fmt + clippy
  --all-targets -- -D warnings parity checked. VALIDATE: clean, zero findings.
• Final cycle clean: zero critical/high; zero medium correctness/security.

Criterion 4 — CI green (local equivalents):

• cargo fmt --all -- --check → clean (pre-commit gate passed).
• cargo clippy --release --no-deps -- -D warnings → SUCCESS (pre-commit gate).
• cargo clippy --all-targets --all-features --locked -- -D warnings → clean
  (pre-push gate passed).
• cargo test --all-features --locked --lib --no-fail-fast →
  test result: ok. 8665 passed; 0 failed; 7 ignored.
• Pre-push memory_ipc suite → 423 passed; 0 failed.
• CI will confirm the full matrix on this PR.

Criterion 6 — focused diff: 2 source files + 1 new qa scenario; no unrelated
edits (git diff --stat: src/knowledge_context.rs,
src/memory_consolidation/mod.rs).

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[rysweet]

CI 100% green (run 29622310725): pre-commit pass 12m2s (fmt + release clippy -D warnings + full cargo test + clippy --all-targets --all-features --locked -D warnings + minimal-binary build + coin-gym gate), install-real pass, e2e-dashboard pass, cargo-audit/deny/vet pass, npm-audit pass, GitGuardian pass. 0 failures — criterion 4 confirmed.