fix(cognition): whole-word, distinct-token knowledge-pack relevance scoring (recall quality) by rysweet · Pull Request #4241 · rysweet/Simard

rysweet · 2026-07-17T00:54:58Z

Summary

Advances the standing cognition-improvement goal on the recall-quality axis.

Planning-context knowledge-pack selection (knowledge_context::relevance_score, consumed by the reasoner's planning path enrich_planning_context → base_type_turn) scored packs by raw-substring keyword overlap with the objective. Two defects degraded recall precision:

Substring, not whole-word. A short objective token matched when merely embedded in an unrelated pack word — go inside "category"/"algorithm", test inside "latest", own inside "download". These spurious hits inflated an off-topic pack's relevance and could crowd a genuinely relevant pack out of the MAX_PACKS_PER_OBJECTIVE (top-3) cut, injecting off-topic knowledge into the planning prompt and hurting reasoner reliability.
Repetition double-counted. A word repeated in the objective counted once per occurrence, letting a verbose objective that restates one term distort the ranking.

Fix

Score now counts distinct objective tokens (len ≥ 2) that appear as whole words in the pack's word set. This aligns the seam with the word-boundary matching policy already adopted by memory_consolidation::classifier (which explicitly replaced a "naive substring scan") and fact_reliability::distinct_informative_words — one consistent tokenization philosophy across the cognition stack.

Behavior is a strict precision improvement: every legitimate whole-word match still scores; only spurious embedded-substring and repeated-token inflation are removed.

Evidence

Criterion 1 — qa-team scenario written + tests run

Added outside-in scenario tests/qa-scenarios/knowledge-context-whole-word-relevance.yaml (modeled on the peer classifier-failure-signal-fidelity.yaml format; YAML parse-validated). It drives the underlying cargo tests:
- cargo test --locked --lib knowledge_context → 9 passed; 0 failed (incl. 2 new tests: relevance_score_requires_whole_word_match, relevance_score_counts_distinct_tokens_only).
- cargo test --locked --lib base_type_turn (no-regression on the consuming planning path) → 16 passed; 0 failed.
- cargo test --locked --test knowledge (integration) → 15 passed; 0 failed.
Note: the gadugi-test binary (rysweet/gadugi-agentic-test) is not installed in this build environment, so gadugi-test validate/run could not be executed directly; the scenario's driven cargo commands were run manually and pass.

Criterion 2 — docs

No user-facing surface changed. enrich_planning_context's public signature and documented contract ("select relevant packs by keyword overlap") are unchanged; this refines a private helper (relevance_score). The precision refinement is fully documented in rustdoc on the function. No docs/ page references the substring-vs-whole-word internal detail. → internal-only justification.

Criterion 3 — quality audit

SEEK→VALIDATE→FIX self-review across: correctness of whole-word semantics, preservation of all pre-existing tests, no other callers of relevance_score (grep-verified: only enrich_planning_context + tests), ruthless-simplicity (no fuzzy/prefix matching added), and consistency with existing codebase tokenization policy. Zero critical/high/medium correctness or security findings on the final pass.

Criterion 4 — CI

Local gates mirroring CI all green: pre-commit cargo fmt --all -- --check + cargo clippy --release --no-deps -- -D warnings; pre-push race-subset release tests + cargo clippy --all-targets --all-features --locked -- -D warnings. Awaiting CI to confirm 100% green.

Criterion 6 — focused diff

One source file (src/knowledge_context.rs) + one new qa-scenario. No unrelated edits.

Durability

Durable code improvement (not a snapshot doc), consistent with the perpetual goal's mandate.

…coring (recall quality) Planning-context knowledge-pack selection (`knowledge_context::relevance_score`) scored packs by RAW-SUBSTRING keyword overlap with the objective. Two defects degraded recall precision on the reasoner's planning path (`enrich_planning_context`, consumed by `base_type_turn`): - Substring, not whole-word: a short objective token matched when merely *embedded* in an unrelated pack word — `go` inside "category"/"algorithm", `test` inside "latest", `own` inside "download". Spurious hits inflated an off-topic pack's relevance and could crowd a genuinely relevant pack out of the `MAX_PACKS_PER_OBJECTIVE` cut, injecting off-topic knowledge into the planning prompt. - Repetition double-counted: a word repeated in the objective counted once per occurrence, letting a verbose objective distort the ranking. Score now counts DISTINCT objective tokens (len >= 2) that appear as WHOLE WORDS in the pack's word set — aligning this seam with the word-boundary policy already adopted by `memory_consolidation::classifier` and `fact_reliability`. Public surface unchanged (`enrich_planning_context` signature/contract intact); this refines a private helper only. New unit tests pin the whole-word and distinct-token contract; a qa-scenario drives them outside-in. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-07-17T01:02:49Z

📊 Coverage Summary

Generated by cargo llvm-cov --workspace --summary-only (nightly, excluding test files)

Module	Lines	Covered	Coverage
Total	184093	153598	83.4%

_{Coverage data from CI run. Test files matching tests?/ are excluded from line counts.}

…lity / fact-yield) (#4245) The episode-ingestion classifier (`memory_consolidation::classifier::classify`) decides, at every `store_episode` intake site, whether an event is operational noise (DROP), low-value bookkeeping (DOWN-SCOPE), or a meaningful episodic worth full-importance storage (STORE). Distillation later mines the STORED, high-importance episodes into semantic facts, so a misclassification here propagates straight into recall top-k. Rule 3's durable-completion signal scored a single token with a raw-substring `content_lc.contains("merged")`. That fired inside the git-status vocabulary `unmerged` / `submerged` — which name the OPPOSITE of a completed merge (an outstanding, un-completed one) — promoting benign/negative bookkeeping to a durable `ActionCompleted` episode at 0.7 importance. Distillation would then mine that phantom completion into a phantom "action completed" fact, polluting recall top-k and dragging recall precision down. The `merged` signal now matches at WORD BOUNDARIES via a new `contains_word` helper, aligning this seam with the word-boundary policy already adopted by the same classifier's failure-signal pass (#4080) and the knowledge-pack relevance scorer (#4241). `unmerged`/`submerged` down-scope to the operational tier; a genuine whole-word `merged` still classifies as a durable completion, so no real completion changes disposition. The multi-word phrases (`opened pr`, `pull request`) cannot embed in a single word, so they remain substring checks. Internal surface only — `classify` signature/contract unchanged, no CLI/operator surface touched. New unit tests pin both the negative (`unmerged`/`submerged`) and positive (whole-word `merged`) contract; a qa-team gadugi scenario drives them outside-in. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…g (recall quality) (#4250) Planning-context knowledge-pack selection (`knowledge_context::relevance_score`, consumed by `enrich_planning_context` → `base_type_turn`) scored packs by EXACT whole-word overlap with the objective. A genuinely relevant pack was therefore missed whenever the objective and the pack name/description differed only by a regular English inflection — `container` vs `containers`, `image` vs `images`, `library` vs `libraries`, `category` vs `categories`. Such a pack scored below its true relevance (often zero), so it could be crowded out of the top-N (`MAX_PACKS_PER_OBJECTIVE`) cut and its knowledge never reached the planning prompt, deflating recall and thus reasoner reliability. Relevance now matches an objective token against a pack word exactly OR via a conservative lemma variant: the regular plural forms (`+s`/`+es`, and the singular obtained by stripping a trailing `-s`/`-es`) and the `-y`↔`-ies` pair. Folding is deliberately additive over — never a regression of — the whole-word rule adopted in #4241: - each generated variant must clear `MIN_TOKEN_LEN`, so a short token cannot fold onto a one-character fragment; and - a variant match only fires when the variant is ACTUALLY a word in the pack, so non-plural words that merely end in `s`/`es`/`y` (`class`, `focus`, `status`, `hierarchy`) still match themselves and are never mangled onto an unrelated pack word. Public surface unchanged (`enrich_planning_context` signature/contract intact); this refines two PRIVATE helpers only (`relevance_score`, new `token_matches_pack`). New unit tests pin the fold-both-directions, `-y`↔`-ies`, no-mangle, and whole-word-preserved contract; a qa-scenario drives them outside-in. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…4257) `LibraryCognitiveMemory::recall_episodes_ranked` gated the library's multi-signal ranked episodic recall to "keyword-relevant" episodes with a raw-substring test (`content.to_lowercase().contains(kw)`). A query token matched wherever it was *embedded* — including the interior or suffix of an unrelated content word (`act` in "reactor"/"contract", `test` in "latest", `own` in "download") — floating off-topic episodes into the ranked set that feeds the OODA cycle's working context (via `memory_consolidation::prepare_context`) and degrading recall precision. Replace the inline substring gate with a word-boundary PREFIX gate: two new helpers `tokenize_words` (split on non-alphanumeric runs, lowercase, distinct) and `shares_word_prefix` (a query token must be a prefix of a whole word in the episode content). Prefix — not pure whole-word equality — is deliberate: it drops interior/suffix noise while PRESERVING inflectional recall the live path depends on (`deploy` still recalls "deployed"/"deploys"). This aligns episodic recall with the word-boundary relevance policy already used by `knowledge_context::relevance_score`, `memory_consolidation::classifier`, and `fact_reliability` (PR #4241 lineage). The new gate is a strict subset of the old one, so only the intended interior/suffix matches are newly excluded; every whole-word and inflectional recall is preserved. Scope is limited to `recall_episodes_ranked`; `search_episodes_by_keywords` keeps its exact-substring + `.take(limit)` semantics, which its exact-marker callers in `reflection_lessons.rs` (`occurrence_already_reflected`, limit=1) depend on. Tests: 7 helper unit tests + 5 ranked-episodic integration tests (interior/ suffix rejection, word-boundary acceptance, inflectional recall, punctuation folding, compressed-source UNION backfill gated). qa-team scenario `episodic-recall-word-boundary-relevance.yaml` drives the contract. Doc updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…recision + fact-yield) `LibraryCognitiveMemory::search_facts` — the fact-recall seam the OODA turn/context path (`base_type_turn::prepare_turn_context`) and every natural-language fact caller reach — delegated matching to the upstream library, which matches each query token as a RAW case-insensitive SUBSTRING of a fact's concept OR content. A clean natural-language token therefore floated facts in on the INTERIOR/SUFFIX of an unrelated word. Empirically (live in-memory backend), for content "the reactor overheated": * search_facts("act") -> matched "re(act)or" and "artif(act)" [interior] * search_facts("own") -> matched "d(own)load" [interior] * search_facts("test") -> matched "la(test)" [suffix] while whole-word ("reactor"), prefix ("react"), and inflectional ("deploy" -> "deployed") matches were legitimate. Those off-topic facts crowd the CAPPED working-context recall the OODA cycle feeds to reasoning, dragging fact recall precision — and effective distillation fact-yield — down. This is the same defect the EPISODIC recall gate already removed (`recall_episodes_ranked` / `search_episodes_by_keywords`, PR #4241 lineage); fact recall was never gated. Fix: apply the analogous marker-safe word-boundary gate to the FACT path, reusing the existing `shares_word_prefix` helper and mirroring `search_episodes_by_keywords`'s clean/raw partition: * a CLEAN query token (all-alphanumeric) is kept only when it is a prefix of a whole word in the concept OR content (word-boundary), dropping interior/suffix noise while preserving inflectional recall; * a RAW token (any non-alphanumeric char — a hyphenated concept like "bug-pattern", or a "journal:"/"goal-edge:"/"sub:" marker) keeps the library's exact contiguous-substring semantics its callers store and re-filter on; a query with NO clean token bypasses the gate entirely, so the many concept/marker callers are provably unaffected. Both concept AND content are checked (the library matches both). Truncation to `limit` is deferred until AFTER the gate so a relevant fact ranked behind an interior-substring false positive is not dropped before the gate runs (mirroring `recall_episodes_ranked`). Wildcard/empty queries keep the return-all path. The gate only ever REMOVES interior/suffix false positives — it never adds a fact the backend did not return and never reorders. Verified empirically that the library treats a hyphenated token as a contiguous substring (matching the raw-token check exactly), so mixed clean+marker queries do not regress. Tests: * library_adapter::fact_query_gate_tests — pure helpers (partition_fact_query, fact_shares_query_relevance): clean/raw partition, interior-vs-word-boundary, inflection, concept-field match, marker substring, mixed query. * cognitive_memory::tests_fact_recall_word_boundary — end-to-end over the live in-memory backend: interior/suffix dropped, word-boundary + inflectional preserved, concept-field match, marker/concept substring preserved, deferred truncation honours limit, wildcard/empty bypass. qa-team: tests/gadugi/fact-recall-word-boundary-precision.yaml drives the end-to-end + unit tests plus an episodic-gate regression guard. Docs: docs/reference/cognitive-memory-fact-recall.md (new "Word-boundary relevance gate" section) and docs/architecture/cognitive-memory-library-adapter.md (fact-path gate subsection + in-adapter table row). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…recision + fact-yield) (#4309) `LibraryCognitiveMemory::search_facts` — the fact-recall seam the OODA turn/context path (`base_type_turn::prepare_turn_context`) and every natural-language fact caller reach — delegated matching to the upstream library, which matches each query token as a RAW case-insensitive SUBSTRING of a fact's concept OR content. A clean natural-language token therefore floated facts in on the INTERIOR/SUFFIX of an unrelated word. Empirically (live in-memory backend), for content "the reactor overheated": * search_facts("act") -> matched "re(act)or" and "artif(act)" [interior] * search_facts("own") -> matched "d(own)load" [interior] * search_facts("test") -> matched "la(test)" [suffix] while whole-word ("reactor"), prefix ("react"), and inflectional ("deploy" -> "deployed") matches were legitimate. Those off-topic facts crowd the CAPPED working-context recall the OODA cycle feeds to reasoning, dragging fact recall precision — and effective distillation fact-yield — down. This is the same defect the EPISODIC recall gate already removed (`recall_episodes_ranked` / `search_episodes_by_keywords`, PR #4241 lineage); fact recall was never gated. Fix: apply the analogous marker-safe word-boundary gate to the FACT path, reusing the existing `shares_word_prefix` helper and mirroring `search_episodes_by_keywords`'s clean/raw partition: * a CLEAN query token (all-alphanumeric) is kept only when it is a prefix of a whole word in the concept OR content (word-boundary), dropping interior/suffix noise while preserving inflectional recall; * a RAW token (any non-alphanumeric char — a hyphenated concept like "bug-pattern", or a "journal:"/"goal-edge:"/"sub:" marker) keeps the library's exact contiguous-substring semantics its callers store and re-filter on; a query with NO clean token bypasses the gate entirely, so the many concept/marker callers are provably unaffected. Both concept AND content are checked (the library matches both). Truncation to `limit` is deferred until AFTER the gate so a relevant fact ranked behind an interior-substring false positive is not dropped before the gate runs (mirroring `recall_episodes_ranked`). Wildcard/empty queries keep the return-all path. The gate only ever REMOVES interior/suffix false positives — it never adds a fact the backend did not return and never reorders. Verified empirically that the library treats a hyphenated token as a contiguous substring (matching the raw-token check exactly), so mixed clean+marker queries do not regress. Tests: * library_adapter::fact_query_gate_tests — pure helpers (partition_fact_query, fact_shares_query_relevance): clean/raw partition, interior-vs-word-boundary, inflection, concept-field match, marker substring, mixed query. * cognitive_memory::tests_fact_recall_word_boundary — end-to-end over the live in-memory backend: interior/suffix dropped, word-boundary + inflectional preserved, concept-field match, marker/concept substring preserved, deferred truncation honours limit, wildcard/empty bypass. qa-team: tests/gadugi/fact-recall-word-boundary-precision.yaml drives the end-to-end + unit tests plus an episodic-gate regression guard. Docs: docs/reference/cognitive-memory-fact-recall.md (new "Word-boundary relevance gate" section) and docs/architecture/cognitive-memory-library-adapter.md (fact-path gate subsection + in-adapter table row). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… (recall quality) The shared recall relevance gate `shares_word_prefix` (src/cognitive_memory/library_adapter.rs) — used by ranked episodic recall, keyword episode recall, and fact recall — anchors a query token to the start of a content word. That word-boundary prefix is directional: it recalls `test` -> "tests" but NOT the reverse, because `"test".starts_with("tests")` is false. So an objective phrased in the plural ("stabilize the flaky tests") silently missed an episode written in the singular ("wrote a test ..."), dropping a genuinely relevant prior from the reasoner's working context (`memory_consolidation::prepare_context`). Close that asymmetry with a conservative, length-guarded singular/plural fold (`needle_matches_word`): a plural query token matches the singular content word by whole-word EQUALITY of a generated variant — regular `-s`/`-es` (`tests` -> "test", `caches` -> "cache") and the shape-changing `-y` <-> `-ies` pair in both directions (`categories` <-> "category"). Folding matches only on equality (never a prefix) and only when the stripped stem clears a two-character minimum, so — unlike a prefix on a stripped stem — it cannot re-introduce the interior over-matching the word-boundary rule removed (`buses` does not surface "business"; `is` does not fold to "i"). This mirrors the folding `knowledge_context::token_matches_pack` already applies to pack selection (PR #4241 lineage), now shared uniformly across episodic/keyword/fact recall. Scope discipline: the raw/marker recall paths (reflection-occurrence prefiltering and `count_recurring_failures`) do NOT call `shares_word_prefix` and are untouched; folding cannot broaden their exact-substring contracts. Tests: 4 new unit tests in `word_boundary_gate_tests` (plural->singular, `-y`/`-ies` both directions, no interior re-introduction, short-token guard) and 1 integration test in `tests_ranked_episodic` (plural query recalls the singular-form episode). Existing gate/inflectional/backfill invariants unchanged. Docs: new "Singular/plural folding closes the prefix asymmetry" subsection. qa-scenario: tests/qa-scenarios/recall-gate-plural-folding-relevance.yaml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… (recall quality) (#4313) The shared recall relevance gate `shares_word_prefix` (src/cognitive_memory/library_adapter.rs) — used by ranked episodic recall, keyword episode recall, and fact recall — anchors a query token to the start of a content word. That word-boundary prefix is directional: it recalls `test` -> "tests" but NOT the reverse, because `"test".starts_with("tests")` is false. So an objective phrased in the plural ("stabilize the flaky tests") silently missed an episode written in the singular ("wrote a test ..."), dropping a genuinely relevant prior from the reasoner's working context (`memory_consolidation::prepare_context`). Close that asymmetry with a conservative, length-guarded singular/plural fold (`needle_matches_word`): a plural query token matches the singular content word by whole-word EQUALITY of a generated variant — regular `-s`/`-es` (`tests` -> "test", `caches` -> "cache") and the shape-changing `-y` <-> `-ies` pair in both directions (`categories` <-> "category"). Folding matches only on equality (never a prefix) and only when the stripped stem clears a two-character minimum, so — unlike a prefix on a stripped stem — it cannot re-introduce the interior over-matching the word-boundary rule removed (`buses` does not surface "business"; `is` does not fold to "i"). This mirrors the folding `knowledge_context::token_matches_pack` already applies to pack selection (PR #4241 lineage), now shared uniformly across episodic/keyword/fact recall. Scope discipline: the raw/marker recall paths (reflection-occurrence prefiltering and `count_recurring_failures`) do NOT call `shares_word_prefix` and are untouched; folding cannot broaden their exact-substring contracts. Tests: 4 new unit tests in `word_boundary_gate_tests` (plural->singular, `-y`/`-ies` both directions, no interior re-introduction, short-token guard) and 1 integration test in `tests_ranked_episodic` (plural query recalls the singular-form episode). Existing gate/inflectional/backfill invariants unchanged. Docs: new "Singular/plural folding closes the prefix asymmetry" subsection. qa-scenario: tests/qa-scenarios/recall-gate-plural-folding-relevance.yaml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

rysweet merged commit e0151dd into main Jul 17, 2026
16 checks passed

rysweet deleted the engineer/continuously-research-and-improve-your-own-cogn-70ab8541-1784247377-0d08a7 branch July 17, 2026 01:52

rysweet mentioned this pull request Jul 17, 2026

fix(cognition): whole-word merged in episode classifier (recall quality / distillation fact-yield) #4245

Merged

rysweet mentioned this pull request Jul 17, 2026

feat(cognition): singular/plural-fold knowledge-pack relevance scoring (recall quality) #4250

Merged

rysweet mentioned this pull request Jul 17, 2026

fix(cognition/recall): word-boundary gate for ranked episodic recall #4257

Merged

rysweet mentioned this pull request Jul 18, 2026

fix(cognition): word-boundary relevance gate on fact recall (recall precision + fact-yield) #4309

Merged

rysweet mentioned this pull request Jul 18, 2026

Recall quality: singular/plural folding for the recall word-boundary gate #4313

Merged

Navigation Menu

fix(cognition): whole-word, distinct-token knowledge-pack relevance scoring (recall quality) - #4241

fix(cognition): whole-word, distinct-token knowledge-pack relevance scoring (recall quality)#4241
rysweet merged 1 commit into
mainfrom
engineer/continuously-research-and-improve-your-own-cogn-70ab8541-1784247377-0d08a7

rysweet commented Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

rysweet commented Jul 17, 2026

Summary

Fix

Evidence

Durability

Uh oh!

github-actions Bot commented Jul 17, 2026

📊 Coverage Summary

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant