fix(loki): find the contact that IS there — the refusal was a false negative - #222
Merged
Merged
Conversation
…egative Browser-tested the shipped harness against the exact question that started this: "Who is Ilya Druzhnikov in my contacts, and what is his affiliation? Also do research on Elena Weber." No fabrication — both halves refused correctly, no UZH, no invented biography. But the refusal was WRONG: Ilya Druzhnikov is one of 1285 rows in the people table. Hardening against fabrication had moved the failure to the opposite end, and this end fails QUIETLY — a confident refusal looks like diligence. A grounded assistant that cannot find real records is not safer than one that invents them; it is differently useless. Two bugs, both found only by driving the real UI: 1. peopleFacts passed the WHOLE user message to searchPeople, which filters with `name ILIKE '%q%'`. A sentence can never match a name column, so every lookup fell through to "12 most recent contacts" and the named person was simply absent from the context. Now candidate names are extracted from the message and searched individually. The extraction had its own bug, caught by a test: matching only CAPITALISED tokens makes the lowercase words between two names invisible, so "Ilya Grün the same person as Jean-Luc" merged into one nonsense run. Tokenise every word; break the run on anything that is not a capitalised non-stopword. 2. The tool loop hit `groq 429` and fell back to the gateway — a path with WEAKER retrieval. A rate limit is a CAPACITY problem, not a capability one, so a capacity blip was silently degrading answers. It now steps down to llama-3.1-8b-instant, already verified to drive the loop via the text protocol (scripts/probe-models.ts). Both are pinned by scripts/test/agent-name-lookup.ts, including the exact production message. 67/67 unit files pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Browser-tested the shipped harness (#212, #214) against the exact question that started all of this:
The good news: no fabrication. No
UZH. No invented biography. Both halves refused cleanly:The bad news: that refusal was wrong.
Ilya Druzhnikovis one of 1285 rows in the people table.Hardening against fabrication moved the failure to the opposite end — and this end fails quietly, because a confident refusal looks like diligence. A grounded assistant that cannot find real records is not safer than one that invents them; it is differently useless.
Two bugs, both found only by driving the real UI
1.
peopleFactspassed the whole message as a name filter.searchPeoplefilters withname ILIKE '%q%', so a sentence can never match a name column. Every lookup silently fell through to "12 most recent contacts", and the named person was simply absent from the context — after which "Not in your data" was the correct answer to a context that was wrong.Now candidate names are extracted and searched individually. That extraction had its own bug, caught by a test before shipping: matching only capitalised tokens makes the lowercase words between two names invisible, so
"Ilya Grün the same person as Jean-Luc"merged into one nonsense run. Fixed by tokenising every word and breaking the run on anything that isn't a capitalised non-stopword.2. A
groq 429abandoned the tool loop. The box log:It fell back to the gateway — a path with weaker retrieval. A rate limit is a capacity problem, not a capability one, so a capacity blip was silently degrading answer quality. It now steps down to
llama-3.1-8b-instant, already verified to drive the loop via the text protocol (npm run probe:models).Verification
scripts/test/agent-name-lookup.tspins both, including the exact production message, plus first-name-only lookups, stopword rejection, accented (Ilya Grün) and hyphenated (Jean-Luc) names, and the 429 step-down. 67/67 unit files pass, tsc clean, 0 lint errors.🤖 Generated with Claude Code