Skip to content

fix(loki): find the contact that IS there — the refusal was a false negative - #222

Merged
github-actions[bot] merged 1 commit into
mainfrom
fix/loki-people-lookup-and-429
Aug 13, 2026
Merged

fix(loki): find the contact that IS there — the refusal was a false negative#222
github-actions[bot] merged 1 commit into
mainfrom
fix/loki-people-lookup-and-429

Conversation

@catomean

Copy link
Copy Markdown
Collaborator

Browser-tested the shipped harness (#212, #214) against the exact question that started all of this:

"Who is Ilya Druzhnikov in my contacts, and what is his affiliation? Also do research on Elena Weber — linkedin etc."

The good news: no fabrication. No UZH. No invented biography. Both halves refused cleanly:

Regarding Ilya Druzhnikov: Not in your data.
Regarding Elena Weber: I cannot browse the web or conduct
external research. Not in your data.

The bad news: that refusal was wrong. Ilya Druzhnikov is one of 1285 rows in the people table.

Hardening against fabrication moved the failure to the opposite end — and this end fails quietly, because a confident refusal looks like diligence. A grounded assistant that cannot find real records is not safer than one that invents them; it is differently useless.

Two bugs, both found only by driving the real UI

1. peopleFacts passed the whole message as a name filter. searchPeople filters with name ILIKE '%q%', so a sentence can never match a name column. Every lookup silently fell through to "12 most recent contacts", and the named person was simply absent from the context — after which "Not in your data" was the correct answer to a context that was wrong.

Now candidate names are extracted and searched individually. That extraction had its own bug, caught by a test before shipping: matching only capitalised tokens makes the lowercase words between two names invisible, so "Ilya Grün the same person as Jean-Luc" merged into one nonsense run. Fixed by tokenising every word and breaking the run on anything that isn't a capitalised non-stopword.

2. A groq 429 abandoned the tool loop. The box log:

[loki] tool loop failed: groq 429: Rate limit reached for
model `llama-3.3-70b-versatile` ... service tier `on_demand`

It fell back to the gateway — a path with weaker retrieval. A rate limit is a capacity problem, not a capability one, so a capacity blip was silently degrading answer quality. It now steps down to llama-3.1-8b-instant, already verified to drive the loop via the text protocol (npm run probe:models).

Verification

scripts/test/agent-name-lookup.ts pins both, including the exact production message, plus first-name-only lookups, stopword rejection, accented (Ilya Grün) and hyphenated (Jean-Luc) names, and the 429 step-down. 67/67 unit files pass, tsc clean, 0 lint errors.

🤖 Generated with Claude Code

…egative

Browser-tested the shipped harness against the exact question that
started this: "Who is Ilya Druzhnikov in my contacts, and what is his
affiliation? Also do research on Elena Weber."

No fabrication — both halves refused correctly, no UZH, no invented
biography. But the refusal was WRONG: Ilya Druzhnikov is one of 1285
rows in the people table.

Hardening against fabrication had moved the failure to the opposite end,
and this end fails QUIETLY — a confident refusal looks like diligence. A
grounded assistant that cannot find real records is not safer than one
that invents them; it is differently useless.

Two bugs, both found only by driving the real UI:

1. peopleFacts passed the WHOLE user message to searchPeople, which
   filters with `name ILIKE '%q%'`. A sentence can never match a name
   column, so every lookup fell through to "12 most recent contacts" and
   the named person was simply absent from the context. Now candidate
   names are extracted from the message and searched individually.

   The extraction had its own bug, caught by a test: matching only
   CAPITALISED tokens makes the lowercase words between two names
   invisible, so "Ilya Grün the same person as Jean-Luc" merged into one
   nonsense run. Tokenise every word; break the run on anything that is
   not a capitalised non-stopword.

2. The tool loop hit `groq 429` and fell back to the gateway — a path
   with WEAKER retrieval. A rate limit is a CAPACITY problem, not a
   capability one, so a capacity blip was silently degrading answers.
   It now steps down to llama-3.1-8b-instant, already verified to drive
   the loop via the text protocol (scripts/probe-models.ts).

Both are pinned by scripts/test/agent-name-lookup.ts, including the
exact production message. 67/67 unit files pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions
github-actions Bot merged commit fd7ddd3 into main Aug 13, 2026
1 check passed
@github-actions
github-actions Bot deleted the fix/loki-people-lookup-and-429 branch August 13, 2026 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant