How we debugged "the role wiring works but doesn't work" (#133)

[antonyrag]

How we debugged "the role wiring works but doesn't work" (#133)

Wiring AI Employee role context into /chat looked simple on paper: pull a role's personality + learned context, fold it into the system prompt, done. It took three separate, unrelated bugs before it actually worked — writing this up because each one is a real trap, not specific to this repo.

The setup

core/employees/ (from #130) already had working role personas, owner instructions, and pgvector-backed learned memory, with its own routes (/employees/{role}/context) proving the data was real. The remaining piece was folding that into /chat's generation prompt. Should have been a one-line change.

Bug 1: the model was reading the instruction and ignoring it anyway

First attempt: prepend the role's personality and context (including an owner instruction — "always mention our 24-hour support line when asked about support") to the system prompt, ahead of the existing grounding instructions.

Compiled fine. Ran fine. The model's answer never mentioned the support line — not even when asked about it directly ("do you have a 24 hour support line?"). Printing the exact prompt sent to the model confirmed the instruction really was there, worded clearly, marked "MANDATORY."

Turns out: in a long system prompt, a model tends to weight instructions near the end — right next to the actual question — more heavily than ones stated once near the top, even with strong framing. This is recency bias, and it's a known thing in prompt engineering, but it's easy to assume "the instruction is in the prompt somewhere" is sufficient. It isn't. Position matters as much as content.

Fix: in addition to the system prompt, append a short reminder directly onto the query text itself — right next to the question, at the very end of what the model sees before "Answer:". Not a replacement for the system prompt version, an addition.

Bug 2: silent truncation that looked like a real (wrong) answer

Testing bug 1's fix, answers started coming back as complete-looking-but-wrong sentences: "I do not have that information in the provided context." Reasonable-sounding, but still missing the instruction. Assumed this was still a prompt-wording problem. It wasn't.

We'd recently switched GEMINI_CHAT_MODEL from gemini-2.5-flash to gemini-3.5-flash (the old model got blocked for newer API keys ahead of its official deprecation date — a real, current Gemini API change with no clean migration warning). gemini-3.5-flash reasons by default. Its internal "thinking" tokens count against max_output_tokens — so a max_tokens=200 test came back with thoughts_token_count=190, leaving 10 tokens for the actual visible answer. finish_reason: MAX_TOKENS, not a real stopping point.

The visible symptom (a short, plausible, complete-sounding sentence) looked exactly like a grounding decision, not a truncation. Only checking finish_reason and the raw usage metadata directly against the API — bypassing our own code — surfaced it.

Fix: raised MAX_OUTPUT_TOKENS from 1024 to 4096. Not a prompt problem at all.

Bug 3: the fix was correct and never ran

With both of the above genuinely fixed, direct Python calls to ask(role="support", ...) worked perfectly. The real /chat HTTP endpoint, hit with ?role=support via curl, still didn't. Same broken answer as if role were never passed.

FastAPI doesn't error on an unrecognized query parameter — it just silently drops it. role had never been added to the /chat route's function signature. Every role= sent over the wire for the entire debugging session was discarded before it reached the code that, it turns out, had already been correct since bug 2's fix.

Caught by comparing prompt_tokens between the internal call (which worked, and used more tokens because of the injected context) and the HTTP call (fewer tokens, identical to the roleless baseline) — the token count was the tell, not the wording of the answer.

What actually mattered

Two lessons, more general than this specific feature:

• A clean compile and a 200 response prove nothing about correctness. All three bugs passed py_compile. All three returned valid JSON. Only checking the actual content of the response — and in bug 2's case, metadata most people never look at (finish_reason, thoughts_token_count) — surfaced the real problem.
• When something "should" work and doesn't, isolate rather than reword. Each fix attempt that didn't work bought real diagnostic information (bug 1's failed fix ruled out "instruction not present"; the identical-answer test ruled out phrasing). Iterating on wording without isolating the actual failure point would have taken much longer.

Full technical detail is in the commit message for d53e333 if you want the exact prompts and outputs at each stage.