A "coach" for the agent: should per-turn, judge-driven guidance be first class in eve?

[TechFeatured]

We run an eve agent in production and have been moving behaviour rules out of
the system prompt and into small per-turn checks. A cheap judge model reads the
incoming message (or the turn that just finished), and when a rule applies, the
agent gets one short line of guidance for that turn only. When no rule applies,
the agent pays nothing. We've started calling it a coach: it doesn't rewrite
answers or block them, it just says "do X this time".

We think this pattern is general enough that it might belong in the framework,
and we'd like to hear whether others are doing something similar.

Why we went this way

Every rule we added to the system prompt to fix a one-off behaviour seemed to
regress something else, and the prompt kept growing. Most of those rules only
matter in some turns. Paying for them, and risking collisions with them, on
every turn felt wrong.

What we measured

1. Does a one-line, per-turn nudge reach the model at all? We tested with a
nonsense rule ("always include the word banana") so there was no baseline
behaviour to confound it, across multi-turn conversations:

Placement	Turns that complied
No nudge (control)	0/8
Appended as user-role context for the turn	8/8
System-scope instruction for the turn	8/8
Client-supplied ephemeral context on send	8/8

2. Does it change a real behaviour? On the first turn of a new
conversation, our agent often reached for generic web search when the person
wanted our own domain search. With a one-line nudge, sent only when a judge
says the message is that kind of ask:

	Picked the domain tool on turn 1
Run 1, no nudge	0/10
Run 1, nudged	10/10
Run 2 (fuller index), no nudge	1/10
Run 2, nudged	10/10

3. What does it cost in prompt caching? This decided the placement for us.
Putting per-turn content into system scope dropped prefix reuse on the first
model call of that turn from roughly 94–97% to 0% (it recovered on later calls
within the turn). Appending the nudge as ephemeral context at the end of the
conversation kept reuse intact. A framework-level primitive could guarantee the
cache-safe placement, so nobody has to discover this the hard way.

4. Can a judge decide when to coach, cheaply and safely? One batched
classification call per message, made before the turn starts, with a high
threshold because a wrong nudge costs more than a missed one (a miss just falls
back to default behaviour):

Set	Labels correct	False nudges	Misses	Latency p50 / p90
Tuning (60 cases)	57–58/60	0	4–5	~270 ms / ~470 ms
Fresh holdout, run once (22 cases)	18/22	1	4	~277 ms / ~455 ms

5. Instructions alone don't always hold. One prompt rule ("offer choices
rather than ending on a question in prose") was broken in 10 of 10 sampled
answers. That's the kind of rule we're now moving to a deterministic
after-the-turn check, with a nudge on the next turn only when it was broken.

Not everything works: one nudge we tested was clearly received and then
ignored. Each nudge has to earn its place with its own measurement.

What first-class support could look like

• Pre-turn coaching. A hook that sees the incoming message (and optionally
  the recent transcript) and returns guidance that applies to this turn only:
  ephemeral, never written into history, placed where it can't break the
  prompt cache. One friction point we hit: instruction resolvers at turn start
  can't see the incoming message, so today pre-turn coaching has to be done
  outside the agent or by the client.
• Post-turn coaching. A check after the turn that can leave a pending note,
  delivered once on the next turn and then cleared.
• Measurement built in. For every coaching rule: how often it fired, and
  whether the next answer complied. Without that, nudges turn into hope.
• Budgets and sampling. Cheap deterministic checks every turn, judge calls
  sampled for low-risk rules, and per-rule thresholds that can be tuned without
  a deploy.

Questions for the eve team and others

1. Is this something you'd consider as a framework primitive, or is
   instructions plus client context the intended way?
2. Should pre-turn hooks be able to see the incoming message?
3. Would you guarantee a cache-safe placement for per-turn context?
4. Is anyone else running judge-in-the-loop guidance like this, and what did
   you measure?