A working advisor/reviewer skill: field report and lessons for a native advisor

[pcarielo]

A working advisor/reviewer skill: field report and lessons for a native advisor

Category: Feature requests | prime-agent 0.8.1 | Linux

Summary

We wanted the /advisor behavior Claude Code has: a stronger reviewer model that
sees the whole session and tells you the approach is wrong before it hardens.
0.8.1 already ships the primitives (recursive sub-agents, kernel Python
skills, agent messaging), so we built it as a skill instead of waiting for a
native feature.

Then we made the reviewer review its own skill, adversarially, over five
rounds. Every round found a real bug until the last one, which came back
sound. The result ships with a 41-check regression suite. Both the private
deployment and this exact model-agnostic variant (loaded via
--skill <path>) were smoked end to end in fresh root sessions: native load,
spawn, reply retrieval through latest(handle), cleanup.

Why post this: there is a related closed architecture proposal (#1448, an
outside contributor's unmerged PR with zero maintainer reviews; read-only
reviewer subruntime, severity routing, terminal-triggered reviews). We have no
insight into maintainer intent for it. What we can report is that the concept
works well in practice from the skill layer, and where the skill layer had to
fight the runtime to get there.

What the skill does

handle = await advisor("is this plan sound before I edit src/foo.cpp?")
# ... later turn: the child replies via agent_message with
# VERDICT / KEY RISKS / CORRECTIONS / VERIFY
advice = await advisor.latest(handle)
await rlm.delete_subagent(handle.name)

It builds a compact transcript from the current session log (roles labelled
[USER] / [AGENT] / [TOOL x] / [RESULT x] / [AGENT MESSAGE x], opening
task kept, recent tail within a character budget), spawns a reviewer child with
a reviewer persona plus the transcript plus the focus question, and the reply
arrives as an ordinary agent message. latest(handle) returns the reply of
that exact child, matched by session UUID.

What five adversarial rounds taught us

1. The consult policy is the product. The consult matters at the moment an
   approach is about to harden: before substantive work, before declaring done
   (deliverable made durable first), when stuck, when changing approach. Short
   reactive tasks do not need it.
2. Reply identity must be exact. Child names are reusable and replies are
   agent_message records in the PARENT session log, so matching by name returns
   a deleted predecessor's advice after a name reuse. We match the child's
   session UUID (from rlm-subagent.json, falling back to the newest *.jsonl
   in that child's session dir) against the From: <name>, active <id>, session <uuid> envelope. Unresolvable identity returns None. Never a name fallback.
3. Redact before clipping. A transcript budget truncates text. A credential
   cut in half by a clip boundary slips past length-threshold redaction
   patterns. Redact every block before clipping, then redact the assembled
   transcript again.
4. Prior agent messages belong in the transcript. Reconcile consults ("I
   found X, you suggested Y, which constraint breaks the tie?") need to see the
   previous critique without manual summarization.
5. Ban recursion. A reviewer that inherits the consult policy will consult
   another reviewer, recursively, until the depth limit. The persona forbids
   it.
6. A structured reply contract pays off. VERDICT / KEY RISKS / CORRECTIONS
   / VERIFY keeps critiques actionable and machine-checkable. It maps cleanly
   onto the severity routing sketched in docs(coding-agent): advisor, prompt-settlement, and ask-user architecture docs #1448, and terminal-triggered review
   is exactly the before-declaring-done consult.

Where the skill layer had to fight the runtime

1. No official transcript access for skills. The session JSONL format is
   documented (session-format.md), but a skill must locate its own session log
   (we derive it from RLM_SESSION_DIR's basename and fail closed when that
   exact path cannot be resolved; in one fresh root session that basename did
   not even match the session log filename, and the skill refused until given
   transcript_path=, which is the correct failure mode) and parse internals
   like the custom_message / agent_message envelope. A
   rlm.session_log() host request, or a rendered transcript, would remove the
   whole class of format-drift risk.
2. No API to retrieve a child's reply. rlm.run() is admission-only by
   design and replies arrive on later turns. For consult flows we parsed the
   parent log's agent_message records. handle.latest_reply() or
   agent_message.latest(handle) would make the consult pattern first-class.
3. Child identity is eventually consistent. rlm-subagent.json appears
   shortly after admission. A latest(handle) called in the same breath as the
   spawn cannot resolve identity yet. Registry-before-return, or the session
   UUID on the spawn handle, would close the race.
4. Transcripts leaving the host are a privacy surface. Whatever ships the
   native advisor will forward conversation content to another model. Pre-clip
   redaction and a documented stance belong in the design from day one.

One further runtime observation (forkserver template caching skill preimports
for children of older sessions) is now a bug report with a reproduction:
#1925
A second observation about scheduled refinement timing could not be
reproduced reliably and is not reported.

What we can offer

Happy to contribute when invited, per CONTRIBUTING. No unsolicited PR. The
skill works as a built-in candidate or as a documented recipe; it is
model-agnostic (free model selection by default, ADVISOR_MODEL and
ADVISOR_ALLOWED_MODELS for pinning and fleet policy), the regression suite
comes with it, and the transcript renderer plus reply matcher is a ready-made
reference implementation for the client side of a native feature.