th-f19853: agentic conversation bench (LLM driver + judge)

[brentrager]

Problem

The agentic bench scores ONE turn against a workspace. That misses the failure that actually bit us: the ical mess, where Big Smooth gave three mutually contradictory calendar answers in one conversation. No single turn was obviously wrong — the CONVERSATION was.

Solution

smooth-bench convo holds a real multi-turn conversation against a live Big Smooth:

1. One canonical-protocol session, same sessionId for every turn — the agent's own memory of the thread is what's under test.
2. An LLM driver plays a realistic user turn after turn from a persona, stopping early when the goal is met (or hopeless).
3. An LLM judge grades the whole thread 1–5 on helpfulness, correctness, tool use and consistency across turns, plus a rubric PASS/FAIL and a per-turn note.
4. Per-trial transcripts land as JSON-lines under ~/.smooth/bench-runs/convo-*/.

Scenarios

id	what
calendar-list	the ical ask — claims must be grounded in a tool call, and the thread must not contradict itself
rapid-correction	fires a correction 1.5s into the first turn (th-3a912a interrupt gap) — expect_fail
workspace-orientation	ordinary ask; must inspect the workspace rather than guess
scoped-file-task	write a file, then change the scope — final state must match the final claim

expect_fail records XFAIL while the gap is open (suite green) and XPASS — non-zero exit — the day interrupts land, which is the signal to drop the flag and keep it as a plain regression test.

Reuse, not reinvention

CanonicalSession is a new multi-turn form of the existing canonical driver: send and collect are split so a message can be fired before the previous turn finishes (that concurrency is the point of the barge-in scenario). run_via_canonical is now a thin wrapper over it, with its lenient hangup behaviour preserved (new CanonicalOutput::completed tells a finished turn from a dropped one). Gateway plumbing, parse_verdict, truncate and the daemon-spawning ProcessBooter are all the existing ones.

Not in CI

Every scenario is several live LLM turns — slow, networked, costs money. It's a smooth-bench subcommand, never part of cargo test; only the pure parsing/rendering logic is unit-tested (16 new tests, including "a partial grade is INCONCLUSIVE, never a silent pass").

cargo run -p smooai-smooth-bench -- convo                    # whole suite, spawns its own daemon
cargo run -p smooai-smooth-bench -- convo --only calendar-list
cargo run -p smooai-smooth-bench -- convo --url http://127.0.0.1:8788 --token "$SMOOTH_LOCAL_TOKEN"

Credentials: SMOOAI_GATEWAY_KEY, falling back to ~/.smooth/providers.json (the store the daemon already reads), so it runs with no env setup.

Verification

cargo fmt --check, cargo clippy --all-targets (clean), cargo test -p smooai-smooth-bench (186 pass). Trial-run findings against a locally spawned Big Smooth are in the pearl/PR comments.

Pearl th-f19853.

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: dd875db

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@smooai/smooth	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[brentrager]

Trial run against a locally spawned Big Smooth (deepseek-v4-flash, driver + judge same model)

SCENARIO               TRIAL  STATUS   TURNS  SCORES(H/C/T/Cons)
calendar-list          1      PASS     1      5 5 5 5
rapid-correction       1      XFAIL    3      2 3 4 1
workspace-orientation  1      FAIL     3      4 3 5 2
scoped-file-task       1      PASS     2      5 5 5 5

(b) reproduces — and it's worse than "contradictory"

The barge-in didn't just produce a stale answer, it swapped the replies:

turn	user asked	assistant answered
1	"What files are in my home directory?"	"Today is December 19, 2024."
2	"actually ignore that — just tell me today's date"	(the full home-directory listing)
3	"I already changed the request"	"You're right, my bad. Today is …"

The correction arriving mid-turn doesn't cancel turn 1; both turns run and their responses land against the wrong prompts. Judge: consistency 1/5, "answered the superseded file-listing request after the user changed to the date question." That's th-3a912a with a receipt — recorded XFAIL, and it flips to XPASS (non-zero exit) the day interrupts land.

Bonus real finding: workspace-orientation FAILED on its own merits

Scaffolded a Node project, told the user "Run npm start", then two turns later: "There's no npm start script defined yet — I misspoke." Judge: consistency 2/5. Exactly the class of self-contradiction this suite exists to catch, and nothing to do with interrupts.

Also observed: the agent hallucinates today's date (no date/clock tool), and bash reads outside SMOOTH_WORKSPACE (expected — the sandbox confines writes, not reads).