feat(v1): per-token advantages on the node

[mikasenghaas]

Summary

Credit assigned by an RL algorithm is token-aligned, so it belongs where the tokens are.

• MessageNode.advantages: list[float] | None — per-sampled-token, the same compact layout as logprobs (length equals the True entries in mask).
• Branch.advantages: list[float] | None — the node values spread onto token_ids, 0.0 on non-sampled positions, exactly like Branch.logprobs.
• The spreading both properties do is factored into one spread helper rather than written twice.

Nothing in verifiers writes the field — like Trace.run, it is consumer-stamped.

Why the node

Branches share nodes. A consumer holding a flat per-branch advantage stream can give the same generated token different credit in two branches of one trace — compaction and subagent forks both produce that shape. On the node it is unrepresentable: one node, one value, every branch through it agrees.

It also means a consumer stops re-implementing the path walk. branch.advantages falls out of the same traversal as token_ids, sampled_mask and logprobs, so the alignment is structural rather than asserted by the caller.

Why nullable

Unassigned credit and zero credit are different facts, and a trainer acts on them differently: a group whose rewards were all equal is assigned zeros and carries no gradient, while a rollout that was never scored ships no advantage stream at all. Storing zeros for both would erase that, so the field is None until assigned — at the node, and at the branch when no node on the path holds any:

unassigned :  None
zero credit:  [0.0, 0.0, 0.0, 0.0, 0.0]
real credit:  [0.0, 0.0, 0.0, 0.5, 0.5]

A partially assigned path spreads, with the unassigned nodes reading 0.0.

Verification

• uv run pytest tests/v1 -m "not e2e" green; ruff check / ruff format --check; ty clean on the touched files.

🤖 Generated with Claude Code

---

Note

Low Risk
Additive schema and view-layer refactor with no writers in verifiers; behavior change is limited to new optional fields and deduplicated spread logic for logprobs.

Overview
Adds per-token RL credit on the message graph so trainers can align advantages with the same token layout as logprobs.

MessageNode.advantages is optional per-sampled-token credit (compact layout matching mask / logprobs). None means never scored; zeros mean explicit zero credit (e.g. tied rewards)—distinct semantics for downstream RL.

Branch.advantages spreads node values onto full token_ids like Branch.logprobs, and stays None when no node on the path was assigned credit so it is not confused with an all-zero branch.

Branch.spread() factors out the shared widening logic; logprobs now delegates to it. Verifiers does not populate advantages—consumers stamp them after scoring.

^{Reviewed by Cursor Bugbot for commit 7eba3e6. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add per-token advantages field to MessageNode and Branch

• Adds an optional advantages: list[float] | None field to MessageNode to store per-sampled-token credit assignments.
• Adds a Branch.advantages property in trace.py that aligns node-level advantages to the full token sequence, returning None if no nodes have advantages set and 0.0 for unassigned positions otherwise.
• Refactors Branch.logprobs to delegate to a new Branch.spread helper, which generalizes widening any per-sampled-token node field to full-token alignment.

^{Macroscope summarized 7eba3e6.}

[macroscopeapp]

Approvability

Verdict: Approved 7eba3e6

Additive change: new optional advantages field on MessageNode and corresponding Branch.advantages property, following the established logprobs pattern. The spread logic refactor is mechanical and preserves existing behavior. Author has extensive history with these files.

^{You can customize Macroscope's approvability policy. Learn more.}