feat(base): per-message / per-role analytics on RenderedTokens

[hallerite]

Summary

• New RenderedTokens methods — tokens_per_message, tokens_by_role, message_token_spans, role_token_spans — derived from message_indices / sampled_mask and a new message_roles field every renderer now populates. Use cases: per-role length penalties for RL trainers (e.g. prime-rl), per-turn statistics over logprobs / attention / gradients via (start, end) spans.
• Fixes attribution loss in every bridge_to_next_turn: bridges previously discarded msg_idx / is_sampled even though local emit helpers received them. Now bridges populate message_indices (relative to new_messages), sampled_mask (uniformly False — bridge output is a prompt), and message_roles. Consumers can run the new analytics on bridge output for incremental per-message accounting without re-rendering.
• 17 renderers updated. 35 new parametrized tests; 1310 total passing.

Example

rendered = renderer.render(messages)
rendered.tokens_by_role(sampled_only=True)["assistant"]   # length-penalty signal
rendered.role_token_spans()["tool"]                       # [(start, end), ...]

Test plan

• pytest tests/ — 1310 passed, 52 skipped (gpt-oss HF parity), 1 xfailed (pre-existing)
• New tests parametrized across all 17 renderer fixtures in tests/test_tokens_per_message.py
• End-to-end smoke with multi-turn tool-use trajectory (single render vs bridge-incremental) — counts match

🤖 Generated with Claude Code

[snimu]

this isn't truncated to a max of len(self.message_roles) though, but it's returned directly below. So len(out) > len(self.message_roles) is possible. Is that intended?

[hallerite]

addressed in 954ae83