Skip to content

chore: timezone-stable uv.lock and trace record cleanups - #2172

Merged
mikasenghaas merged 17 commits into
mainfrom
manual-cleanups
Jul 30, 2026
Merged

chore: timezone-stable uv.lock and trace record cleanups#2172
mikasenghaas merged 17 commits into
mainfrom
manual-cleanups

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Jul 29, 2026

Copy link
Copy Markdown
Member

Summary

Companion: PrimeIntellect-ai/prime-rl#3157 (merge together).

  • Pin [tool.uv.exclude-newer-package] cutoffs as full RFC 3339 UTC timestamps: bare dates resolve to midnight local time, so teammates in different timezones kept relocking uv.lock and failing the --locked pre-commit hooks.
  • Treat an expired rollout deadline as an agent failure (HarnessError, ok=False, no scoring) instead of a clean harness_timeout stop that scored the partial trajectory; rename the concept harness timeout → agent timeout throughout.
  • Tighten the Trace schema (TRACE_VERSION reset to 1 — a clean break, the old counter is retired): agent, verifiers, and RunInfo.id are required (verifiers auto-stamps the current build; the v0 bridge and validate/debug CLIs synthesize their seats), tools defaults to []. run and stop_condition keep their meaningful None.
  • Slim the Trace surface: one verb for mutators (record_*), one reader per fact — drop the agent_name/trainable/runtime passthroughs and the duplicate error property.
  • Align tool_messages with assistant_messages (nodes-based, branch-independent).
  • Drop the override warnings in record_metric/record_reward: overriding is defined behavior, last write wins.
  • Trim trace docstrings to the constraints they add; hoist TRACE_VERSION; rename _NODE_DUMP_EXCLUDEEXCLUDE_FIELDS.

Breaking

  • TRACE_VERSION resets to 1 on the tightened schema. No backwards compatibility: pre-existing trace records do not load, and old eval runs cannot be resumed or replayed.
  • Timed-out rollouts no longer score: stop_condition="harness_timeout" + ok=True → recorded HarnessError + ok=False + stop_condition="error"; harness_timeout is gone from the stop-condition vocabulary and Trace.is_truncated (the v0 bridge maps timeout_reached through the generic truncation fallback).
  • TaskTimeout.harnessTaskTimeout.agent; cap_remote_harness_timeoutcap_remote_agent_timeout.
  • Trace.agent and Trace.verifiers required: constructors must pass agent=AgentInfo(config=AgentConfig()); verifiers fills itself.
  • Trace.tools: list[Tool] | Nonelist[Tool] (default []).
  • EvalRunInfo.id / TrainRunInfo.id: str | None → required str.
  • Trace.stamp(run, **info)Trace.record_run(run, **info); Trace.capture_error(e)Trace.record_error(e).
  • Trace.errorTrace.last_error; Trace.agent_name / Trace.trainable / Trace.runtime removed — read trace.agent.name / .trainable / .runtime.
  • Trace.stop() requires an explicit condition (the unused "done" default is gone).
  • Trace.tool_messages: last branch only → every tool result across the node graph (identical for linear rollouts).
  • _NODE_DUMP_EXCLUDEEXCLUDE_FIELDS.

Verification

  • uv lock --check green under TZ=Europe/Berlin / America/Los_Angeles / Asia/Tokyo (previously failed cross-TZ with exclude newer timestamp changed).
  • uv run pytest tests/v1 green; ruff + ty hooks green.
  • Live 1-rollout smoke of every entrypoint on the new schema: eval (rich dashboard + hub push), validate, debug, replay (strict re-read of a v5 record), serve (boot), legacy bridge via live test_legacy e2e.

🤖 Generated with Claude Code


Note

High Risk
Breaking trace schema and API (TRACE_VERSION reset, no old record compatibility) plus changed timeout semantics affect eval replay, trainers, and any code reading removed Trace properties.

Overview
Resets TRACE_VERSION to 1 with a tighter trace record: agent and auto-stamped verifiers are required, tools defaults to [], and top-level shims (agent_name, trainable, runtime, error) are removed in favor of trace.agent.* and trace.last_error. Mutators are unified as record_run / record_error (replacing stamp / capture_error).

Rollout wall-clock limits are renamed harness → agent timeout (TaskTimeout.agent, cap_remote_agent_timeout). When that budget expires, the rollout now fail()s with HarnessError (ok=False, stop_condition="error") instead of a clean harness_timeout stop that could still score a partial trajectory; harness_timeout is dropped from truncation semantics and the v0 bridge mapping.

pyproject.toml pins exclude-newer-package cutoffs as full UTC RFC 3339 timestamps so uv.lock does not churn across timezones.

Call sites (envs, dashboard, push, GEPA, tests) and docs are updated to the new trace fields; validate/debug synthesize AgentInfo on traces that never run a full agent.

Reviewed by Cursor Bugbot for commit de8d521. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Rename harness timeout to agent timeout and migrate Trace API to agent-scoped fields

  • Renames harness_timeout to agent_timeout across the stack: TaskTimeout.harnessTaskTimeout.agent, RolloutRun constructor, cap_remote_harness_timeoutcap_remote_agent_timeout, and Harbor taskset parsing.
  • Restructures Trace so agent (an AgentInfo object) is required; removes top-level convenience properties agent_name, trainable, runtime, and error in favor of trace.agent.name, trace.agent.trainable, trace.agent.runtime, and trace.last_error.
  • Renames Trace.stampTrace.record_run, Trace.capture_errorTrace.record_error, and updates all call sites across CLI, eval runner, rollout, retries, push, and environment code.
  • Trace.tools now defaults to an empty list instead of None; tool_messages returns results across all graph nodes rather than only the last branch.
  • Updates uv.lock exclusion timestamps to full UTC format with an explanatory comment in pyproject.toml.
  • Risk: EvalRunInfo.id is now required (was optional); is_truncated no longer treats harness_timeout as a truncation condition; any code reading removed Trace properties will break.

Changes since #2172 opened

  • Reworded inline comments in verifiers.v1.cli.validate._run_gold and verifiers.v1.cli.validate._run_setup functions to clarify that no agent runs occur and the info only records the runtime policy [de8d521]
  • Added class docstrings to verifiers.v1.trace.TimeSpan and verifiers.v1.trace.TimeSplit classes documenting their timestamp and duration semantics [de8d521]
  • Updated comments in verifiers.v1.legacy.rollout_output_to_trace function to clarify that v0 rollouts contain no agent config rather than seat config [de8d521]

Macroscope summarized f0723ca.

mikasenghaas and others added 5 commits July 29, 2026 12:39
Bare dates resolve to midnight in the machine's local timezone, so every
timezone relocks uv.lock with different exclude-newer timestamps and the
--locked pre-commit hooks fail for anyone outside the tz that produced
the lock.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
stamp -> record_run, capture_error -> record_error, matching the
existing record_metric/record_reward/record_judge family and the trace's
own 'record' vocabulary (to_record, record schema). Also drop the
override warnings on record_metric/record_reward: overriding is defined
behavior, last write wins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rollout deadline expiring now records a HarnessError (ok=False, no
scoring) instead of the clean harness_timeout stop that scored the
partial trajectory: a timeout is the agent breaking its time budget,
not a healthy run cut short. The stop-condition name is gone from the
vocabulary; the legacy v0 bridge's timeout_reached maps through the
generic truncation fallback.

The concept is renamed harness timeout -> agent timeout throughout:
TaskTimeout.harness -> TaskTimeout.agent, the rollout plumbing, and
cap_remote_harness_timeout -> cap_remote_agent_timeout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the agent_name/trainable/runtime passthroughs (read trace.agent
directly) and the duplicate error property (last_error is the one
reader); align tool_messages with assistant_messages (nodes-based,
branch-independent); require an explicit stop condition (the 'done'
default was never used); correct the stop_condition docstring to the
real vocabulary; tighten field docstrings and ordering.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread verifiers/v1/task.py
Comment thread verifiers/v1/trace.py
mikasenghaas and others added 2 commits July 29, 2026 14:13
agent: every producer sets a seat — the rollout its resolved config, the
debug CLI its synthetic seat, and now the v0 bridge and validate CLI
theirs — so the None guards at every consumer were dead weight.
verifiers: stamped by default_factory at construction; stored records
keep their serialized build. tools: the empty state was unreachable
(dialects normalize [] to None to avoid clearing a recording), so None
carried no signal over []. Bumps TRACE_VERSION to 5: old records
without an agent no longer validate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread verifiers/v1/trace.py
Comment thread verifiers/v1/trace.py
Comment thread verifiers/v1/trace.py
Every consumer stamps a run id (the eval CLI its uuid, trainers their
own), so the None default was unreachable. Docstrings across the trace
models trimmed to the constraint they actually add; kept_tokens aligned
with routed_experts' shape-first style.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread verifiers/v1/trace.py
mikasenghaas and others added 5 commits July 29, 2026 14:33
Agent.interaction's two borrowed-runtime stamps and the dashboard's
boot-vs-build stage probe still read the removed Trace.runtime
passthrough; all three now read trace.agent.runtime. Verified live:
eval with the rich dashboard renders and pushes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A version-gated before-validator upgrades v4 records to the v5 shape
(drop the explicit nulls v5 defaults now fill, seat records that
predate the required agent, drop an id-less run stamp) so eval resume
and replay keep reading old outputs. Current-version records validate
strictly: a v5 record with tools=null is rejected. Delete the validator
when v4 support is dropped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The migrate hook sits at the bottom of Trace and chains one _to_v<n>
util per schema bump; the next bump adds _to_v6 and a new chain link.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One to_v<n> util per historical schema bump, chained by the migrate
hook: v1->v2 lifts node-level usage/finish_reason into synthesized
ModelCalls (#2061), v2->v3 nests the flat agent identity and top-level
runtime into AgentInfo (#2106), v3->v4 wraps float rewards as
Reward(score, weight=1) preserving reward sums (#2119), v4->v5 as
before. Steps copy what they mutate, so validating the same dict twice
is stable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tightened schema restarts the version counter; pre-existing records
are not loadable and old eval runs cannot be resumed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mikasenghaas
mikasenghaas requested a review from hallerite July 29, 2026 22:21
@mikasenghaas
mikasenghaas marked this pull request as ready for review July 30, 2026 00:04

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 58e3a6a. Configure here.

Comment thread verifiers/v1/trace.py
@macroscopeapp

macroscopeapp Bot commented Jul 30, 2026

Copy link
Copy Markdown

Approvability

Verdict: Needs human review

This PR makes significant API/schema changes to the Trace model: making agent required, renaming properties and methods, and changing timeout handling from a clean stop condition to a failure. These are substantive contract changes beyond a typical 'chore' refactor and warrant human review.

You can customize Macroscope's approvability policy. Learn more.

Comment thread verifiers/v1/legacy.py Outdated
Comment thread verifiers/v1/cli/validate.py Outdated
Comment thread verifiers/v1/cli/validate.py Outdated
Comment thread verifiers/v1/trace.py
Comment thread verifiers/v1/trace.py

@hallerite hallerite left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few comments, almost lgtm

@mikasenghaas
mikasenghaas merged commit f646beb into main Jul 30, 2026
13 checks passed
mikasenghaas added a commit to PrimeIntellect-ai/prime-rl that referenced this pull request Jul 30, 2026
* rename training environments to sources

* fix CPU test config fixtures

* rename evaluation environments to sources

* chore(orchestrator)!: rename max_inflight_rollouts to max_inflight_episodes

A dispatcher permit has always bought one episode, not one rollout: for a v1
env one `run` request is one episode (however many agents play it), and a
legacy group-scoring call reserves one permit per bridged v0 rollout, each of
which is its own single-agent episode. Name the knob what it counts.

`max_inflight_rollouts` still parses as an alias, as `rollouts_per_example`
does for `group_size`.

* feat(orchestrator)!: compose the verifiers env, serve and legacy blocks

verifiers split `EnvServerConfig` into three blocks, so an env entry composes
them instead of inheriting one class:

    [[orchestrator.train.source]]
    env    = { taskset = { id = "..." } }   # what runs
    serve  = { pool = { ... }, address, max_concurrent }   # how it's hosted
    legacy = { id, args }                   # a classic v0 env instead

`serve.pool` sizes the spawned server (was `pool`), `serve.address` points at an
external one (was `address`), and `serve.max_concurrent` bounds one worker's
episodes in flight — usually left unset, since `max_inflight_episodes` is the
run's bound. Each moved key raises with a pointer home, including the
`num_workers` shorthand the old validator migrated silently.

BREAKING: `pool` / `address` / `num_workers` on an env entry move under `serve`,
and a v0 env's `id` / `args` / `extra_env_kwargs` move under `legacy`.

TEMPORARY: `deps/verifiers` points at the `feat/one-agent-per-episode` branch
(PrimeIntellect-ai/verifiers#2157) so the two sides can be run together; re-pin
to a merged `main` commit before this lands.

* fix the source-config readers the compose commit missed

The Prime monitor still built its run-registration payload from `env.id`, which
the composed entry doesn't have — `AttributeError` at registration whenever
Prime monitoring is on. It reads `env_id` now, the property that spans both the
v1 and the v0 shape.

The packer's fixture still used the bare `id` shorthand, so its three runs were
rejected at config parse and skipped by `discover_runs`. Full CPU unit suite:
416 passed.

Re-pins `deps/verifiers` to the branch tip of verifiers#2157, which fixes the
same class of stale access on that side (`config.legacy.id`). Still a branch
commit — re-pin to a merged `main` before this lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* spell the env annotation out instead of vf.EnvField

verifiers dropped the `EnvField` alias, which was only ever
`SerializeAsAny[EnvConfig]` under a name that collided with `env_field()`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* declare the env default with the factory verifiers exports

`vf.env_field()` is gone: it only wrapped `Field(default_factory=...)`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* inline the legacy helpers, use the config class as the default

Follows verifiers dropping the three `configs/legacy.py` helpers and the
`single_agent_env_config` factory: `is_legacy` and `env_id` are one-liners here,
the mixed-run refusal is spelled out in `validate_env`, and the `env` default is
`vf.SingleAgentEnvConfig()`. `env.max_concurrent` now works as an alias for
`env.max_concurrent_agents`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* drop the serving-key pointers here too

The same call as verifiers a0ebc408: `num_workers` / `pool` / `address` on a source
entry now fail as an ordinary `extra_forbidden` instead of naming their new home.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* bump verifiers: no max_concurrent alias on the env block

`env.max_concurrent` used to bound a served worker's agent runs across episodes, so
accepting it as `max_concurrent_agents` multiplied it by the episodes in flight. It
errors now, like every other moved key. 416 unit tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* bump verifiers to 0.2.2.dev43 (merged main)

deps/verifiers d0bb0ffe5 -> fcb48822e, which is the squash-merge of the composed
env/serve/legacy config blocks (#2157) plus everything else since the last bump:
Harbor 0.20.0 (#2161), runtime-blamed box failures (#2156), standalone MCP clients
pinned to v1 (#2155), and the v1 test/E2E split (#2127).

Lock: harbor 0.14.0 -> 0.20.0, which drops claude-agent-sdk and datasets from its
own deps and adds filelock/platformdirs, so those move with it.

Replaces the temporary branch pin this PR carried.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: spawn the env server again, and let a pip-only install resolve verifiers

Two CI failures from the compose commit.

The orchestrator stopped spawning env servers. It decides spawn-vs-connect by
`address is None`, and prime-rl's own field defaulted to None; `vf.ServingConfig`
defaults to `tcp://127.0.0.1:5000`, so every env looked external and the run waited
120s for a server nobody started. It now asks whether the address was *pinned*
(`model_fields_set`), so an unset `[serve]` spawns on a free port as before, and
pinning any address — including verifiers' own default — connects.

The slim `prime-rl-configs` wheel is installed from /tmp with no workspace context,
so `verifiers` resolves from PyPI, where `>=0.2.1` picks stable 0.2.1 — no
`SingleAgentEnvConfig`. Floor is `>=0.2.2.dev43` now, a pre-release specifier so the
dev release is eligible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: adapt to the verifiers trace record cleanups

Companion to PrimeIntellect-ai/verifiers#2172: record_* mutators, required
agent seat and RunInfo.id, last_error, and agent.name/.trainable reads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: re-pin verifiers to merged main (f646beb37)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: hallerite <git@hallerite.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
jooooonas added a commit to p-doom/prime-rl that referenced this pull request Jul 31, 2026
* fix(trainer): don't resolve Hub envs when parsing orchestrator configs (#3096)

* fix(trainer): don't resolve Hub envs when parsing orchestrator configs

discover_runs() builds OrchestratorConfig to read training-relevant fields, but
constructing a v1 env config eagerly resolves (imports/installs) its taskset/
harness plugins from the Environments Hub. On the trainer that 404s for private
hub envs (taskset.id = owner/name@version) — the trainer has no Hub credentials —
so get_orchestrator_config() returns None and the run is silently skipped.

Parse under verifiers' skip_plugin_install contextvar so the trainer never
touches the Hub; the env server still installs the plugin at runtime.

Depends on verifiers exposing `skip_plugin_install` — bump deps/verifiers to a
commit that includes it before this lands.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(trainer): sort imports to satisfy ruff isort (I001)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: bump verifiers to 0.2.2.dev7, adopt the multi-agent episode wire (#3104)

* feat: bump verifiers to 0.2.2.dev7, adopt the multi-agent episode wire

Companion to PrimeIntellect-ai/verifiers#1939 (multi-agent api). Re-pins
deps/verifiers to 192fd8ef0 (= 0.2.2.dev7) and deps/pydantic-config to
v0.4.2 (verifiers now requires prime-pydantic-config>=0.4.2).

- Serve wire: the env-server route run_rollout -> run, answering one
  Episode envelope ({id, env, ok, errors, traces}) per env-rollout.
  Env.run returns list[Rollout]; an episode that failed before minting
  any trace raises (the dispatcher synthesizes its error marker), and a
  not-ok episode marks its clean traces failed so a broken episode's
  partial seats are never trained on.
- The episode is the pipeline currency: the dispatcher emits one episode
  (list[Rollout]) per completed task onto out_q (a legacy run_group
  result emits its traces as single-trace episodes), and both sinks
  count episodes - never loose traces - toward group_size and eval epoch
  sizes. episode_id is stamped into info so saved records keep grouping.
- ok is the success sentinel: has_error now reads it, so the dispatcher's
  empty-trajectory promotion also flips ok, and error markers carry it.
- Only the trainable subset becomes training samples: a trace stamped
  untrainable (a multi-agent env's frozen agent) is never tokenized,
  never enters the group advantage, and never ships; it still lands in
  the trace files and observation window.
- Config shape: everything env lives under the env block, following the
  new vf.EnvServerConfig (env.taskset, env.agent.harness, per-agent caps
  under env.agent.*). All checked-in TOMLs migrated; the spawned env
  server receives the picklable env block via env_config_data.
- run_group survives only for v0/legacy bridge envs (a v1 server refuses
  it and always reports requires_group_scoring=False); vf's @group_reward
  removal doesn't touch prime-rl's own Algorithm.score_group.

Verified: tests/unit green (487 passed; config-loading params covered all
migrated TOMLs), ruff clean, and a 3-step reverse-text RL run on 2 GPUs
trains at 0% errors with every saved trace carrying ok, agent.name,
agent.trainable, and a distinct info.episode_id.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: trim narrative comments to the load-bearing facts

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: count buffered sink progress in episodes; explicit ok=False on cancel markers

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: keep untrainable traces out of the effective views and the derived avg@k

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* feat(weight-transfer): add NIXL + ModelExpress synchronization (#3060)

* feat(weight-transfer): add NIXL ModelExpress synchronization

* refactor(weight-transfer): simplify NIXL integration

* refactor(weight-transfer): simplify NIXL worker planning

* refactor(weight-transfer): align MX update lifecycle

* feat(inference): pass through online quantization methods

* feat(weight-transfer): add GLM trainer conversion graph

* feat(weight-transfer): stream NIXL updates by layer

* feat(weight-transfer): pipeline NIXL layer reloads

* fix(weight-transfer): preserve single-buffer NIXL behavior

* fix(weight-transfer): reserve observed inference peak

* feat(weight-transfer): preserve model-declared FP32 tensors

* cleanup(weight-transfer): simplify trainer wire schema

* cleanup(weight-transfer): make trainer shard ranges explicit

* cleanup(weight-transfer): resolve model conversions through registry

* cleanup(weight-transfer): clarify lazy copy destinations

* cleanup(weight-transfer): derive lazy metadata from source

* cleanup(weight-transfer): rename weight load tracing

* cleanup(weight-transfer): name recorded tensor operations

* cleanup(weight-transfer): simplify transfer planning

* cleanup(weight-transfer): consolidate load graph

* cleanup(weight-transfer): name trainer tensor table

* cleanup(weight-transfer): clarify tensor routing

* cleanup(weight-transfer): simplify NIXL adapter

* cleanup(weight-transfer): clarify ModelExpress sessions

* cleanup(weight-transfer): clarify CUDA memory sizing

* cleanup(weight-transfer): clarify staged trainer shards

* cleanup(weight-transfer): simplify trainer broadcast guards

* cleanup(weight-transfer): clarify trainer staging state

* cleanup(weight-transfer): simplify trainer constants

* cleanup(weight-transfer): use ModelExpress client name

* cleanup(weight-transfer): clarify trainer shard discovery

* cleanup(weight-transfer): structure trainer initialization

* cleanup(weight-transfer): reuse trainer agent metadata

* cleanup(weight-transfer): reuse trainer table fragments

* cleanup(weight-transfer): simplify trainer staging setup

* cleanup(weight-transfer): simplify inference transfer planning

* cleanup(weight-transfer): split inference plan construction

* cleanup(weight-transfer): clarify inference replay methods

* cleanup(weight-transfer): localize receive arena sizing

* cleanup(weight-transfer): clarify loader recording wrapper

* cleanup(weight-transfer): clarify watcher synchronization

* cleanup(weight-transfer): unify checkpoint wait timing

* cleanup(weight-transfer): align in-memory broadcast lifecycle

* fix(weight-transfer): skip NIXL startup filesystem wait

* cleanup(weight-transfer): use declarative model conversions

* style(weight-transfer): format NIXL implementation

* fix(weight-transfer): stabilize ranks and final watcher step

* fix(tests): use instance conversion API

* cleanup(weight-transfer): scope package to NIXL

* cleanup(weight-transfer): align names and package layout

* chore: bump research-environments submodule to latest main (#3106)

* chore: bump research-environments submodule to latest main

Picks up the registry ref-cleanup wave: platform image naming convention
(openswe/tmax/terminal-lego/openthoughts-tblite/senior-swe-bench switched to
org-less platform refs, multiswe REGISTRY mapping removed) and the tmax Harbor
registry pointer bump.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: relock env versions for research-environments bump

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* feat(monitor): add local file monitor (#3111)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* feat: orchestrator owns v1 tasksets, ships task data to env servers (#3043)

* feat: orchestrator owns v1 tasksets, ships task data to env servers

Companion to PrimeIntellect-ai/verifiers#2039. A v1 env's taskset is loaded once,
in the orchestrator: finite tasksets become the train source's shuffled example
table (real tasks, not index ranges); an infinite taskset's generator is pulled
per example — no server-side materialization, no idx-addressed cache, no
requirement that every pool worker regenerate the same sequence. Each dispatched
rollout ships its task's data (task.data.full_dump()) and the env server
pydantic-validates and runs it.

The legacy (v0) bridge keeps its server-side dataset and task_idx addressing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump deps/verifiers to client-side-tasksets tip

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump deps/verifiers to client-side-tasksets tip

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump deps/verifiers to client-side-tasksets tip

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: plain model_dump task payload; bump deps/verifiers (full_dump removed)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: re-land client-side tasksets on the episode wire

Companion to the rebased PrimeIntellect-ai/verifiers#2039. A v1 env's taskset is
loaded once, in the orchestrator: finite tasksets become the train source's
shuffled example table (real tasks, not index ranges); an infinite taskset's
generator is pulled per example. Each dispatched env-rollout ships its task's
data (task.data.model_dump()) and the env server validates and runs it. The
legacy (v0) bridge keeps its server-side dataset, task_idx addressing, and the
run_group route.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump deps/verifiers (non-optional taskset); drop redundant assert

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: actually drop the redundant taskset assert

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump deps/verifiers (content-identity resume)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump deps/verifiers (simplified task_key)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump deps/verifiers

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump deps/verifiers

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump deps/verifiers to main (client-side tasksets merged, vf#2039)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: one Env.tasks iterator instead of tasks/task_iter

Finite tasksets still materialize at start() (num_tasks needs the count and the
train source shuffles epochs), but the field is a single iterator either way —
consumers pull it uniformly and branch on num_tasks alone. Consumed once, by
TrainSource or EvalEnv.start.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* fix(trainer): use UE8M0 FP8 scales only on SM100 (#3112)

#3022 flipped use_ue8m0 to True unconditionally to support DeepGEMM on
SM100, which also switched SM90 to power-of-2-rounded scale factors —
a pure precision loss on Hopper. On a GLM-5.2 RL prod run this doubled
trainer/inference mismatch KL (0.0038 -> 0.0059 at step 1, verified
A/B on an identical batch: same tokens and inference logprobs read
0.0059 with UE8M0 and 0.0038 with float scales).

Derive use_ue8m0 from the device (compute capability >= 10) so SM100
keeps the required UE8M0 recipe and SM90 returns to exact float scales.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* feat(rl): pack multimodal samples with context parallelism (#3103)

* feat(rl): pack multimodal samples

Multimodal RL samples no longer force one-sample micro-batches: eager
image samples pack with text spans and compatible mm_kwargs samples from
the same run/LoRA, with per-sample boundaries kept in seq_lens. Packing
is always on — VLM training is custom-implementation-only and the custom
Qwen3.5 models advertise supports_packed_multimodal_training; a config
validator requires varlen flash attention for VLM training and
validate_multi_modal_pack fails loudly at startup for models without
packed-MM support.

Ported from #2889 minus its gating (pack_multimodal knob, resolve_pack_multimodal)
and its tokenizer packer params (removed on main since).

(cherry picked from commit 027edc2c510c35ab5666e8f9cbf160f58b1fba48)

* fix: fail loudly when RL micro batches carry multimodal data without [model.vlm]

Mirrors the SFT-side check: transported mm samples would otherwise flow
into forward unvalidated (no packability, attn, or freeze policy checks).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(rl): preserve multimodal packing across workers

Keep multimodal bins intact during worker balancing and reject malformed multimodal samples before they can stop shared multi-run packing.

* fix: sync docs to public-docs on stable release only (#3116)

* fix: sync docs to public-docs on release tags only

Change the Sync Docs to Public Docs workflow to trigger on stable v*
tag pushes instead of every commit to main that touches docs/, so the
public docs only update on releases.

Exclude v*dev* tags so dev pre-release tags never trigger a sync.

Keep workflow_dispatch for manual syncs.

On a tag push github.sha resolves to the tagged commit, so the ref sent
to public-docs remains correct.

Mirrors PrimeIntellect-ai/verifiers#2108.

* fix: trigger docs sync on release published, not on main push

Change the Sync Docs to Public Docs workflow to trigger when a stable
GitHub release is published instead of every commit to main that touches
docs/, so the public docs only update on actual stable releases.

Dev tags (v*dev*) are created via GITHUB_TOKEN which doesn't retrigger
workflows, and they don't have GitHub Releases, so using the release
event with types: [published] naturally excludes them.

Keep workflow_dispatch for manual syncs.

On a release publish, github.sha resolves to the tagged commit, so the
ref sent to public-docs remains correct.

* chore(rl): add message to multimodal packing assert (#3117)

Follow-up to #3103 review feedback: make the mm_kwargs invariant assert self-describing when it fires.

Co-authored-by: eligotts <eligotts@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(orchestrator): hold batch ship on the trainer's published version (#3050)

* fix(orchestrator): enforce target lag strictly at ship time

The dispatch gate only stops new rollouts; on tiny fast envs the sink
keeps forming batches from buffered rollouts, so the orchestrator ships
past TARGET_LAG, finishes, and tears down the watcher while the trainer
still has pending NCCL broadcasts. Those broadcasts then block forever
in the STABLE->NCCL_READY handshake (no watcher to trigger the inference
receive) and the launcher never exits.

hold_for_target_lag blocks shipping batch N until the trainer has
published v{N-1-TARGET_LAG}, bounding in-queue staleness and keeping the
orchestrator alive for every broadcast the trainer will make. Under NCCL
the last published version is max_steps-2, exactly what the final batch
requires, so the hold never waits on an unpublishable version.

Also logs a per-batch staleness decomposition (pre-queue = versions
elapsed during generation, in-queue = versions spent waiting in the
sink) under staleness/*.

Fixes RES-1077

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* remove staleness metrics from ship path

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(orchestrator): step-based staleness with demand-driven dispatch

Measure a rollout's off-policy lag against the batch that consumes it —
batch N trains on policy v{N-1}, so lag = (N-1) - generation version,
queue time included. The old off_policy_steps counter only counted
weight updates during generation, so a rollout that finished fast but
sat buffered across steps trained arbitrarily stale while reporting 0.

- max_off_policy_steps now bounds consumption lag: in-flight groups are
  cancelled pre-pause (cancel_stale_train_groups) and buffered sink
  survivors are swept whenever the cutoff moves (ship, weight update).
  Groups share one generation version, so stale groups drop whole.
- Dispatch is demand-driven: the orchestrator injects train_demand
  (batch shortfall minus buffered + partial-group + in-transit
  rollouts), re-evaluated per scheduled group, replacing the lead-based
  dispatch gate and the #2990 final-batch NCCL exemption it needed.
  out_q backlog counts as coverage so ship-holds don't over-provision.
- Max Off-Policy in logs is now the honest consumption lag, stamped at
  ship (reverse_text: steady-state max 3 vs a dishonest 0 before).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review pass: bool dispatch contract, oversampling-aware demand, inline single-use helpers

- train_demand -> train_needed: the dispatcher only thresholds the value,
  and the magnitude isn't even unit-stable across batch modes, so the
  injected contract is a bool predicate.
- Demand target is max(batch_size, max_inflight_rollouts): explicit
  over-provisioning (oversampling_factor > 1 or a raised max_inflight)
  keeps racing batch stragglers instead of being silently capped at one
  batch; capacities below the batch keep a batch_size target so the
  batch can still complete.
- Reject max_off_policy_steps=0 with NCCL broadcast + max_steps: the
  final batch necessarily trains one version off-policy (the trainer
  never broadcasts the last versions), so a zero budget would hang the
  run collecting the final batch.
- Inline single-call helpers (hold_for_target_lag, stale_cutoff, the
  dispatcher lag gauges) into their call sites.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* require max_off_policy_steps >= 1 at the field

The last NCCL batch necessarily trains one version off-policy (the
trainer never broadcasts the final versions), so a zero budget can never
be met and would hang the run collecting the final batch. A field floor
covers this without a conditional validator.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix CI regressions: partial-group dispatch deadlock + serialized weight applies

Two regressions surfaced by the PR's GPU CI run:

- alphabet_sort/rl_opd deadlocked at 126/128 (+3 buffered, 0 inflight):
  demand hit zero mid-group, so an open group's remaining rollouts were
  never scheduled — its buffered members count as batch coverage but the
  group can never finalize. Already-open groups now keep scheduling
  regardless of demand (try_schedule allow_fresh); only fresh groups are
  demand-gated.
- reverse_text timed out (~26s/step vs ~19s on main): the ship hold
  waited on the applied version, serializing every inference weight
  reload into the ship path. Filesystem broadcast decouples publish from
  apply and the trainer never blocks on the watcher there, so the hold
  now releases on the published version (tracked in on_version_pending).
  NCCL keeps waiting for the apply — the trainer blocks in the receive
  handshake until then, so releasing on publish could strand its final
  handshake at teardown.

Verified: reverse_text e2e clean on both broadcast types (filesystem
6-7s steps, NCCL exits clean), 485 unit tests pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* raise multi-run integration timeout to 600s

Resumed orchestrators now pace to the shared trainer — a batch only
ships once the trainer has produced the required policy version — so
with three runs round-robining one trainer, beta_resume's tail (steps
21-25) can legitimately exceed the old 300s wait while its versions
queue behind the other runs. Matches the 600s budget of the other RL
integration tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* retract stale-dropped rollouts from the payload-size estimate

drop_stale now mirrors what entering pending_batch added to
_payload_total/_payload_count, keeping the token-mode demand estimate
consistent with the buffered set. Also expand the allow_fresh comment
with the group-finalization invariant it protects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* upload integration run logs as artifacts on failure

The multi-run test's failures only surface a sampled tail of one
orchestrator's output; the copied per-run orchestrator/trainer/inference
logs are what's needed to debug them (currently: beta_resume wedging on
a policy version the shared trainer never publishes after resume).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* keep CI test outputs for the failure-artifact upload

The output_dir fixture rmtree'd the whole outputs dir at session
teardown, so the upload step found nothing. Skip the cleanup under CI
(the workspace is discarded with the job) and match log files directly
instead of a directory glob.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* dispatch lookahead: cover TARGET_LAG extra batches when freshness allows

Exact-one-batch demand synchronized dispatch into one 128-rollout wave
per batch: each ship opened demand 0->128 at once, the wave stampeded
the env server together (MCP timeouts + backoff retries in CI
alphabet_sort), and inference idled ~60s per cycle while the whole wave
sat in env round-trips — tripling the step cadence vs main (artifact
logs: vLLM alternating 30s busy / 60s '0 running, 0 waiting').

With up to TARGET_LAG batches of lookahead — allowed only while a
rollout landing that far ahead still fits max_off_policy_steps —
steady-state dispatch is continuous (each arrival frees demand for the
next start), keeping the engine fed through env-phase gaps. Costs one
version of staleness on the lookahead batch (reverse_text plateau 3->4,
within the default cap of 8).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* raise reverse_text mismatch-KL budget to 0.02

Mismatch KL compares generation-time inference logprobs against the
training weights, so it includes async training's policy-version gap on
top of trainer/inference numerics. With strict pacing + dispatch
lookahead the steady-state consumption lag is 2-3 versions (visible and
bounded, vs unmeasured before), which puts the last-steps average at
~0.011-0.012 against the old 0.01 budget calibrated for main's profile.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* raise alphabet_sort and multi-run integration budgets

Artifacts from the failed runs show no wedges — every process was
mid-progress at the deadline: alphabet_sort cycles at ~70s on slow vm
runners (33s on fast ones; trainer MFU varies 2-6% across the pool) and
the multi-run trainer was mid-micro-step serving three runs
round-robin. With strict pacing the orchestrators ride the trainer's
pace instead of finishing early on stale buffers, so wall time tracks
the slowest component.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* raise multi-run inner milestone waits to 600s

The wait_for_log/wait_for_file 300s defaults were calibrated for
orchestrators that sprint ahead of the shared trainer on buffered
batches; with strict pacing three orchestrators ride the trainer's
actual speed, so every internal milestone (alpha steps, checkpoint
STABLE markers, resume completion) lands slower and a different 300s
wait tripped on each CI round.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* count group-scored partial arrivals as batch coverage

buffered_count hides group-scoring envs' partial groups for the
pipeline log's honesty, but demand must count them — otherwise members
sitting between arrival and group completion invite bounded
over-dispatch. Also document why a held batch intentionally does not
count as coverage (dispatch-ahead keeps inference busy through
trainer-bound holds; staleness stays bounded by lookahead + cutoff).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* guard token-export STABLE markers against deleted run dirs

The multi-run trainer crashed with FileNotFoundError touching
run_gamma/token_exports/step_20/STABLE: the multi-run integration test
rmtree's a run's dir once its orchestrator finishes, while the trainer
is still flushing that run's tail steps. The broadcast paths already
skip deleted runs on FileNotFoundError; mark_stable now does the same.

This was the root cause of the recurring multi_run CI failure: the
crashed trainer never published the resumed run's next version, so
beta_resume's ship-hold waited forever ('failed with code None'). On
main the same crash risk exists but the orchestrator never waited on
the trainer, so nothing surfaced.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* recalibrate multi-run early-step reward floor for strict pacing

Batch N's data now comes from v{N-2}..v{N-4} instead of main's
effectively-fresher buffer profile, so reward-at-step-7 reads 1-2
policy versions staler (observed 0.14-0.20 across four CI rounds vs the
0.2 floor). The final-reward thresholds still enforce learning quality;
the early floor drops to 0.1 — comfortably above a broken run's flat
~0.05-0.08.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* raise rl_sft integration budget to 900s

Same class as the other RL integration budgets: the trainer was
SIGTERM'd at step 14 with the orchestrator already finished — killed by
the 600s deadline steps from completion on a slow runner.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* lower multi-run final reward floor to 0.6

With strict pacing the reward-vs-step-index curve sits 1-2 policy
versions lower throughout; gamma — the most trainer-contended of the
five runs — landed at 0.627 while its siblings cleared 0.69-0.75. A
broken run still reads ~0.1-0.3.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* bound dispatch freshness at DISPATCH_LAG instead of the drop cap

Root cause of the reward-at-step gap vs main: the old lead gate doubled
as the dispatch freshness bound (rollouts only started at version >=
step-2, landing lag <= 2). The demand rewrite left freshness gated only
by max_off_policy_steps (8) plus a deliberately-ahead lookahead batch,
so mean consumption lag ran 2-4 versions vs main's 1-2 — visible as
Max Off-Policy 3-4, mismatch KL ~2x main's, and reward-at-step lower by
1-2 versions of learning.

Dispatch now pauses when new rollouts would train more than
DISPATCH_LAG = TARGET_LAG+1 versions behind their generation policy,
with the lookahead scaled inside that budget — the tightest bound that
still keeps inference busy during ship-holds. Landing lag returns to
<= 2 (spill 3); local e2e: Max Off-Policy plateau 4 -> 3, mismatch KL
mean ~0.012 -> ~0.008. max_off_policy_steps returns to being purely the
drop safety net.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* ci: retrigger GPU tests

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* tolerate trainer tail-flush when deleting run dirs in multi-run test

The fixture's rmtree races the shared trainer's token-export writes,
which mkdir the run dir back mid-deletion (OSError: Directory not
empty). Retry until the flush for the finished run drains.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* restore NIXL ModelExpress handshake signals in the version hooks

The merge of #3060 kept this branch's on_version_pending/on_new_version
bodies and dropped main's ModelExpress status updates — the READY
signal the NIXL trainer waits on before each broadcast and the
INITIALIZING reset after it. Post-startup NIXL runs would hang in the
handshake. Both signals are back, ordered as on main (READY pre-update,
INITIALIZING post-update) around this branch's staleness/wake logic.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* reduce to the minimal ship-hold fix + staleness docs

Strip the demand-driven dispatch, consumption-lag drop rework, and test
recalibrations accumulated on this branch back to main's behavior, and
keep only what the spec (docs/staleness.md) strictly requires:

- ADVANCE rule: batch N ships only once the trainer has published
  v{N-1-TARGET_LAG}, so the orchestrator cannot finish and tear down the
  weight watcher while the trainer still has in-memory broadcast
  handshakes pending (RES-1077). Main's dispatch gate (START) and
  in-flight drop aging (DIE) are untouched, so generation freshness and
  throughput are unchanged in both pacing regimes — the hold only moves
  where a finished batch waits, never when the trainer consumes it.
- policy.version advances at publish confirmation (inference applies
  immediately and pauses during the update, so published and applied
  are indistinguishable to consumers), and WeightWatcher.stop() drains
  an in-flight apply before cancelling — teardown can never strand the
  trainer mid-handshake.
- Keep the trainer-side token-export guard, the multi-run test's
  deletion-race helper, and the CI failure-artifact pipeline from the
  earlier iterations; drop everything else.

Verified on the original repro (reverse_text, fast env, NCCL): 10/10
batches, clean trainer exit, 2 holds at the buffer-fed tail.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fold staleness spec into docs/training.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* drop the docs section for now, trim the hold comment

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* never delete the test output dir; clean at run start instead

The output_dir fixture deleted the whole workspace at session teardown,
racing the failure-artifact upload (and hiding logs from post-mortems
generally). Keep it unconditionally and make fresh starts responsible
for their own hygiene: alphabet_sort and reverse_text_moe now pass
--clean-output-dir like the other rl-launcher tests. Resume runs are
unaffected — validate_output_dir returns before cleaning when resuming.
The multi-run orchestrators manage per-run dirs the test creates (and
the standalone orchestrator entrypoint has no clean flag).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* stamp true consumption staleness on shipped rollouts

Batch N trains on policy v{N-1}, so a rollout generated from v{k} is
(N-1)-k versions off-policy — queue time included. The dispatcher's
in-flight counter only sees weight updates during generation, which
undercounts by the queue-wait aging and the dispatch-time baseline gap;
the ship-time stamp overwrites it so Max Off-Policy in logs and the
rollout records report the real number. The counter itself (and the
max_off_policy_steps drop rule it feeds) is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* use the deletion-retry helper for alpha's run dir too

Alpha is SIGTERM'd mid-run, so the trainer is actively flushing its
token-export tail when the dir is deleted — the recreate race is most
likely at exactly this site.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* fix(nixl): replay strided views on inference (#3113)

* feat(ci): nightly FFT workflow (#3118)

* feat(ci): add nightly FFT workflow dispatching hosted training runs

* ci(nightly-fft): pin --image-tag to the latest published dev tag

* ci(nightly-fft): rename to reverse-text and align with pytest nightly setup

* ci(nightly-fft): pin deployment shape (2 train + 6 infer)

* ci(nightly-fft): switch to wordle mirroring examples/basic/wordle/rl.toml

* ci(nightly-fft): update wordle config to match examples/basic/wordle

* ci(nightly-fft): add explicit least-privilege permissions block

* ci(nightly-fft): drop top-level type/name — [deployment] alone triggers FFT dispatch

* ci(nightly-fft): resolve image tag from GHCR to avoid unpublished tags

* ci(nightly-fft): forward WANDB_API_KEY and HF_TOKEN into hosted payload via -e

* fix(trainer): skip no-op weight conversion for dense models (#3069)

Dense models (Qwen3, Qwen3.5, Llama) use identical key names in both
HF and PrimeRL weight formats, so is_hf_state_dict and is_prime_state_dict
both returned True for the same state dict. The auto-conversion logic in
load_dcp_from_hf would fire anyway, performing a no-op convert and
attempting to save the unchanged weights into the snapshot directory —
which crashes with OSError [Errno 30] Read-only file system when the
model cache is mounted read-only.

Fix: make is_hf_state_dict / is_prime_state_dict mutually exclusive for
dense models by having is_prime_state_dict return False (following the
pattern already used by GPT-OSS). This makes the conversion branches
naturally skip when there is no format difference, without needing
extra guards in load_dcp_from_hf.

MoE models are unaffected: their is_hf_state_dict / is_prime_state_dict
checks are already mutually exclusive (expert.0.w1 vs experts.w1 key
patterns).

* fix(trainer): skip identity HF↔PrimeRL conversion (fixes RO model-cache crash on dense Qwen3) (#3124)

* fix(trainer): skip identity HF↔PrimeRL conversion to avoid RO-cache write

* chore: ruff format

* refactor: drop inline explanation; gate branches on unambiguous snapshot format

* fix(ci): build ghcr image on release via explicit dispatch (#3125)

Releases are published with GITHUB_TOKEN, whose events do not retrigger
workflows, so build_image.yaml's `on: release` trigger never fired and
release commits never received a GHCR image. Dispatch build_image.yaml
explicitly from tag-and-release.yaml after publishing (workflow_dispatch
is exempt from the no-retrigger rule), and drop the now-dead `on: release`
trigger plus its unreachable github.event.release.* references.

* ci(nightly-fft): name RFTRun nightly-<config>-<date>-<run_id>; verify GHCR manifest exists before dispatch (#3121)

* ci(nightly-fft): add alphabet-sort, reverse-text, wiki-search, hendrycks-sanity (#3127)

* ci(nightly-fft): add alphabet-sort, reverse-text, wiki-search, hendrycks-sanity, multimodal configs

Broaden nightly-fft coverage beyond wordle. Each config mirrors wordle's
2 train + 6 infer shape and pulls model/taskset/renderer from the
corresponding configs/basic/*/rl.toml.

- multimodal-color-codeword moved from configs/ci/nightly/ (its pytest
  path updated); nightly/ is now empty
- hendrycks-sanity trimmed from 5000 -> 200 steps to fit the nightly
  budget
- wiki-search carries tool_call_parser = 'hermes' for tool use

* ci(nightly-fft): revert multimodal move; restore hendrycks max_steps=5000

* ci(nightly-fft): size deployments per env; add zero-advantage filter to wiki-search

- reverse-text: 1+1 GPUs, batch 128 (was 2+6/512 — overkill for a 0.6B toy loop)
- alphabet-sort: 4+4 GPUs (was 2+6 — 4B FFT needs more trainer memory)
- wiki-search: 4+4 GPUs; adds oversampling_factor=2.0 + zero_advantage
  pre-batch filter mirroring examples/basic/wiki-search — without them sparse
  tool rewards fill batches with all-zero groups and abort the run

* test(nightly): remove LoRA alphabet-sort and wiki-search — covered by FFT nightly workflow

* ci(nightly-fft): align hendrycks-sanity with examples/basic upstream

- drop max_completion_tokens=1024 cap so R1-Distill can produce full CoT
- group_size 16 -> 8, GPUs 2+6 -> 4+4 (upstream)
- add trainer.model.seq_len=16384 for long-context training
- add aime2024 eval block (interval=50, group_size=32)

* fix(nightly-fft): move seq_len under [orchestrator] in hendrycks-sanity

RLConfig rejects setting shared top-level seq_len alongside
trainer.model.seq_len. Mirror examples/basic/hendrycks-sanity/rl.toml:
seq_len=8192 under [orchestrator] for rollout context, seq_len=16384
under [trainer.model] for long-context training.

* revert(nightly): restore test_alphabet_sort and test_wiki_search — keep tests/nightly/ untouched

* fix(examples): set cp=4 for GLM-4.5-Air advanced examples (#3129)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* fix(nightly-fft): pin hendrycks attn to flash_attention_2 (#3131)

DeepSeek-R1-Distill-Qwen-1.5B is Qwen2 arch, not in prime-rl's
custom-impl support list, so impl='auto' resolves to 'hf'. The
platform default attn='flash_attention_3' requires impl='custom',
which raises ValueError and the trainer crash-loops. Pin attn to
fa2 (works with hf impl on H200).

* feat(ci): allow pinning prime-rl image_tag on nightly-fft dispatch (#3132)

* feat(ci): allow pinning prime-rl image_tag on nightly-fft dispatch

Adds an optional image_tag workflow_dispatch input. Empty (default)
keeps the existing auto-resolve behavior (newest v*.dev* with a
published manifest). Non-empty pins the tag directly, after verifying
the manifest is pullable.

* fix(ci): sanitize image_tag input to avoid shell injection

Move the workflow_dispatch input through a step env var so a
malicious value containing $(...) or embedded quotes cannot break
out of shell context during expression substitution. Validate the
result against the OCI/Docker tag charset before curl or GITHUB_OUTPUT
sees it.

* fix(nightly-fft): don't force --image-tag when no dispatch input (#3133)

* fix(nightly-fft): pin image_tag to commit-c505e07a1 in each config

The default 'main' image the platform validator ships is baked ~50
commits behind tip and predates cb74ae2be (which added the env.
wrapper on train.env entries). It rejects env.taskset with
'train.env[0].env — Extra inputs are not permitted' and doesn't know
the 'prime-qwen3' renderer name, so every nightly orchestrator
crash-loops on validation.

- Pin top-level image_tag = 'commit-c505e07a1' in each nightly-fft
  toml. c505e07a1 is post-cb74ae2be and accepts the current v1 shape.
- Workflow: when no dispatch input is set, emit an empty tag and skip
  --image-tag on prime train so the toml pin wins. Explicit
  workflow_dispatch image_tag still overrides.
- Drop this pin once v0.7.1+ (with the v1 schema) is released and the
  backend default resolves to it.

* revert: drop image_tag pin from nightly-fft configs

Keep the workflow change (only pass --image-tag when explicit input);
rely on prime-cli's built-in fallback: --image-tag > toml image_tag >
backend default. No commit-hash pin baked into the checked-in configs.

* fix(nightly-fft): restore auto-resolve latest v*.dev* when no image_tag input (#3134)

* fix(nightly-fft): restore auto-resolve latest v*.dev* when no image_tag input

The nightly's whole point is to catch regressions on tip-of-main, so a
plain 'gh workflow run "Nightly FFT"' must dispatch against the newest
built v*.dev* by default. #3133 removed that fallback and made no-input
dispatches emit an empty tag, which falls through to the platform's
default 'main' alias — currently baked ~50 commits stale — so the run
crashes on validation.

Restores the original behavior: no input → auto-resolve latest dev.
Explicit image_tag input still overrides (from #3132). Dispatch step
always passes --image-tag since the tag is always populated now.

* fix(ci): guard dev-tag resolver against pipefail

GitHub Actions runs the shell under `bash -eo pipefail` by default.
Two failure modes:
- `grep -E ...` returns 1 when no v*.dev* tags exist yet
- `head -n 10` closes the pipe after 10 lines, SIGPIPE-ing `sort`

Either aborts the step before the explicit no-candidates guard runs.
Wrap the pipeline in set +o pipefail / set -o pipefail.

* chore: bump verifiers to latest main; adopt runtime-on-agent, Reward records, textarena player (#3137)

* chore: bump verifiers to latest main; adopt runtime-on-agent, Reward records, textarena player

deps/verifiers b13ba60da → 295a06505 (+12: interleaving agents + kuhn-poker-v1,
runtime moves from the harness onto the agent (#2106), trace rewards become
Reward{score, weight} records (#2119), wordle-v1 rides the rewritten
TextArenaEnv, pluggable harness skills, agentic-judge grading knobs).

Adoption:
- every `env.<agent>.harness = { ..., runtime = ... }` config line splits into
  `harness = { ... }` + `env.<agent>.runtime = ...` (configs/, examples/, docs/,
  skills/)
- wordle configs address the textarena seat as `env.player`
- reward metrics read the weighted `Reward.value` (CustomMetrics gains a value
  extractor); unit-test trace builders record `vf.Reward(score=...)`
- uv.lock re-locked for verifiers 0.2.2.dev29

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(tests): adopt Reward records in prime-monitor rollout builder

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* feat(orchestrator): RAE — role-conditioned advantage estimation for self-play envs (SPIRAL) (#3136)

* feat(orchestrator): RAE — SPIRAL's role-conditioned advantage estimation for self-play envs

Add the `rae` algorithm (SPIRAL, arXiv:2506.24119): each trainable agent
keeps an EMA baseline of its own rewards, and every trace's advantage is
its reward minus its agent's pre-update baseline. Group-relative baselines
mis-credit multi-agent self-play — in a zero-sum game the group mean is ~0
whatever the policy does, and centering both agents against it turns any
structural asymmetry (first-mover edge) into permanent credit. Baselines
are keyed per (env, agent) — the paper's per (game, role) — and any
group_size works, including 1.

configs/debug/algo/rae.toml runs shared-policy Kuhn poker self-play
(kuhn-poker-v1, both agents late-bound to the run's model).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(tests): drop RAE unit tests

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* chore(ci): enable nightly FFT cron (#3142)

* chore: bump verifiers + research-environments to latest main (#3147)

- deps/verifiers -> d0bb0ffe5 (network-policy alignment #2124, Codex MCP tools
  #2140, Ruff 0.16 / ty tooling, and everything since the last bump)
- deps/research-environments -> 6a2dee6fc (judges/swe agentic-judge setups #720,
  MMMU-Pro / CharXiv tasksets, wideseek verifiers-pin fix #731)
- uv.lock relocked (adds charxiv-v1, mmmu-pro-v1; bfcl-v3-v1 0.2.0 -> 0.3.0)

No code changes needed: the runtime-on-agent config addressing and Reward-record
metrics landed with the earlier bump (#3137).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* fix: plot overview time panels on relative wall time (#3148)

* fix(monitor): plot overview time panels on relative wall time

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore: drop view_signature docstring

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Matej Sirovatka <54212263+S1ro1@users.noreply.github.com>

* feat: default rollout transport to zmq and inject host for multi-node (#3149)

* feat(config): shared [rollout_transport] propagated to trainer + orchestrator

Add rollout_transport as a shared RLConfig field (mirroring weight_broadcast /
ckpt / wandb), propagated to trainer.rollout_transport and
orchestrator.rollout_transport via propagate_shared_fields. A single
[rollout_transport] block configures both sub-configs.

auto_setup_rollout_transport (like auto_setup_weight_broadcast) resolves the
shared field from the sub-configs so the launcher's ZMQ host-injection gate is
correct whether transport is set via the shared block or directly on
trainer/orchestrator.rollout_transport (the documented fallback), and rejects
mismatched sub-config transports.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(launcher): inject rollout_transport.host for multi-node zmq transport

ZMQ rollout transport binds/connects a single rollout_transport.host (default
localhost). On multi-node trainer deployments the training-batch PUSH/PULL and
micro-batch PUB/SUB span nodes, so data ranks on non-head nodes connect to their
own localhost and hang at step 0. The launcher already injects
weight_broadcast.host=$MASTER_ADDR for nccl but had no equivalent for
rollout_transport.

When config.rollout_transport resolves to zmq, inject --rollout_transport.host
$MASTER_ADDR into the trainer torchrun and the orchestrator (mirrors the existing
weight_broadcast.host injection). Filesystem transport is unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(config): default rollout transport to zmq

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: single global router in front of engines for all deployments (#3105)

* chore: single global router in front of engines for all deployments

Every deployment shape now runs one client-facing router on server.port
with engines behind it on backend_port: local single-node runs spawn a
vllm-router in the inference entrypoint, and the SLURM templates collapse
per-replica routers into a single global router on inference node 0.
router/backend_port move to InferenceConfig top level; router = "None"
runs a bare engine (used for per-rank SLURM engine processes).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: finish single-node router integration

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: S1ro1 <matej.sirovatka@gmail.com>
Co-authored-by: Matej Sirovatka <54212263+S1ro1@users.noreply.github.com>

* feat(monitor): effective reward/score in wandb overview (#3153)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* fix: compute trainable rate over effective rollouts (#3152)

* fix(orchestrator): compute trainable rate over effective rollouts

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore: drop comment

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* chore: bump renderers to 0.1.9.dev9 (#3158)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* fix: read overview quality metrics from effective subset (#3159)

* fix: read overview quality metrics from effective subset

The all subset counts errored rollouts as zeros for num_branches,
num_turns, num_total_tokens, and is_truncated, which skews the
distributions and makes the panels uninterpretable. Point the wandb
overview panels and the eval success line at the effective subset;
has_error stays on all since effective drops errors by construction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: reword zero-skew rationale, drop subset guidance from skill

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* feat(orchestrator)!: rename env collections to sources, compose the verifiers config blocks (#3150)

* chore(configs)!: remove every config shim (#3156)

* feat(orchestrator): hierarchical GRPO for proposer-solver envs (#3138)

* chore: adapt to the verifiers trace record cleanups (#3157)

* rename training environments to sources

* fix CPU test config fixtures

* rename evaluation environments to sources

* chore(orchestrator)!: rename max_inflight_rollouts to max_inflight_episodes

A dispatcher permit has always bought one episode, not one rollout: for a v1
env one `run` request is one episode (however many agents play it), and a
legacy group-scoring call reserves one permit per bridged v0 rollout, each of
which is its own single-agent episode. Name the knob what it counts.

`max_inflight_rollouts` still parses as an alias, as `rollouts_per_example`
does for `group_size`.

* feat(orchestrator)!: compose the verifiers env, serve and legacy blocks

verifiers split `EnvServerConfig` into three blocks, so an env entry composes
them instead of inheriting one class:

    [[orchestrator.train.source]]
    env    = { taskset = { id = "..." } }   # what runs
    serve  = { pool = { ... }, address, max_concurrent }   # how it's hosted
    legacy = { id, args }                   # a classic v0 env instead

`serve.pool` sizes the spawned server (was `pool`), `serve.address` points at an
external one (was `address`), and `serve.max_concurrent` bounds one worker's
episodes in flight — usually left unset, since `max_inflight_episodes` is the
run's bound. Each moved key raises with a pointer home, including the
`num_workers` shorthand the old validator migrated silently.

BREAKING: `pool` / `address` / `num_workers` on an env entry move under `serve`,
and a v0 env's `id` / `args` / `extra_env_kwargs` move under `legacy`.

TEMPORARY: `deps/verifiers` points at the `feat/one-agent-per-episode` branch
(PrimeIntellect-ai/verifiers#2157) so the two sides can be run together; re-pin
to a merged `main` commit before this lands.

* fix the source-config readers the compose commit missed

The Prime monitor still built its run-registration payload from `env.id`, which
the composed entry doesn't have — `AttributeError` at registration whenever
Prime monitoring is on. It reads `env_id` now, the property that spans both the
v1 and the v0 shape.

The packer's fixture still used the bare `id` shorthand, so its three runs were
rejected at config parse and skipped by `discover_runs`. Full CPU unit suite:
416 passed.

Re-pins `deps/verifiers` to the branch tip of verifiers#2157, which fixes the
same class of stale access on that side (`config.legacy.id`). Still a branch
commit — re-pin to a merged `main` before this lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* spell the env annotation out instead of vf.EnvField

verifiers dropped the `EnvField` alias, which was only ever
`SerializeAsAny[EnvConfig]` under a name that collided with `env_field()`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* declare the env default with the factory verifiers exports

`vf.env_field()` is gone: it only wrapped `Field(default_factory=...)`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* inline the legacy helpers, use the config class as the default

Follows verifiers dropping the three `configs/legacy.py` helpers and the
`single_agent_env_config` factory: `is_legacy` and `env_id` are one-liners here,
the mixed-run refusal is spelled out in `validate_env`, and the `env` default is
`vf.SingleAgentEnvConfig()`. `env.max_concurrent` now works as an alias for
`env.max_concurrent_agents`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* drop the serving-key pointers here too

The same call as verifiers a0ebc408: `num_workers` / `pool` / `address` on a source
entry now fail as an ordinary `extra_forbidden` instead of naming their new home.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* bump verifiers: no max_concurrent alias on the env block

`env.max_concurrent` used to bound a served worker's agent runs across episodes, so
accepting it as `max_concurrent_agents` multiplied it by the episodes in flight. It
errors now, like every other moved key. 416 unit tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* bump verifiers to 0.2.2.dev43 (merged main)

deps/verifiers d0bb0ffe5 -> fcb48822e, which is the squash-merge of the composed
env/serve/legacy config blocks (#2157) plus everything else since the last bump:
Harbor 0.20.0 (#2161), runtime-blamed box failures (#2156), standalone MCP clients
pinned to v1 (#2155), and the v1 test/E2E split (#2127).

Lock: harbor 0.14.0 -> 0.20.0, which drops claude-agent-sdk and datasets from its
own deps and adds filelock/platformdirs, so those move with it.

Replaces the temporary branch pin this PR carried.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: spawn the env server again, and let a pip-only install resolve verifiers

Two CI failures from the compose commit.

The orchestrator stopped spawning env servers. It decides spawn-vs-connect by
`address is None`, and prime-rl's own field defaulted to None; `vf.ServingConfig`
defaults to `tcp://127.0.0.1:5000`, so every env looked external and the run waited
120s for a server nobody started. It now asks whether the address was *pinned*
(`model_fields_set`), so an unset `[serve]` spawns on a free port as before, and
pinning any address — including verifiers' own default — connects.

The slim `prime-rl-configs` wheel is installed from /tmp with no workspace context,
so `verifiers` resolves from PyPI, where `>=0.2.1` picks stable 0.2.1 — no
`SingleAgentEnvConfig`. Floor is `>=0.2.2.dev43` now, a pre-release specifier so the
dev release is eligible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: adapt to the verifiers trace record cleanups

Companion to PrimeIntellect-ai/verifiers#2172: record_* mutators, required
agent seat and RunInfo.id, last_error, and agent.name/.trainable reads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: re-pin verifiers to merged main (f646beb37)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: hallerite <git@hallerite.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore(inference): bump vLLM to 0.26.0 (#3166)

* chore(configs): expand inline TOML tables to sections (#3167)

Every checked-in config, example, and doc snippet now uses [section]
headers instead of key = { ... } inline tables. Purely stylistic: all
48 rewritten TOML files parse identically to their previous versions.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: minh hoang <13672394+eexwhyzee@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: hallerite <git@hallerite.com>
Co-authored-by: Matej Sirovatka <54212263+S1ro1@users.noreply.github.com>
Co-authored-by: rasdani <73563550+rasdani@users.noreply.github.com>
Co-authored-by: Mika Senghaas <mail@mikasenghaas.de>
Co-authored-by: samsja <55492238+samsja@users.noreply.github.com>
Co-authored-by: Hubert <163992334+hubert-marek@users.noreply.github.com>
Co-authored-by: eligotts <78387377+eligotts@users.noreply.github.com>
Co-authored-by: eligotts <eligotts@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: JannikSt <JannikSt@users.noreply.github.com>
Co-authored-by: S1ro1 <matej.sirovatka@gmail.com>
Co-authored-by: Jonas Lossin <jonas.lossin@hai-login1.haicore.berlin>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants