Repository navigation
[Agentflow Refactor] How agent flows, sandboxes, tasks, and evaluators fit together #639
listar2000
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
This is a knowledge-base walkthrough of the system as it stands after the three stacked PRs (#636 → #637 → #638, all targeting
terminal-rl). It is written for someone who knows rLLM well but hasn't followed the design discussions — it describes what the system is, the contracts each piece signs up to, and what to look at when reviewing each PR. It is not a changelog.Why we touched this at all (short version)
Three pressures, one theme: the question "does this rollout need a sandbox, and who is responsible for it?" was answered implicitly, in several places, with type checks. That made some legitimate combinations impossible (a plain
@rllm.rolloutfunction over a dataset graded by an in-sandbox script), made others fragile (sandbox lifecycle split between the flow and the hook), and left the biggest per-rollout cost (installing a CLI agent inside a fresh sandbox, up to ~10 minutes) outside the snapshot system that exists to amortize exactly that kind of cost. The theme of the fix: each piece declares what it needs; one place combines the declarations; one owner manages each resource.The life of one rollout
Everything below hangs off this sequence, so it's worth having in your head first. For each task in a step:
rllm/engine/agentflow_engine.py) asks its hooks to set the task up. The standard hooks areSandboxTaskHooksinrllm/hooks.py.AgentConfig(gateway session URL, model, sampling params) and calls the flow:run(task, config)for host-only flows,run(task, config, *, env=sandbox)for flows that need one.None— the engine reconstructs the trajectory from traces (strictly during training, where token IDs are required for loss math; leniently during validation).The four pieces and their contracts
Tasks declare what they need
A task is data (
rllm/types.py: Task). If it needs an execution environment, it says so on disk or in metadata: anenvironment/directory (with a Dockerfile and resource limits) next to the task or dataset, or harbor-styletask_pathmetadata. If it knows how it should be graded, it says so in its[verifier]config. Tasks never run code on the host and never decide policy — they are declarations the rest of the system reads.Flows are programs, not resource owners
A flow is the thing that acts: a
@rllm.rollout-decorated function, aSandboxedAgentFlowsubclass (rllm/sandbox/sandboxed_flow.py), or aBaseCliHarnesssubclass (rllm/harnesses/cli_harness.py). The contract has three parts:needs_env(does my program require a sandbox —TrueforSandboxedAgentFlow,Falsefor decorated functions) andllm_inside_env(does my LLM client process run inside the sandbox —Trueonly for CLI harnesses, where the agent binary itself calls the gateway from within the container).def run(self, task, config, *, env). There is noself.sandbox. This is what lets one flow instance serve many parallel rollouts safely, and it's checked once when the engine binds the flow (flow_accepts_envinrllm/types.py) — aneeds_envflow without the parameter fails at construction with the exact signature to add, not mid-rollout.configure(overrides) -> leftovers. The CLI's--sandbox-backend/--sandbox-concurrencyflags go through it; whatever the flow returns unconsumed, the wiring layer warns about. A flag the agent can't honor is visible, not a silent no-op.HarborRuntime(which provisions its own environments) mapssandbox_backendonto itsenvironment_typethrough the same method.A CLI harness additionally describes itself through four methods the lifecycle machinery consumes:
install_script()(how to install the CLI),build_env()(env vars),write_configs()(in-sandbox config files),build_invocation()(the command line). The base class'srun()is just the fixed sequence over those four; subclasses describe, they don't orchestrate.Sandboxes have exactly one owner
SandboxTaskHookscreates the sandbox, prepares it, hands it to the flow for the duration of one rollout, and closes it. Flows and evaluators borrow it; nobody else closes it. The supporting cast:rllm/sandbox/snapshot.py): a snapshot's name is a content hash (env_key) of everything that went into building it — backend, base image, the Dockerfile's RUN steps, and (since feat(sandbox): bake CLI installs into snapshots (AgentFlow phase 2) #637) the harness's install script. Identical inputs share a snapshot across datasets; any change is a clean miss, never a stale hit.rllm snapshot create <dataset> --agent <name>(rllm/cli/snapshot_cmd.py) builds the CLI install into the image, so rollouts boot with the agent already installed.get_sandboxrecordssandbox.baked_install— the install script the image actually contains. The hook runs the install only when that doesn't exactly match the current harness's script. No "it's probably installed" heuristics: the scripts' own already-installed checks depend on PATH entries the installer creates, so re-running them is not a safe fallback.rllm/sandbox/warm_queue.py) prefetches sandboxes a few tasks ahead of the training schedule. It is orthogonal to all of the above — it just callsget_sandboxearly.Evaluators follow one calling convention
The hook resolves one evaluator per task through an evaluation policy (
rllm/hooks.py):FixedEvaluation(evaluator)means one host-side evaluator scores every task (what a passedevaluator=becomes);FromTaskEvaluation()means each task's[verifier]config decides (registered reward functions, Python modules, bare callables, or in-sandbox kinds likesandbox-shell, which runs the task'stests/test.shinside the sandbox and reads a reward file). Resolution lives inrllm/eval/_resolution.py.Every evaluator receives the real
Taskobject. The dict-style convention still works, but only when explicitly requested — adictannotation or a legacy parameter name. Hybrid Python verifiers that want the sandbox declare asandboxparameter and get it as an attribute onPythonModuleEvaluator(rllm/eval/module_evaluator.py); no signature sniffing at call time.The one rule that ties it together
"Does this rollout need a sandbox?" is computed in exactly one place,
resolve_rollout_planinrllm/hooks.py, as a three-way OR of declarations:with one escape hatch: if a task ships an environment that neither the flow nor the verifier can actually use, we skip provisioning and warn once (no sandboxes built for nobody). The same predicate, evaluated over all dataset rows before the run starts (
scan_env_requirements), drives the wiring decisions: whether to install sandbox hooks at all, and whether the gateway needs a public tunnel (yes, iff any task or flow resolves to a remote backend).The gateway URL each flow sees follows from
llm_inside_env: host-side flows always get the local URL (they must never depend on a tunnel hostname, which can rotate mid-run on free tunnels); in-sandbox CLIs get the public tunnel URL on remote backends, or a loopback rewritten tohost.docker.internalon local Docker (container_reachable_urlinrllm/gateway/manager.py, keyed on the backend that actually provisioned this task's sandbox — not a class attribute that could disagree with it). The tunnel itself blocks until the hostname actually answers over HTTP (rllm/gateway/tunnel.py), because cloudflared printing a URL only means it was assigned.The philosophy, compressed
needs_env,llm_inside_env,install_script()), and consumers read declarations instead of switching on types.Reviewing the three PRs
#636 (phases 0–1: the join, always-hooks, two bug fixes). The heart is
rllm/hooks.py(the join, the scan, the two evaluation policies) and the engine's collapse to a single execution path. Worth your scrutiny: (a) the join's truth table against the combinations you originally designed for — is there a legitimate setup where the three-way OR or the no-consumer downgrade gives the wrong answer? (b) the strict-enrichment change — training now hard-fails on traces with empty token IDs and retries, validation stays lenient; isstrict = not is_validationthe right line for every backend you know of? (c) thellm_inside_envURL split — convince yourself host-side loops really never need the tunnel. New tests intests/eval/test_env_join.pyencode the intended truth table; reviewing those is reviewing the design.#637 (phase 2: install baking). Small diff, one invariant to verify: a snapshot key can only ever match an image built from exactly the inputs hashed into it. Check the key derivation in
rllm/sandbox/snapshot.py(empty install must contribute nothing — that's the no-migration guarantee), the build paths inrllm/sandbox/backends/daytona.pyandmodal_backend.py(does the image really contain what the key claims, including the base64 single-line wrapping for Daytona's declarative builds?), and thebaked_installexact-match skip. The one judgment call to weigh in on: one task used by two harnesses produces two snapshots (double build time and storage) — we accepted that as the correct identity rather than sharing a base.#638 (phase 3: stateless flows). This is the PR closest to the original AgentFlow design, so your read matters most here. The claim to test: no flow needs per-rollout mutable state. We audited every harness for
self-mutation duringrunand found none, but you know the historical reasonsself.sandboxexisted — if any of them are still live (a subclass in someone's fork, a pattern we missed), this is where to say so. Also worth attention: the*, envkeyword convention and its bind-time check inrllm/types.py(too strict? not strict enough forarun-only flows?), the install moving intoSandboxTaskHooks.setupincluding the close-on-failure path, and whetherconfigure(overrides) -> leftoverscarries enough information for the flags your agents care about. The harness diffs themselves are mechanical (sandbox threading throughwrite_configs/_exec_agent); the base-class contract inrllm/harnesses/cli_harness.pyis the part to read carefully.What we deliberately did not build
So reviews don't relitigate them: no shipping of user Python into containers (host flows stay on the host, CLI harnesses stay in the sandbox; violations error at bind time); no step-by-step action/observation protocol between host and sandbox (flows either hold a handle or live inside the env — the remote runtime remains its own engine); no provisioner class stacks or new spec dataclasses (snapshot-or-cold is one function, the warm queue is one class, and the install layer is one string argument to the existing hash).
Where the evidence lives
Each PR was verified end-to-end on Tinker + Daytona with snapshots and the warm pool on, across the three canonical shapes: a
@rllm.rolloutmath solver with a host evaluator (no sandbox anywhere), aBashHarnesshost loop against Daytona sandboxes, and a mini-swe-agent CLI harness graded by an in-sandboxtest.sh. The strongest single piece of evidence for #637/#638: a dataset whose Dockerfile contains no CLI at all (tmp/terminal_rl/agentflow_redesign/case4_cli_*) trains successfully from a--agent-baked snapshot, with rewards verified per episode and zero leaked sandboxes after every run. Run logs and the datasets are undertmp/terminal_rl/agentflow_redesign/(untracked); unit coverage lives intests/eval/test_env_join.py,tests/sandbox/test_snapshot.py,tests/sandbox/test_sandboxed_flow.py,tests/engine/test_agentflow_engine.py, andtests/harnesses/test_cli_harness.py.All reactions