Repository navigation
Toward a dependable rollout record in rLLM #856
listar2000
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Summary
rLLM currently carries an agent rollout through several representations: gateway traces, messages,
ModelOutput,Step,Trajectory,Episode, persisted JSON, and finally backend-specific training inputs. These representations grew at different times and for different consumers. They overlap, but none is the unambiguous source of truth.That ambiguity is now causing correctness and maintenance problems. The same model response can be stored under several field names; parsed tool calls lose information retained by the raw message; declared fields disappear during an
Episode.to_dict()round-trip; and training and evaluation write different JSON shapes for the same runtime object. Newer data such as routing matrices, weight versions, and subagent lineage has been added to the same structures without clarifying which layer owns it.We propose a bounded prototype: introduce a versioned, runtime-validated rollout record for gateway-mediated AgentFlow RL, while continuing to project into today's
Step/Trajectory/Episodeobjects for the existing Tinker, Fireworks, and verl trainers. The prototype would support one linear trajectory and lineage, text and function-tool interactions, and exact token evidence. It would explicitly reject or defer cases whose semantics are not yet reliable.The goal is not to declare the final rLLM format. It is to create one end-to-end seam where we can test whether clearer ownership and lossless persistence actually make the system easier to reason about.
How rollout data moves today
For the AgentFlow RL path, the simplified flow is:
flowchart LR A[AgentFlow program] --> G[Model gateway TraceRecord] G --> C[trace_record_to_step] C --> S[Step] S --> E[Trajectory and Episode] E --> T[Tinker / Fireworks transform] E --> V[verl transform] E --> L1[training episode logger] E --> L2[eval episode store]The model gateway captures one inference attempt as a
TraceRecord.trace_record_to_stepfilters attempts without a model response and turns successful calls into the sharedSteptype. AgentFlow then adds agent-side information such as action and reward, and trainers derive token sequences and loss inputs from the resulting trajectories.This is a useful shared boundary: all three RL backends already consume the same
Stepdata. It is also where several distinct concepts have accumulated in one object.What is difficult in the current model
1.
Stephas several overlapping ownersToday one
Stepmixes at least four kinds of state:chat_completions,thought,model_response,input,outputprompt_ids,response_ids,logprobs,routing_matrices,weight_version,model_outputaction,observation,reward,donemc_return,advantageThe distinction matters because these values have different lifetimes. A model response and its sampled token IDs are evidence from inference time. An environment reward is attached later. Advantages are derived later still and may change with the training algorithm. Treating all of them as peer fields makes it easy to overwrite evidence with a derived view or persist an incomplete intermediate state without realizing it.
2. One response can have several representations
trace_record_to_stepcurrently keeps the gateway response message, builds aModelOutput, appends the response tochat_completions, and repeats visible output and reasoning in severalStepfields. For one assistant call, equivalent data can therefore appear as:Those copies are not guaranteed to stay equal. The current runtime also encounters both
reasoningandreasoning_content. Tool calls are even more revealing: the raw gateway message can retain a tool-call ID and the exact JSON argument string, while conversion to rLLM's parsedToolCallkeeps only the name and parsed arguments. Both views remain reachable, but they no longer contain the same information.Shape differences also affect live behavior. The gateway's cumulative-token path compares message fingerprints; semantically equivalent histories can fail that comparison when one occurrence uses
reasoning, another usesreasoning_content, or a provider adds envelope-only fields. Normalizing only when an episode is saved would make the log cleaner without fixing this earlier comparison boundary.This is not solved by adding type annotations to every dictionary. We first need to decide which representation owns the evidence and which representations are derived compatibility views.
3. Persistence is not one contract
The current classes declare fields that their compatibility serializers do not preserve. On the current branch,
Episode.to_dict()followed byEpisode.from_dict()loses or changes, among other fields:Step.id,Step.input, andStep.output;Trajectory.input,Trajectory.output, andTrajectory.signals;Episode.session_idandEpisode.artifacts;Nonetermination reason, which cannot be reloaded through this path.Meanwhile, evaluation persistence uses
episode.model_dump(mode="json"), and the training episode logger writes a hand-selected subset. These are three different serialized views of the same runtime concepts, with no schema version that tells a reader which contract it received.This makes saved episodes weaker as debugging, curation, or reproducibility artifacts. A clean load is not enough if the reader cannot know what was omitted.
4. Recent features make implicit semantics more costly
These structures were already broad; newer requirements have made their ambiguity more consequential:
weight_version;lineage_id, but current local enrichment still pairs agent-side steps and gateway traces positionally;TraceRecorddoes not currently retain a normalized copy unless the much larger raw request is captured.A future format should not claim information that its source path cannot provide. For example, count equality does not prove that two interleaved lineages were paired correctly, and a remote rollout should not fabricate per-step observations that were never returned.
Properties we want from a better boundary
The following are design goals, not commitments to a particular Python class hierarchy.
Stepfields can remain during the experiment, but they should be projections with parity tests and a clear rollback boundary.Prototype direction: a versioned AgentFlow rollout record
As a working name, consider
AgentFlowRolloutV1. The exact names and implementation library are open for discussion; the important part is the ownership boundary.The persisted models would be strict at runtime, use a schema-version dispatcher, and reserve an explicit JSON-safe extension namespace rather than accepting arbitrary undeclared fields. Inference evidence should be write-once after validation; compatibility objects should receive defensive projections so mutating a legacy
Stepcannot change the stored record.Canonical messages inside the record
The prototype needs a small role-discriminated message model:
reasoning_content, and zero or more function tool calls;tool_call_id;arguments_json, preserving the exact string when the source supplied one and using deterministic JSON encoding when the source supplied an object; parsing remains a helper rather than replacing the source text.Ingress adapters may accept current aliases such as
reasoning, but emit one canonical shape. The same normalization should run before the gateway fingerprints cumulative histories and when it constructs response envelopes; otherwise the persisted record and live token path would still disagree. Image, audio, and unknown content parts would not be admitted by the first prototype; existing non-prototype paths would keep their current behavior.Tool definitions also belong to the inference-time record. Rather than enable full raw-request capture, the gateway could add a small normalized
TraceRecord.tool_definitionsfield containing only the offered function type, name, description, parameters, and strictness. This would preserve the model's available action space without retaining headers or unrelated provider payloads.Local and remote records should be honest about their differences
The gateway call evidence is available to both local and remote AgentFlow. Per-step environment evidence is not. The prototype would therefore populate
transitionfor local rollouts and leave it absent for remote rollouts, while keeping remote trajectory-level reward and episode outcome at their actual scope.Likewise, advantages, loss masks, and other trainer-derived values would stay outside this first persisted record. They can be represented by a separate versioned trainer artifact if exact post-advantage replay becomes a requirement.
Proposed experimental scope
Included
Episode,Trajectory, andStepobjects;Deliberately deferred
Workflow/UnifiedWorkflowEnginerollouts;These exclusions do not imply that the cases are unimportant. They identify places where the current code does not yet provide enough reliable provenance for a prototype to make a strong promise. Multi-lineage support, for example, should follow a stable agent-step-to-trace correlation key rather than preserve today's positional assumption. SFT is deferred for a different reason: it already has a validated row schema and trainability semantics, so merging it into a rollout prototype would conflate two distinct contracts.
How the experiment could be introduced
One possible sequence is:
trace_record_to_stepoutput. Production consumers remain unchanged.This sequence is intentionally reversible. It lets us test the schema against real backend behavior without first rewriting trainers, SFT, direct workflows, or every logger.
Evidence the prototype should produce
The prototype should be judged on exactness rather than the elegance of its type definitions.
Nonetermination all have explicit behavior.If these conditions hold, the experiment will have established a trustworthy seam even if we later choose a different final schema.
Questions for discussion
reasoning_contentthe right canonical spelling? It already matches current renderer and SFT paths, while ingress adapters can continue to read legacy aliases during migration.What this proposal is not
This is not a request to replace every rLLM message,
Step, or dataset row in one change. It is not a claim that one external format can represent every RL backend. It is not a performance proposal, and it does not assume that a strict class definition by itself fixes inconsistent data ownership.It is a proposal to make one important path explicit, lossless, and testable, then use what we learn to decide whether the boundary should expand, shrink, or change shape.
All reactions