Repository navigation
Standardized rLLM Message Format: Analysis & Proposal #492
Closed
listar2000
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Standardized rLLM Message Format: Analysis & Proposal
1. Current State: What "messages" Are in rLLM Today
Every component in rLLM operates on
list[dict[str, Any]]— there is no formal type. The de facto standard is OpenAI Chat Completions format, enforced by convention rather than code.1.1 Complete Field Inventory
Surveyed across all engines, agents, workflows, parsers, and SDK integrations:
rolestr(system/user/assistant/tool)contentstr | list[ContentBlock] | Nonereasoningstr | Nonetool_callslist[dict] | Nonetool_call_idstr | Nonenamestr | Noneimageslist[PIL.Image] | None1.2 Message Lifecycle
1.3 Existing Conversion Helpers (scattered)
_convert_openai_messagesopenai_messages_to_tinkeropenai_tools_to_tinkerTinkerChatTemplateParser._convert_messageTinkerChatTemplateParser._convert_tools_strands_messages_to_openai_langchain_messages_to_openai_responses_input_to_messages_smolagents_messages_to_openaimsg_to_dict_parse_openai_tool_callsparse_responsebuild_tool_message2. Format Comparison
2.1 rLLM (current) vs OpenAI vs Tinker vs Verl
list[dict](untyped)MessageTypedDictlist[dict](untyped)str | Nonestr | list[ContentPart]str | list[ContentPart]str | list[ContentPart]reasoningfieldThinkingPartinside content list{id, type, function: {name, arguments: str}}ToolCall(function=FunctionBody(...))ToolCall(name, arguments: dict)stron tool msgsstrstrstrimagesfieldimage_urlin content blocksImagePartin contentimagesfieldtrainable: bool2.2 Key Divergences
1.
reasoningrepresentationreasoning: strfield alongsidecontent— simple, explicit, training-friendlyThinkingPartembedded incontentlist — structured but requires content iteration2.
tool_calls.argumentstypeToolCall:arguments: dict(already parsed)arguments: str(JSON string)dict → strbefore adding to messages, parsers deserializestr → dictwhen extracting. Two conversions per tool call round-trip.3.
imagesfieldimages: list[PIL.Image]as a top-level message field (non-standard){"type": "image_url", "image_url": {"url": "data:..."}}content blocksImagePartin structured content list3. Deepcopy / Mutability Analysis
3.1 Why Deepcopy Exists (13 call sites found)
The primary pattern is step snapshotting:
Each Step captures the conversation history up to that point. Without deepcopy, all Steps would share the same growing list.
3.2 Mutation Patterns
messages.append(dict)messages.extend(list)messages + [dict](concat)msg["content"] = ...chat_completionsmessages.pop(0)3.3 Sharing Structure
A persistent/structural-sharing list could eliminate O(n*m) copying (n steps, m messages per step) and reduce it to O(n) (one node per append).
3.4 Performance Perspective
4. Answers to Feature Request Questions
Q1: What exactly are "messages" in practice?
Answer: They are OpenAI Chat Completions-format dicts with two rLLM-specific extensions:
reasoning: str— separate field for thinking/reasoning content (not in OpenAI spec)images: list[PIL.Image]— top-level field for image data (OpenAI uses content blocks)All five SDK integrations convert their native format TO this OpenAI-plus-extensions format. All three local engines (Tinker, Verl, OpenAI) accept it. The gateway captures it as-is.
Q2: Can we have a thin typing layer?
Yes — and it should be a
TypedDict(not a dataclass or Pydantic model) to preserve backward compatibility withdictoperations. ATypedDictgives:isinstancechecking (Python 3.12+, or viatyping_extensions){"role": "user", "content": "hi"}already satisfies the typeQ3: Can we make messages append-only / immutable?
Yes, with a
MessageListwrapper that:append()and__getitem__(read)However, two mutation patterns need accommodation:
msg["content"] = stripped_content(thinking tag stripping) — happens ONLY inchat_completionsproperty, on a copy. The underlyingself.messagesis never mutated this way.messages.pop(0)in one parser — should be refactored to slice instead.So the core
MessageListcan be append-only; thechat_completionsoutput can remain a plainlist[dict](copy).Q4: Does Verl 0.7.1 have a better message spec?
No. Verl still uses
list[dict]with no formal type. It relies on HuggingFace'sapply_chat_template()which acceptslist[dict]withroleandcontent. For multi-modal, Verl'sRLHFDataset._build_messages()constructs structured content blocks ({"type": "image", "image": PIL.Image}) inline — similar to OpenAI's format but with PIL images instead of URLs.Verl's multi-modal pipeline:
The image data flows alongside messages, not inside them. rLLM already follows this pattern.
5. Proposed Design
5.1 Design Goals
dictliterals must work unchanged5.2 Type Definitions
Key decisions:
tool_calls.argumentsisstr(JSON), matching OpenAI wire format. rLLM'sToolCall(arguments: dict)remains as the parsed representation; the message format stores the wire representation. This eliminates the serialize/deserialize friction.reasoningstays as a separate field, not embedded in content blocks. This is rLLM's one deliberate divergence from OpenAI, motivated by training: reasoning tokens need separate loss masking.imagesstays as a top-level field onUserMessage. This is pragmatic: PIL images can't be JSON-serialized, so they can't go in content blocks. They flow separately through the processor pipeline anyway.5.3 MessageList: Append-Only History with Snapshots
Usage in agents:
The snapshot shares memory with the live list.
to_list()does a shallow copy (O(n) but no dict copying) only when serialization is needed.5.4 Conversion Utilities
5.5 Migration Strategy
Phase 1: Types only (non-breaking)
rllm/messages.pywith TypedDict definitionsrllm/__init__.pyRolloutEngine._get_model_response(messages: Messages)Step.chat_completions: MessagesPhase 2: MessageList in agents (opt-in)
MessageListclassToolAgent.messagesfromlist[dict]toMessageListdeepcopy(self.messages)withself.messages.snapshot().to_list()Phase 3: Enforce at boundaries (optional)
_get_model_responseentry point (debug mode only)5.6 What NOT To Do
msg["key"]access everywherelist + [msg]patterns shouldn't be forced to changeimagesfield: PIL images can't live in content blocks; the separate field is pragmaticToolCall(arguments: dict): the rLLM ToolCall dataclass is the parsed form, intentionally different from the wire format. Both are valid at their layer.trainableflag to messages: this is a Tinker-specific concept; rLLM handles loss masking viaresponse_maskin the transform pipeline6. Open Questions for Discussion
Should
reasoningbe a first-class field onAssistantMessage, or should we follow OpenAI and keep it out of messages entirely? Current rLLM code puts it in messages for convenience (workflows construct{"role": "assistant", "content": ..., "reasoning": ...}), but it's not standard.Should
MessageSnapshotstore references to dicts (shallow) or frozen copies (deep)? Shallow is faster but allows mutation of individual message dicts. Deep is safer but slower. The thinking-tag-stripping pattern mutates msg content, but only on copies — so shallow should be fine.Should
MessageListbe the type ofStep.chat_completions? This would be a deeper change.chat_completionsis serialized to JSON for logging/storage, so it may need to remainlist[dict]. The snapshot'sto_list()bridges this.How aggressive should Phase 3 validation be? Options range from "debug-only asserts" to "always validate, raise on unknown fields". The latter could break users who add custom fields to messages.
All reactions