Function-calling RL reproduction and transfer study

[ReinforcedKnowledge]

Parent: #220

Goal

Reproduce an open verifier-backed agentic RL recipe, establish that it works reliably on the available infrastructure, and then use it as a reference for function-calling post-training research.

This issue tracks the reproduction and transfer methodology at a high level. Exact launch scripts, environment patches, and run manifests will be linked after they are cleaned for release.

Why TMAX first

TMAX is the first RL target because it is a compact and unusually complete open calibration point for online agent training:

• the repository, training changes, environment code, released task data, and checkpoints are public;
• the reference method trains against executable environments with verifier rewards, which directly tests whether the RL stack can learn from tool-mediated outcomes;
• the released Qwen path provides a small reference model for calibration before attempting larger models or cross-family transfer;
• its Open-Instruct-based trainer is close enough to the existing post-training stack to make the reproduction useful beyond a single benchmark.

OpenThoughts-Agent remains valuable, especially as an agentic SFT data and trajectory-curation reference. Its flagship result is primarily a larger SFT recipe, while its RL path uses a different stack and is not the cleanest first test of the immediate question: can the current infrastructure reproduce and transfer verifier-backed online RL? The intended sequence is therefore TMAX for the RL calibration spine, followed later by OpenThoughts-Agent-style trajectory and mixture studies where useful.

RL recipe being reproduced

The reference path starts from a released Qwen checkpoint and uses TMAX's Open-Instruct fork to run online RL in stateful terminal environments.

1. Sample multiple agent rollouts for executable tasks.
2. Let the model interact with the sandbox through parsed tool calls.
3. Score completed trajectories with task verifiers, producing binary outcome rewards.
4. Use active sampling and soft filtering so training focuses on task groups with useful reward variation.
5. Apply DPPO updates with the reference trust-region settings.
6. Synchronize the updated learner weights to the rollout engine and repeat.
7. Evaluate the base and trained checkpoints through the same pinned task and sandbox harness.

The first meaningful target is a small Qwen calibration with nonzero reward variance and a measurable policy update. A bounded RL pilot follows only after that calibration passes. A larger reference reproduction and any OLMo-family transfer come afterward.

Problems encountered and resolutions

Problem	Resolution
The released environments reference many row-specific Docker images, while the target cluster exposes Apptainer rather than a Docker service. Converting every image independently would be impractical.	Built a base-image plus task-delta prepared-state path and validated its reset and runtime semantics against a converted prebuilt-image slice.
A one-GPU learner and rollout engine either exhausted memory or failed the native CUDA weight-transfer path.	Moved the smoke test to the reference-style multi-GPU path. The two-GPU plumbing smoke completed rollout, one DPPO update, checkpoint save, and clean shutdown.
The upstream runtime assumed site-specific cache, temporary, dataset, and rollout paths.	Isolated and pinned the environment, redirected caches and temporary sockets to appropriate project-local locations, added local parquet handling, and used a writable rollout root.
Some task setup scripts assumed privileged filesystem paths, services, package managers, or fixed ports that do not map cleanly onto the target cluster.	Added narrow, versioned compatibility handling, focused repair tests, strict content audits, and explicit process, port, scratch, and cleanup checks.
A tiny successful smoke produced zero reward and therefore proved only plumbing, not learning.	Added a separate nonzero-signal calibration gate requiring reward variation, finite RL tensors, a nonzero update, valid parsed actions, and same-harness base-versus-trained evaluation before calling the method reproduced.

Current recorded status

• Source and release-package auditing is complete.
• A real Qwen3.5-2B TMAX smoke has completed rollout, training, checkpointing, and shutdown on the multi-GPU path. It is correctly classified as plumbing evidence only.
• The prepared-state backend has passed focused parity, reset, runtime, and cleanup tests on increasingly broad task samples.
• The top-200 exact gates passed. The next 300-task diagnostic produced 270 passes and 30 classified environment failures. Focused repairs and the corrected validation contract are staged.
• At the latest recorded checkpoint, the guarded asset build for those repairs was still waiting for resources. No meaningful RL training run had started.
• After the remaining environment and runtime gates pass, the fixed sequence is a nonzero-signal Qwen3.5-2B calibration, a bounded 2B RL pilot, and then a larger Qwen reproduction candidate if the pilot shows a credible positive direction.

Open source implementation

The current code-only reproduction snapshot is public at OpenEuroLLM/tmax-reproduction. It starts from the TMAX Open-Instruct tree at 1cce58a2 and adds the Leonardo prepared-state backend, runtime compatibility changes, and bounded bring-up scripts used so far.

The imported source is tagged tmax-upstream-1cce58a2, and the current Leonardo work-in-progress state is tagged leonardo-prepared-apptainer-wip-20260720. This release contains code and configuration only. Private experiment-control material, harnesses, reports, logs, notes, prepared caches, datasets, and checkpoints are not included.

Deliverables

• Publish the current code-only Open-Instruct and Leonardo prepared-state snapshot.
• Close the remaining prepared-environment and runtime replay gates.
• Complete a nonzero-signal small-model calibration.
• Run and evaluate a bounded small-model RL pilot.
• Reproduce the selected larger TMAX reference result if the pilot passes.
• Test transfer to the function-calling post-training stack and additional model families.
• Publish cleaned reproduction scripts, pinned revisions, and a concise failure and repair log.