Training Gym

Modal Training Gym is an open-source Python SDK for RL post-training on Modal.

The library takes care of infrastructure concerns such as cluster topology, Ray/NCCL bring-up, volume mounts, checkpointing, and serving for eval and rollouts, so you (or your agent!) can focus on training models.

To get started, you can go through the Quickstart section below, or paste the prompt below into your favorite agent:

Install the `training-gym` library. Then, install the skill bundle into the local project with `training-gym skills install`. Finally, tell the user that they can ask you to do something like: "Given a budget of 1 B300, train Qwen3.8 27B to correctly solve problems taken from agentica-org/DeepCoder-Preview-Dataset with xhigh reasoning and a length penalty to limit the number of reasoning tokens."

Quickstart

Requirements:

• Python 3.12

Install the package directly:

uv pip install -q git+https://github.com/modal-projects/training-gym.git@main

Or pin it in pyproject.toml:

training-gym = { git = "https://github.com/modal-projects/training-gym.git", branch = "main" }

Authenticate with Modal:

modal setup

Set up the dashboard:

training-gym setup

And empower your agents with the Gym's skill bundle:

training-gym skills install

Then, it's as easy as:

from modal_training_gym import (
    HuggingFaceDataset,
    Qwen3_4B,
    Qwen3_4B_Recipe,
    TrainConfig,
)


class MathDataset(HuggingFaceDataset):
    hf_repo = "zhuzilin/dapo-math-17k"
    input_key = "prompt"
    label_key = "label"
    output_format = "jsonl"
    apply_chat_template = True


config = TrainConfig(
    model=Qwen3_4B(),
    dataset=MathDataset(n_rows=120),
    recipe=Qwen3_4B_Recipe(
        gpu_type="H100",
        actor_num_nodes=1,
        actor_num_gpus_per_node=8,
        tensor_model_parallel_size=1,
        sequence_parallel=False,
        rollout_num_gpus=8,
        rollout_num_gpus_per_engine=1,
        colocate=True,
        num_rollout=1,
        n_samples_per_prompt=4,
        rollout_batch_size=8,
        rollout_max_response_len=2048,
        max_tokens_per_gpu=4096,
        sglang_mem_fraction_static=0.6,
        rm_type="deepscaler",
    ),
)
run = config.launch()
print(run.training_run_id)

Supported models

Family	Models
GLM	GLM-4.7
Gemma	gemma-4-26B-A4B-it
Inkling	Inkling-Small
Moonlight	Moonlight-16B-A3B-Instruct
Qwen3	Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3-ASR-1.7B Qwen3-VL-8B-Instruct
Qwen3.5	Qwen3.5-0.8B Qwen3.5-2B Qwen3.5-4B Qwen3.5-9B
Qwen3.6	Qwen3.6-27B Qwen3.6-35B-A3B
Qwen3.8	Qwen3.8-27B

Multi-node access

Important

Single-node training is open to everyone. Multi-node clusters are still in Beta and are required for larger models. Contact us on Slack for access.