Consolidating text, image, and audio teachers into a single omni-modal student
Paper · Models · Method · Results · Quick Start
Omni-modal models provide a unified interface for text, images, and audio, but improving these abilities together is hard: post-training on pooled multimodal data often fails to preserve the strengths of modality specialists.
On-policy distillation (OPD) samples responses from the current student and compares the teacher's and student's next-token distributions along those responses, giving dense supervision while reducing the train/inference mismatch. However, standard OPD does not readily extend to several modality teachers:
- Conflicting guidance — different teachers push the shared backbone in different directions, so a single shared guidance weight cannot regulate the three signals.
- Teacher ceiling — symmetric distribution matching constrains the student even on tokens where it is already better than its teacher.
OPOD addresses both. It routes each student rollout to the matching modality teacher, applies guidance only where the teacher is ahead, controls each modality with its own constraint budget, and reuses the routed teacher as a verifier of the answer and of the reasoning that supports it. Teachers are discarded after post-training — only the student is deployed, with no teacher-ensemble latency or memory.
Student rollouts are routed by input modality to the corresponding teacher trained offline with GRPO. OPOD combines one-sided token guidance, modality-specific constraint control, and teacher-based verification (answer confidence + reasoning gain).
OPOD converts the routed teacher's evaluation of a student rollout into three components, which are optimized jointly with a reference regularizer:
| Component | What it does | Why |
|---|---|---|
| One-Sided Teacher Guidance | Applies the token constraint only where the routed teacher assigns a higher likelihood than the student; stops once the student matches or exceeds it. | Keeps useful corrections without imposing a teacher ceiling. |
| Adaptive Modality Control | Maintains a separate constraint budget |
Lets each teacher's influence move independently instead of sharing one weight. |
| Verification Reward | Reuses the routed teacher as a trajectory verifier: correctness-gated answer confidence plus ungated reasoning gain. | Adds process-level feedback with no step annotations and no tree search. |
Motivating observations — specialists develop complementary strengths that pooled training does not recover, and their parameter-change directions frequently conflict:
Post-trained students from the paper (only the student is retained; teachers are discarded):
| Model | Backbone | Overall Avg. | Gain over base | Download |
|---|---|---|---|---|
| OPOD-Qwen3-Omni-30B-A3B | Qwen3-Omni-30B-A3B-Instruct (MoE) | 70.8 | +3.8 | 🤗 Tung111/OPOD-Qwen3-Omni-30B-A3B |
| OPOD-Qwen2.5-Omni-7B | Qwen2.5-Omni-7B (dense) | 51.7 | +4.8 | 🤗 Tung111/OPOD-Qwen2.5-Omni-7B |
| OPOD-Qwen2.5-Omni-3B | Qwen2.5-Omni-3B (dense) | 46.2 | +5.5 | 🤗 Tung111/OPOD-Qwen2.5-Omni-3B |
Overall Avg. is the mean accuracy over the twelve benchmarks in Results.
All numbers are accuracy (%). Bold = best, underlined = second best within each backbone block; specialist teachers are included in the 30B comparison.
| Model | AIME25 | AIME26 | HQA | MMLU-Pro | GPQA | MMMU | MathV. | ChartQA | A-OKVQA | MMAU | AVQA | OmniBench | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-Omni-30B-A3B | |||||||||||||
| Text teacher | 47.9 | 57.7 | 41.5 | 76.8 | 65.8 | 67.7 | 76.3 | 86.5 | 87.7 | 76.0 | 79.8 | 61.1 | 68.7 |
| Image teacher | 47.9 | 52.1 | 27.3 | 77.3 | 64.8 | 68.2 | 77.3 | 87.8 | 88.6 | 75.7 | 80.8 | 61.5 | 67.4 |
| Audio teacher | 52.1 | 59.2 | 27.3 | 76.9 | 64.0 | 67.4 | 76.4 | 87.2 | 88.9 | 79.1 | 84.4 | 55.6 | 68.2 |
| Base | 49.2 | 52.9 | 26.8 | 77.0 | 65.3 | 67.9 | 76.3 | 85.8 | 87.6 | 75.6 | 79.3 | 60.3 | 67.0 |
| GRPO | 49.2 | 53.8 | 28.0 | 76.7 | 65.3 | 67.9 | 76.7 | 86.2 | 87.9 | 76.2 | 80.5 | 59.7 | 67.3 |
| Native OPD | 51.7 | 55.8 | 41.1 | 77.5 | 64.9 | 68.4 | 76.4 | 73.4 | 88.0 | 75.8 | 81.2 | 60.3 | 67.9 |
| ExOPD | 50.0 | 64.2 | 41.0 | 77.1 | 63.5 | 68.0 | 76.5 | 87.2 | 88.7 | 65.1 | 83.2 | 58.8 | 68.6 |
| OPOD | 55.0 | 63.3 | 41.1 | 78.2 | 68.7 | 68.7 | 77.6 | 87.6 | 89.1 | 76.6 | 82.6 | 61.1 | 70.8 |
| Qwen2.5-Omni-7B | |||||||||||||
| Base | 5.0 | 3.3 | 24.1 | 32.5 | 31.8 | 51.7 | 67.5 | 66.2 | 85.1 | 71.4 | 76.8 | 47.2 | 46.9 |
| GRPO | 10.0 | 3.3 | 22.9 | 38.5 | 32.1 | 51.4 | 67.7 | 67.2 | 86.5 | 73.0 | 80.3 | 44.8 | 48.1 |
| Native OPD | 6.7 | 3.3 | 28.8 | 52.4 | 32.3 | 53.1 | 66.8 | 69.1 | 81.7 | 73.1 | 80.0 | 47.3 | 49.5 |
| ExOPD | 6.7 | 8.3 | 29.5 | 52.4 | 30.8 | 51.4 | 66.4 | 73.8 | 82.6 | 72.0 | 78.4 | 46.3 | 49.9 |
| OPOD | 11.7 | 6.7 | 29.8 | 52.5 | 31.1 | 54.0 | 67.5 | 79.7 | 85.7 | 75.0 | 79.1 | 48.0 | 51.7 |
| Qwen2.5-Omni-3B | |||||||||||||
| Base | 1.7 | 1.7 | 18.1 | 22.8 | 24.5 | 42.8 | 53.8 | 57.3 | 81.9 | 69.9 | 72.4 | 41.2 | 40.7 |
| GRPO | 3.3 | 1.7 | 17.4 | 24.5 | 29.0 | 44.6 | 56.4 | 59.7 | 83.8 | 71.6 | 74.9 | 41.7 | 42.4 |
| Native OPD | 1.7 | 3.3 | 23.5 | 36.4 | 26.5 | 44.9 | 59.4 | 61.2 | 84.3 | 69.3 | 75.4 | 42.6 | 44.0 |
| ExOPD | 1.7 | 3.3 | 24.5 | 37.9 | 30.3 | 43.9 | 59.9 | 62.3 | 83.6 | 69.9 | 76.3 | 40.2 | 44.5 |
| OPOD | 5.0 | 6.7 | 25.6 | 41.5 | 28.5 | 45.8 | 60.5 | 67.5 | 83.7 | 70.8 | 76.9 | 41.5 | 46.2 |
At 30B, OPOD surpasses the base model and pooled GRPO on all 12 benchmarks, beats native OPD on 11, and ExOPD on 10. An oracle that picks the best teacher per benchmark reaches 70.3 — still 0.5 points below the single OPOD student.
OPOD delivers the largest gain at every scale: +5.5 (3B), +4.8 (7B), +3.8 (30B-A3B).
| Variant | AIME25 | AIME26 | HQA | MMLU-Pro | GPQA | MMMU | MathV. | ChartQA | A-OKVQA | MMAU | AVQA | OmniBench | Avg. | Δ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full OPOD | 55.0 | 63.3 | 41.1 | 78.2 | 68.7 | 68.7 | 77.6 | 87.6 | 89.1 | 76.6 | 82.6 | 61.1 | 70.8 | — |
| − One-Sided Guidance | 50.0 | 60.0 | 40.5 | 77.0 | 63.4 | 68.2 | 76.7 | 87.2 | 87.7 | 76.3 | 80.9 | 60.3 | 69.0 | −1.8 |
| − Modality Control | 47.1 | 60.4 | 36.9 | 77.6 | 65.6 | 67.9 | 76.0 | 86.5 | 88.5 | 76.0 | 80.3 | 60.5 | 68.6 | −2.2 |
| − Verification Reward | 48.3 | 60.0 | 40.6 | 77.3 | 65.9 | 66.9 | 76.1 | 86.5 | 87.9 | 75.3 | 80.8 | 59.7 | 68.8 | −2.0 |
| − Reasoning Gain | 52.9 | 63.3 | 38.8 | 77.1 | 65.0 | 67.7 | 76.4 | 85.8 | 88.0 | 76.0 | 80.1 | 59.1 | 69.2 | −1.6 |
| − Answer Confidence | 51.7 | 60.0 | 40.9 | 77.1 | 64.1 | 68.1 | 76.8 | 87.8 | 88.6 | 76.0 | 81.7 | 59.8 | 69.4 | −1.4 |
Every component contributes; removing adaptive modality control costs the most (−2.2). Across the 60 benchmark–variant comparisons, Full OPOD matches or exceeds the corresponding ablation in 59. The two verification terms are complementary: reasoning gain matters more on AVQA and OmniBench, answer confidence more on AIME25/26 and GPQA.
The 10-step warm-up yields per-modality budgets
This repository contains the evaluation harness and the OPOD-specific training code. Raw data, checkpoints, logs, and caches are not included.
opod/
├── eval/ # Evaluation harness (12 multimodal benchmarks)
│ ├── eval.sh # Convert a framework checkpoint, serve it, evaluate
│ ├── eval_hf.sh # Evaluate a local HF checkpoint
│ ├── smoke_test.sh # Quick sanity check
│ ├── runner/run_eval.py # Core evaluation loop
│ ├── adapters/ # Per-benchmark adapters (aime25, mmmu, mmau, omnibench, ...)
│ ├── prompts/ # System prompts (identical to the training prompts)
│ ├── server/ # vLLM serving helpers
│ └── tools/ # Megatron → HF checkpoint conversion
│
└── training/ # OPOD-specific training code (framework NOT bundled)
├── method/ # Self-contained OPOD modules + patch excerpts
│ ├── on_policy_distillation.py # Multi-teacher on-policy distillation rollout
│ ├── opd_process.py # Verification reward (answer confidence + reasoning gain)
│ ├── opd_loss_patch.py # Advantage injection, dual KL control, adaptive entropy
│ ├── opd_args_patch.py # OPOD command-line flags
│ └── tests/ # Unit tests for the reward path
├── integration/ # ⭐ Ready-to-apply patch against Relax @ cbb1a82
└── recipes/ # Teacher / OPOD / GRPO recipes for 3B, 7B, 30B-A3B
OPOD is implemented on top of Relax — an open-source
multimodal on-policy RL framework (Apache-2.0), imported as relax. Relax itself is not vendored
here; training/integration/ ships only OPOD's changes to it, distributed as a patch under
Apache-2.0.
The fastest way to integrate OPOD into Relax is the patch in
training/integration/, generated against Relax main @ cbb1a82
(the exact baseline used in the paper):
git clone https://github.com/redai-infra/Relax.git && cd Relax
git checkout cbb1a82
git apply /path/to/opod/training/integration/opod-relax.patch
pip install -e .It contains the OPOD method, the sample_verifier task reward that every recipe requires, and
Qwen2.5-Omni model support. You still need to supply your own training data in the format
described in training/integration/README.md. See that file
for the file-by-file breakdown and for manual-integration instructions.
- Python 3.10+, PyTorch with CUDA, NVIDIA GPUs (the paper uses 32× H20).
- Relax with the OPOD patch applied, plus its
model-parallel backend (Megatron-LM) and SGLang — see
training/integration/README.md. - vLLM (or another OpenAI-compatible server) for evaluation and for hosting teacher servers.
- Benchmark datasets for the adapters you want to run.
All machine-specific paths use /path/to/... placeholders — adapt them to your environment.
cd eval
# 1) sanity check against a served model
bash smoke_test.sh --teacher omni --model <served-model-name>
# 2) evaluate a local HF checkpoint end to end
bash eval_hf.sh --hf-dir /path/to/OPOD-Qwen2.5-Omni-7B --teacher omni \
--tag opod-7b --datasets all
# 3) or run the evaluator directly against an existing endpoint
python runner/run_eval.py --teacher omni --datasets all \
--base-url http://127.0.0.1:8000/v1 --model <served-model-name> \
--out-dir results/opod-7bEach run writes results/<run>/overall.json with per-dataset and aggregate scores.
Evaluation protocol (identical for all methods in the paper): accuracy, temperature 0.7,
top-p 0.7, max prompt / response length 8,192 tokens. Prompts follow the training format —
reasoning inside <think></think>, final answer inside <answer> </answer> — see
eval/prompts/system_prompts.py.
# 0) install Relax with the OPOD patch (see training/integration/README.md)
git clone https://github.com/redai-infra/Relax.git && cd Relax
git checkout cbb1a82
git apply /path/to/opod/training/integration/opod-relax.patch
pip install -e . && cd -
export RELAX_LOCAL_ENV_SCRIPT=/path/to/Relax/scripts/entrypoint/local.sh
export MODEL_CONFIG_DIR=/path/to/Relax/scripts/models
# 1) point the recipes at your prompt manifests
export DATA_DIR=/path/to/datasets
export PROMPT_SET=$DATA_DIR/manifests/train_omni.jsonl
# 2) train the three modality specialists (teachers) with GRPO
bash training/recipes/run-qwen3-30B-A3B-omni-teacher-text-16xgpu.sh
bash training/recipes/run-qwen3-30B-A3B-omni-teacher-image-16xgpu.sh
bash training/recipes/run-qwen3-30B-A3B-omni-teacher-audio-16xgpu.sh
# 3) distill them into one student with OPOD (run on every node)
bash training/recipes/spmd-multinode-opod.sh \
training/recipes/run-qwen3-30B-A3B-opod-32xgpu.shOPOD components are exposed as environment variables:
| Component | Configuration |
|---|---|
| Routed teachers | TEACHER_TEXT_URL, TEACHER_IMAGE_URL, TEACHER_AUDIO_URL |
| One-sided teacher guidance | OPD_RECTIFIED=1 |
| Adaptive modality control | OPD_DUAL_KL=1, OPD_TARGET_KL_AUTO=1 |
| Warm-up calibration | OPD_TARGET_KL_WARMUP=10 |
| Controller range | OPD_BETA_MIN=0.1, OPD_BETA_MAX=1.5 |
| Verification reward | OPD_PROCESS_REWARD=1 |
| Reasoning gain | OPD_PROCESS_COUNTERFACTUAL=1 |
Key hyperparameters from the paper: 8 samples per prompt, rollout batch size 16, global batch
size 64, verification weights
See training/recipes/README_opod.md and
QUICKSTART_opod.md for the full configuration
template, and eval/README.md for evaluation details.
- The recipes read a modality-balanced prompt manifest supplied via
PROMPT_SET; the required row schema (modality label + verifier fields) is documented intraining/integration/README.md. Data sources, per-modality composition, filtering, and licenses are described in the paper's supplementary material. - Training prompts are disjoint from every evaluation benchmark: exact and normalized-string overlaps against all test sets are removed.
- Every result uses a single run per configuration with the same random seed.
This repository (code and documentation) is released under the Apache License 2.0. The released model weights follow the license of their respective base model — see each model's Hugging Face card. The training data, where applicable, is not covered by this license.
@article{zhao2026opod,
title = {OPOD: On-Policy Omni Distillation},
author = {Zhao, Tong and Hu, Yuyang and Zhu, Yutao and Li, Reed and
Liang, Haijin and Shi, Haibo and Lu, Yu and Dou, Zhicheng},
journal = {arXiv preprint arXiv:2607.20918},
year = {2026},
url = {https://arxiv.org/abs/2607.20918}
}Built on the Qwen omni-modal model family (Qwen2.5-Omni, Qwen3-Omni) and an open-source multimodal on-policy RL framework. We thank the maintainers of the twelve benchmarks used in our evaluation.