⚗️ Awesome On-Policy Distillation

A curated collection of papers, technical reports, frameworks, and tools for on-policy distillation (OPD) of large language models.

On-policy distillation trains a student on samples from its own evolving policy, while a teacher (external, privileged, or self-conditioned) provides dense supervision on those same samples.

On-policy distillation (OPD) trains a student on trajectories sampled from its own policy while a teacher scores the student-visited prefixes with dense token-level guidance. This on-policy data collection reduces the train-inference distribution gap that affects off-policy KD/SFT on fixed traces. Depending on the estimator, OPD looks like GKD on student rollouts or policy-gradient/RL with teacher-defined per-token KL/log-prob rewards, making the natural contrast sparse outcome-reward RL rather than RL as a whole. As of 2026, OPD is a standard post-training primitive at Alibaba (Qwen3), DeepSeek (V4), Xiaomi (MiMo), Zhipu (GLM-5), NVIDIA (Nemotron-Cascade 2), and others.

Shipping today? Jump to Frameworks and Implementations. New to OPD? Read Start Here.

Start Here

A fast path through the field:

Survey. OPD Survey — taxonomy, methods, and open problems in one place.
Foundations. MiniLLM, GKD, and ExOPD — the core student-rollout plus teacher-supervision loop, including its dense KL-constrained RL framing.
Practical intuition. Thinking Machines blog — the clearest end-to-end explanation of why and when OPD applies.
When OPD works and when it breaks. Revisiting OPD, Entropy-Aware OPD, and Rethinking OPD — failure modes (instability, diversity collapse, tokenizer mismatch) and success conditions (compatible thinking patterns, novel teacher capability).
No teacher logits. Black-Box OPD — discriminator-based reward when the teacher is API-only.
No teacher at all. OPSD and SDFT — same model as student and self-teacher.
Context and experience. OPCD and OEL — distill prompts and deployment traces into weights.
Industrial recipes. Qwen3, DeepSeek-V4, MiMo-V2-Flash, GLM-5 — how labs ship OPD in production.

Key decision: access to teacher logits? Yes → white-box (GKD, Veto, Entropy-Aware OPD). No → black-box (GAD, OVD) or self-distillation (OPSD, SDFT).

Surveys and Essays

Surveys and Position Papers

A Survey of On-Policy Distillation for Large Language Models (2026) — First dedicated OPD survey; organizes methods by feedback signal, teacher access mode, and loss scope.
A Brief Overview: On-Policy Self-Distillation in Large Language Models (2026) — Beginner-oriented overview of on-policy self-distillation, cataloguing privileged-context designs where a single model is both teacher and student.
Post-Training is About States, Not Tokens: A State Distribution View of SFT, RL, and On-Policy Distillation (2026) — Reframes SFT/RL/OPD by training-state source rather than loss, explaining why OPD's student-sampled states beat a degraded teacher.
On the Geometry of On-Policy Distillation (2026) — Parameter-space diagnostics placing OPD in a relaxed off-principal regime with early subspace locking, a distinct update geometry between SFT and RLVR.
Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation (2026) — Analyzes OPD checkpoint deltas, showing dense teacher supervision still yields sparse, off-principal weight edits resembling RLVR rather than dense supervised rewriting. (Code)
A Formula-Driven Survey and Research Agenda for On-Policy Distillation (2026) — Formula-driven OPD survey and research agenda splitting temporal credit from vocabulary routing.
Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning (2026) — Unifies post-training by data-policy mismatch, with a dedicated framework for on-policy teacher-guided transfer.
Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling (2026) — Test-time scaling analysis finds OPD improves sampling efficiency while often shrinking the student's high-budget capability boundary.

Essays, Blog Posts, and Walkthroughs

Thinking Machines: On-Policy Distillation (2025) — Best single-article introduction. Covers concepts, intuition, and practical use cases.
Unlocking On-Policy Distillation for Any Model Family (GOLD) (2025) — Cross-tokenizer OPD walkthrough with TRL code.
The Magic of LLM Distillation — Rishabh Agarwal, Google DeepMind (2025) — Latent Space talk tracing distillation from logits to on-policy student rollouts, framed as the DAgger-style fix for exposure bias.
Distilling 100B+ Models 40x Faster with TRL (2026) — HF engineering walkthrough of TRL's DistillationTrainer scaling tricks; ~40× speedup, validated on Qwen3-235B → Qwen3-4B math.
Multi-Teacher On-Policy Distillation: A New Post-Training Primitive (2026) — Yumo Xu surveys MOPD as a post-training primitive across MiMo-V2-Flash, GLM-5, Nemotron-Cascade 2, DeepSeek-V4.
On-Policy Distillation: Theory & Practice in Model Merging (2026) — ByteDance Seed framing OPD as entropy-regularized RL; cross-tokenizer pitfalls and reward hacking in agent merging.
On SFT, RL, and on-policy distillation (2026) — Will Brown's essay on OPD via SFT-vs-RL compounding and gradient geometry; pointers toward an optimal teacher.
SFT, RL, and OPD Through a Distributional Lens (2026) — wh's distributional-geometry framing; experiment shows OPD students from SFT and RL teachers converge and forget less.
On Policy Self Distillation (2026) — KL-geometry study showing OPSD inverts OPD's per-token sign and suffers larger KL shocks that GEPA hint evolution roughly halves.
What Apple found out about On-Policy Distillation (2026) — AVB's tutorial-style breakdown of "Unmasking OPD"; training-free gradient-alignment for predicting student-teacher fit.
OPD深度解析：从数学推导到DeepSeek V4、SWIFT与verl实践 / OPD Deep Dive: From Mathematical Derivation to DeepSeek V4, SWIFT, and verl Practice (2026) — Chinese-language Zhihu deep-dive deriving OPD's sequence- and token-level reverse-KL; maps variants to MiniLLM, GKD, verl, DeepSeek V4.
重温 On-Policy Distillation / Revisiting On-Policy Distillation (2026) — Chinese-language notes deriving OPD as both a SeqKD student-rollout mirror and RL with token-level teacher supervision.
The Imitation Game: State of Policy Distillation in Language Model training (2026) — Long-form OPD/OPSD survey with a four-axis failure-modes taxonomy; argues hybrid OPSD and cross-tokenizer OPD as the highest-leverage open problems.
Scaling SDPO (2026) — Trajectory field note extending self-distillation policy optimization to stale off-policy production rollouts for single-trajectory continual learning on live interactions.
My Continuous Learning Pipeline with Self-Distillation on Agent Traces (2026) — HuggingFace continuous-learning walkthrough that self-distills coding-agent traces with TRL's SDPO trainer, using an evaluator model's targeted feedback as privileged context.
On-Policy Distillation: Promise, Pitfalls, and Prospects (2026) — Li Jiang organizes recent OPD failure analyses into three connected mechanisms — local teacher noise, horizon coverage decay, and myopic per-token supervision.
Solving OPSD (basically) (2026) — Continuation arguing OPSD's positive teacher-agreement pressure carries the useful signal while negative pressure drives length collapse, so keeping only positive matches OPD.
Why On-Policy Distillation Works and Naive Self-Distillation Doesn't (2026) — Frames on-policy distillation as KL-regularized RL toward a reward-tilted teacher, explaining why naive self-distillation distills a feedback-hallucinating template.
Self-Distilled Reasoner: On-Policy Self-Distillation (2026) — Author walkthrough deriving on-policy self-distillation from a privileged self-teacher as dense token-wise distribution matching, with a policy-gradient interpretation.
On-policy Distillation (2026) — Accessible long-form guide framing on-policy distillation as teacher-graded student trajectories that fix exposure bias, built around the GKD loop.
Understanding Self-Distillation and Privileged Information Distillation (2026) — Walkthrough deriving self-distillation and privileged-information distillation through an RL-as-variational-inference lens, spanning reverse-KL self-teachers to reward-tilted joint training.

Core OPD Papers

The papers that define on-policy distillation for LLMs.

Scope rule: A paper belongs here if its primary contribution is a new component of the OPD training loop itself — an objective, divergence formulation, stability fix, teacher access-mode variant, self-distillation variant, context-internalization mechanism, or systems/efficiency/privacy constraint applied to that loop — with student rollouts central to the learning signal, evaluated on LLM text generation or reasoning. Operational test: if removing the OPD-loop component leaves a working contribution (a working RL recipe, preference method, or KD baseline), the OPD piece is auxiliary → Adjacent. Papers that enable OPD (cross-tokenizer alignment, calibration), compose with OPD as one component of a larger RL/preference structure, or apply OPD to non-text-reasoning substrates live in Adjacent and Enabling Work or Domain Extensions.

Foundations

MiniLLM: On-Policy Distillation of Large Language Models (2023) — Reverse-KL framing for generative LMs; the paper that named the field. (Code)
GKD: On-Policy Distillation of Language Models — Learning from Self-Generated Mistakes (2023) — Unifying formulation spanning on-/off-policy mixtures with flexible divergences.

Gap-Bridging

Speculative Knowledge Distillation (2024) — Interleaved teacher/student sampling mitigates poor student rollout quality. (Code)
Black-Box On-Policy Distillation of Large Language Models (2025) — GAD: discriminator-based reward on student rollouts; no teacher logits required. (Code)
SOD: Step-wise On-policy Distillation for Small Language Model Agents (2026) — Reweights teacher guidance by step-level divergence to avoid tool-induced cascade drift.
MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate (2026) — Multi-agent debate consensus as the OPD teacher; extends to agentic tasks via step-level sampling.
ROPD: Rubric-based On-policy Distillation (2026) — Black-box OPD using prompt-specific rubrics distilled from teacher-student contrasts to score rollouts. (Code)
Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation (2026) — Backtracks straying student rollouts to the last safe state for teacher correction, targeting the reversed exposure bias on-policy distillation introduces. (Code)
Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation (2026) — Counteraction-aware multi-teacher OPD that decouples conflicting recovery and preservation gradients, recovering general capability from proxy prompts without teacher-aligned prompt coverage.
Trust-Region Behavior Blending for On-Policy Distillation (2026) — Warmup samples early prefixes from a teacher-blended behavior policy within a student-centered KL trust region, annealed to zero by warmup's end.
Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance (2026) — Spreads teacher guidance across a near-future token window, using trajectory drift to find true reasoning forks rather than high-loss single tokens.
Trust Region On-Policy Distillation (2026) — Restricts reverse-KL distillation to teacher-reliable trust regions on student rollouts, applying forward-KL to mismatched outlier tokens instead.
OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification (2026) — Replaces teacher logits with chunk-level semantic verification from Monte Carlo rollouts, enabling on-policy distillation from black-box teachers.
Trajectory-Refined Distillation (2026) — Teacher revises the student's failed rollout within on-policy support before distilling, correcting prefix failure at the trajectory rather than token level. (Code)
SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models (2024) — Teacher selectively intervenes during student generation at high token-probability discrepancies, curbing teacher misguidance over long student rollouts.
Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients (2026) — Embeds teacher candidate answers into RL prompts so the student learns on-policy, recovering signal on hard questions where every rollout fails.
SGPO: Strategy-Guided Policy Optimization for LLM Reasoning (2026) — Distills reusable strategy descriptions from teacher responses via forward-KL on student trajectories, replacing instance-level trajectory imitation.
MOPD: Multi-Teacher On-Policy Distillation for Capability Integration (2026) — Distills multiple per-domain RL teachers into one student on its own rollouts, integrating their specialized capabilities into a single model.
Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selection (2026) — Student perplexity steers the teacher's chunk-by-chunk generation toward learnable reasoning paths, selecting branches during generation instead of filtering afterward.
Weak-to-Strong Generalization via Direct On-Policy Distillation (2026) — Applies a weak teacher's pre/post-RL log-ratio as a dense reward on student rollouts, transferring the RL-induced shift rather than the final policy.
CoPD: Co-Evolving Policy Distillation (2026) — Parallel expert training with bidirectional OPD; experts co-evolve as mutual teachers during RLVR.
CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation (2026) — Schedules a per-token forward-to-reverse-KL mixture on student rollouts, gating the transition on student coverage of teacher tokens rather than time.
Weak-to-Strong On-Policy Distillation (2026) — Synthesizes a proxy teacher from two weak models' logit difference anchored on the student's base, lifting a student stronger than both. (Code)
Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold (2026) — Reliability-gated multi-teacher on-policy distillation whose teachers are deliberately trained on examples the accumulated teacher union fails to cover.
Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models (2026) — Builds a corrective teacher from how successful versus failed weak-model patches shift continuations at the strong model's prefixes, never imitating the patches.
RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection (2026) — Contrasts rollout-conditioned teacher guidance with retention anchors, injecting knowledge without erasing existing capabilities.
Pass the Baton: Trajectory-Relayed On-Policy Distillation (2026) — Lets the teacher repair failed prefixes before returning control, keeping supervision near the student's reachable trajectory. (Code)
CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning (2026) — Finds the first causal error with a teacher, then repairs short continuations through on-policy reinforcement.

Stability and Objective Design

DistiLLM: Towards Streamlined Distillation for Large Language Models (2024) — Skew-KL divergence with adaptive off-policy use of student-generated outputs; foundational OPD objective formulation. (Code)
DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs (2025) — Contrastive extension of skew-KL; student-generated outputs collected per epoch. (Code)
Veto: Stable On-Policy Distillation through Adaptive Target Reformulation (2026) — Intermediate target distribution in logit space stabilizes training.
Entropy-Aware On-Policy Distillation of Language Models (2026) — Forward-KL on high-entropy teacher tokens preserves output diversity.
ExOPD: Learning beyond Teacher via Generalized On-Policy Distillation with Reward Extrapolation (2026) — Casts OPD as dense KL-constrained RL; reward scaling enables teacher-surpassing behavior. (Code)
Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention (2026) — Black-box oracle edits student failures into proximal rollouts for reward-based distillation that preserves prior knowledge. (Code)
REOPOLD: Scaling Reasoning Efficiently via Relaxed On-Policy Distillation (2026) — Relaxes imitation with reward clipping, entropy-based dynamic sampling, and explore-to-refine training.
PACED: Distillation at the Frontier of Student Competence (2026) — Pass-rate weighting focuses learning on the student's competence frontier.
Revisiting On-Policy Distillation — Empirical Failure Modes and Simple Fixes (2026) — Truncated reverse-KL with teacher top-K support matching; fixes imbalanced signals and tokenizer mismatch. (Code)
Rethinking On-Policy Distillation — Phenomenology, Mechanism, and Recipe (2026) — Identifies compatible thinking patterns and novel teacher capability as OPD success conditions.
The Illusion of Certainty — Decoupling Capability and Calibration in OPD (2026) — Diagnoses OPD-induced overconfidence; CaOPD replaces confidence targets with student-grounded empirical success rates.
Demystifying OPD — Length Inflation and Stabilization Strategies (2026) — Repetition-driven length inflation in iterative OPD; Stable-OPD adds divergence constraints and a rollout-mixture anchor.
SCOPE: Signal-Calibrated On-Policy Distillation with Dual-Path Adaptive Weighting (2026) — Routes correct student rollouts to student-PPL-weighted MLE and incorrect to teacher-PPL-weighted KL; dual-path OPD loss design.
HPD: Hybrid Policy Distillation for LLMs (2026) — Unified reweighted-log-likelihood framework combining forward/reverse KL with off-policy and on-policy sampling. (Code)
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe (2026) — Offline difficulty-aware and online correctness-aware data balancing with outcome-guided margin calibration. (Code)
AOPD: Asymmetric On-Policy Distillation (2026) — Replaces ineffective negative reinforcement with localized teacher-distribution matching in non-positive advantage regions.
vOPD: On-Policy Distillation with a Control Variate Baseline (2026) — Closed-form per-token reverse-KL value baseline; unbiased lower-variance single-sample estimator with no extra critic.
Unmasking On-Policy Distillation — Where It Helps, Where It Hurts, and Why (2026) — Training-free gradient-alignment diagnostic; best teacher flips with student capacity and task; wrong demos hurt self-distillation except on hard math.
The Many Faces of On-Policy Distillation — Pitfalls, Mechanisms, and Fixes (2026) — Names three failure modes (student-prefix teacher-state mismatch, biased Top-K gradients, PI-free OPSD aggregation) and three stabilizers (stop-grad Top-K KL, RLVR teachers, SFT-stabilized students).
Rock Tokens — Deciphering High-Loss Tokens in On-Policy Distillation (2026) — High-loss tokens (up to 18%) persist after apparent convergence; masking them streamlines alignment. (Code)
BRTS: On-Policy Distillation with Best-of-N Teacher Rollout Selection (2026) — Auxiliary teacher-context branch alongside standard OPD; selects best-of-N teacher rollouts by correctness then student-alignment. (Code)
Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation (2026) — Dynamic release rule truncates dense supervision where the teacher's local margin collapses; counters suffix degradation in strong-to-weak OPD.
Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for LLM Post-Training (2026) — Sparse-to-dense post-training workflow framing OPD as the dense teacher-induced reward between GRPO stages.
The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs (2026) — Reward-extrapolation OPD collapses past a clip threshold on near-deterministic structured outputs, mapping where teacher-surpassing reward scaling stops working.
MOPD: Multi-Rollout On-Policy Distillation via Peer Successes and Failures (2026) — Conditions the teacher on successful and failed peer rollouts from the student's local group, sharpening token-level supervision over independent per-rollout distillation.
Teacher-Guided Policy Optimization for LLM Distillation (2026) — Feeds teacher tokens conditioned on the student's rollout as explicit on-policy-SFT targets, replacing reverse-KL's uninformative negative feedback under large teacher gaps.
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation (2026) — Applies OPD loss only to "teachable" tokens where the teacher's corrective mass lands within the student's support, separating learnable from incompatible disagreement.
AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment (2026) — Reflection-bottlenecked privileged self-distillation converting diagnostics into ReLU-gated token-level advantages, preventing the late-stage collapse of raw-oracle conditioning.
Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation (2026) — Rewards the student's top-K candidate tokens by the teacher confidence they induce one step ahead, countering supervision-fidelity decay over long reasoning chains.
OPD+: Rethinking the Advantage Design for On-Policy Distillation (2026) — Corrects on-policy distillation's biased stop-gradient advantage estimator, generalizing the objective to any f-divergence beyond the usual reverse KL.
SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment (2026) — Confines reverse-KL on-policy distillation to a mined sparse subset of safety tokens, aligning behavior while sidestepping the alignment tax.
FiRe-OPD: Filter, Then Reweight — Rethinking Optimization Granularity in On-Policy Distillation (2026) — Drops low teacher-likelihood trajectories, then soft-reweights remaining tokens by teacher confidence and student confusion, replacing hard token selection. (Code)
Reinforcement Learning from Rich Feedback with Distributional DAgger (2026) — Distills a feedback-conditioned teacher via forward cross-entropy on student-visited states, provably guaranteeing the monotonic policy improvement reverse-KL self-distillation lacks. (Code)
OPRD: On-Policy Representation Distillation (2026) — Supervises student rollouts by aligning intermediate hidden states with the teacher across layers, replacing high-variance output-space logit matching. (Code)
Physics-Guided Policy Optimization with Self-Distillation (2026) — Modulates self-distillation step size by student-teacher mutual information, stabilizing privileged-feedback updates that otherwise collapse late in training.
Constitutional On-Policy Safe Distillation (2026) — Recalibrates a constitution-conditioned teacher via cross-SFT cold-start before on-policy distillation, preventing reverse-KL collapse into over-cautious safety refusals.
When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation (2026) — Gates each historical-teacher refresh on consolidation evidence, replacing clock-driven updates to prevent state-oblivious collapse in self on-policy distillation.
SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling (2026) — Routes each distillation token by sign agreement between a binary verifier and the teacher, extrapolating consensus tokens and interpolating conflicts.
Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation (2026) — Unifies SFT, DAgger, offline RL, and on-policy distillation by decomposing the distillation KL along prefix-source and direction axes.
Escaping the KL Agreement Trap in On-Policy Distillation (2026) — Terminates student rollouts trapped in persistent low-KL teacher agreement, filtering degenerate supervision that yields no corrective on-policy signal.
Validity-Calibrated Reasoning Distillation (2026) — Reweights per-token distillation updates by the relative local validity of teacher and student next-step proposals, instead of uniform trajectory imitation.
PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation (2026) — Replaces on-policy distillation's unbounded log-ratio reward with a natively bounded Box–Cox power transformation, curbing high-variance gradients that post-hoc scaling cannot. (Code)
On the Position Bias of On-Policy Distillation (2026) — Importance-Weighted OPD upweights each token by its cumulative student-teacher prefix discrepancy, correcting position bias where later rollout tokens drift off-teacher. (Code)
Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillation (2026) — Finds decision tokens via student entropy and evidence tokens via hidden-state similarity plus teacher-student divergence for selective OPD loss.
ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation (2026) — Reweights each student trajectory by a student-teacher log-ratio score over pivotal tokens, emphasizing likely-negative rollouts without observing the final answer. (Code)
Blockwise Policy-Drift Gating for On-Policy Distillation (2026) — Reweights OPD position losses with student-only block-level policy-drift gates, stabilizing reused rollouts without altering teacher targets.
SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (2026) — Uses joint teacher-student entropy as a competence probe to tailor token-level supervision, addressing the competence-dependent degradation uniform on-policy distillation ignores.
DOPD: Dual On-Policy Distillation (2026) — Routes each token's supervision between a privileged teacher and privileged student by advantage gap, disentangling closable capability gaps from un-replicable information asymmetry.
Reward-Gated On-Policy Distillation (2026) — Gates teacher distillation on each rollout by agreement between verifier reward sign and teacher–student likelihood gap, trusting only directionally reliable teacher signal.
Trust Region Policy Distillation (2026) — Interpolates teacher and student probabilities into a proximal teacher, replacing on-policy distillation's unbounded log-ratio reward with a provably variance-bounded signal.
Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation (2026) — Anchors suppression-prone tokens to a frozen-base reference prior via an asymmetric divergence gate, preventing OPSD's collapse of native reasoning.
On-Policy Delta Distillation (2026) — Rewards on-policy distillation by a delta signal, the teacher minus its pre-tuning base, transferring reasoning traces rather than inherited preferences.
Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations (2026) — Diagnoses length-exploitation shortcuts in token-level on-policy distillation, regulating the advantage with hard clipping or order-preserving log-scale compression.
Distilled Reinforcement Learning for LLM Post-training (2026) — Weights student rollouts by clipped teacher-to-student token ratios inside the RL objective, replacing on-policy distillation's unconditional logit matching.
Outcome-Confounded Local Supervision in On-Policy Distillation (2026) — Crosses token-level teacher-student divergence with final-answer correctness, finding agreement-on-failure dominates because local divergence cannot localize where trajectories became unrecoverable.
β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation (2026) — Distills a scheduled reference-to-teacher logit interpolant derived as the optimal policy of a KL-regularized family generalizing vanilla self-distillation.
Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models (2026) — Subtracts the cross-rollout-predictable part of teacher-reference disagreement, relaxing the requirement that the distillation teacher generated the fine-tuning data.
SAF-OPD: Stable Advantage Fusion for On-Policy Distillation (2026) — Rescales the token-level distillation advantage into a bounded, time-decaying signal before fusing it with GRPO's, preventing fixed-coefficient entropy collapse.
Not Every Divergence Should Be Suppressed: Counterfactual Recoverability in On-Policy Distillation (2026) — Labels each student error state as recoverable or not by replaying teacher-continuation and rollback branches, replacing divergence-based supervision selection.
SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation (2026) — Reshapes teacher targets at sparsely probed positions using verifier-scored student continuations, correcting reverse-KL's neglect of plausible alternatives.
On-Policy Delta Distillation for Multilingual Math Reasoning (2026) — Empirical study of delta-signal on-policy distillation for multilingual math reasoning, finding English-only training transfers accuracy but drags responses toward English.
DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models (2026) — Weights self-distillation tokens by backward multi-step aggregation of divergence gaps, restoring the order dependence vanilla OPSD's uniform coefficients discard. (Code)
Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance (2026) — Applies teacher guidance only to zero-reward rollout groups, restoring gradients where group-relative reinforcement learning stalls.
DAPD: Dual-Anchored Policy Distillation (2026) — Dual self-teacher anchors constrain privileged targets, preventing unreachable guidance from destabilizing student rollouts. (Code)
When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation (2026) — Filters teacher signals by input grounding plus distributional extremity, suppressing confident but spurious guidance. (Code)
Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation (2026) — Audits privileged likelihood as token credit, exposing hindsight signals with weak outcome alignment.
Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation (2026) — Jointly adapts privileged exposure plus token weighting under a capacity budget to match evolving student learnability. (Code)
Mismatch Matters: On-Policy Distillation Beyond Token Agreement (2026) — Routes directional mismatch through bounded excess suppression plus top-K deficit recovery to avoid degenerate agreement. (Code)
SR-OPSD: Self-Referenced On-Policy Self-Distillation (2026) — Projects a reference-anchored self-teacher through Rényi divergence to decouple target placement from rollout projection geometry.
WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training (2026) — Co-trains rollout and auxiliary policies through geometric-mixture reverse KL to buffer teacher corrections from visited states.
ReOrder-OPD: Reliability-Aware Prompt Ordering for On-Policy Distillation (2026) — Schedules prompts by predicted teacher-continuation reliability before fresh student-rollout OPD.
Simple-OPD: Demystifying Warm-up for On-policy Distillation (2026) — Teacher-generated reasoning warm-up initializes OPD for faster convergence with less out-of-domain degradation. (Code)
REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation (2026) — Reliability-weighted token extrapolation with batch-adaptive budgets prevents fixed-coefficient reward hacking across domains.
CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation (2026) — Selects token-level supervision by paraphrase-calibrated counterfactual sensitivity, targeting task-relevant corrections.

Self-Distillation

OPSD: Self-Distilled Reasoner (2026) — Single model as both teacher and student via privileged information; no external teacher.
SDFT: Self-Distillation Enables Continual Learning (2026) — Demonstration-conditioned self-teaching for continual learning with less forgetting. (Code)
SDPO: Reinforcement Learning via Self-Distillation (2026) — Converts textual feedback into dense self-teacher signals for RL-like training. (Code)
Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? (2026) — Traces failures to suppression of epistemic verbalization; task coverage determines whether conciseness helps. (Code)
OPSDC: On-Policy Self-Distillation for Reasoning Compression (2026) — Compresses verbose reasoning using concise privileged self-teachers. (Code)
BIRD: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning (2026) — SFT-bootstraps the rollout distribution toward cleaner prefixes before on-policy reverse-KL self-distillation, fixing OPSD's cold-start prefix bottleneck. (Code)
GATES: Self-Distillation under Privileged Context with Consensus Gating (2026) — Consensus-gated asymmetric-context self-distillation without labels or rewards.
HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation (2026) — Privileged self-distillation on cliff prompts where RL gradients vanish; recovers KL-regularized optimal policy.
RLSD: Self-Distilled RLVR (2026) — Self-distillation as token-level credit assignment within GRPO; OPSD-style matching leaks privileged information.
SDZero: Self-Revision Turns Binary Rewards into Dense Supervision (2026) — Generator-reviser dual roles; reviser converts binary feedback into token-level supervision with no external teacher.
OPSDL: On-Policy Self-Distillation for Long-Context Language Models (2026) — Short-context distribution of the same model as co-evolving reverse-KL teacher under long context.
PBSD: Preference-Based Self-Distillation — Beyond KL Matching via Reward Regularization (2026) — DPO-style preference learning between context-augmented teacher positives and on-policy student negatives.
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models (2026) — Unifies self-distillation across supervision reliability, representation alignment, and training stability. (Code)
OPSD Compresses What RLVR Teaches — A Post-RL Compaction Stage (2026) — Correct-only OPSD preserves accuracy and shortens responses; proposes SFT → RLVR → OPSD as post-RL compaction.
ATESD: Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning (2026) — Treats teacher reveal ratio as a learnable control variable via Beta-policy controller with discounted learning-progress reward.
OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering (2026) — Contrasts averaged teacher logits over correct vs. incorrect rollouts to form outcome-guided steering on anchor logits.
RLRT: Rebellious Student — Reversing Teacher Signals for Reasoning Exploration (2026) — Upweights student tokens that diverged from teacher but still succeeded as a "valuable exploration" signal added to GRPO.
EGRSD: Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning (2026) — Teacher-entropy confidence gate over RLSD's direction-magnitude signal; causal-lookahead variant preserves transient pivot tokens.
CREDIT: From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation (2026) — Recasts the self-distillation token reward as Bayesian filtering; batch-contrastive teacher baseline strips input-generic shortcuts.
OPHSD: Training with Harnesses — On-Policy Harness Self-Distillation for Complex Reasoning (2026) — Generalizes self-distillation privileged context from a static variable (reference solution, environment trace) to a harness-driven workflow (draft-verify, plan-solve); harness is a removable training scaffold. (Code)
MixSD: Mixed Contextual Self-Distillation for Knowledge Injection (2026) — Per-token Bernoulli mix of fact-conditioned and naive-conditioned base-model samples; replaces SFT for knowledge injection without collapsing held-out capability.
AntiSD: Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information (2026) — Identifies the OPSD token reward as a PMI that suppresses deliberation tokens, then reverses its sign under an entropy-triggered gate. (Code)
TRACE: Distilling Where It Matters via Token-Routed Self On-Policy Alignment (2026) — Routes self-distillation KL only to annotator-marked spans to cure the all-token "distillation tax" of SDPO/SRPO.
AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals (2026) — Multi-view privileged self-distillation that gates teacher-specific residuals so they can adjust update magnitude but cannot reverse the cross-view consensus direction. (Code)
It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs (2026) — Two same-model teachers conditioned on self-generated feedback supply complementary reverse-KL targets on the student's own rollouts, intersected as a product-of-experts.
VPD: Learning from Language Feedback via Variational Policy Distillation (2026) — Variational-EM self-distillation refines a feedback-conditioned self-teacher in the E-step before distilling it back via token-level KL in the M-step.
RMSD: Bringing Capabilities in Distribution via Relevance-Masked Self-Distillation (2026) — Applied Compute's OPSD variant masking the reverse-KL loss to LLM-judge-selected behavior-relevant tokens; preserves capabilities where SFT collapses.
SPD: Self-Policy Distillation via Capability-Selective Subspace Projection (2026) — Decode-time KV-subspace projection biases self-rollout generation toward capability-relevant directions, then LoRA-SFTs on those rollouts without any external verifier or teacher.
Multilingual Safety Alignment via Self-Distillation (2026) — Same-model OPSD transfers English safety reasoning to low-resource languages without any response data.
COPSD: Crosslingual On-Policy Self-Distillation for Multilingual Reasoning (2026) — Uses English translations and reference solutions as privileged teacher context for low-resource multilingual reasoning OPSD. (Code)
EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation (2026) — Guides a fraction of student rollouts with the privileged context, then distills only positive-evidence tokens, internalizing rare identities OPSD never samples.
When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning (2026) — Weights OPSD token supervision by within-sequence position, the strongest tested predictor of privileged-teacher reliability, rather than ambiguous teacher entropy.
Ditto: Reinforcing Human Behavior Simulation via Verbal Feedback (2026) — Jointly GRPO-optimizes a draft rollout and its judge-feedback-conditioned refinement so the policy internalizes verbal guidance, targeting subjective human-simulation rather than verifiable rewards.
OISD: On-Policy Internal Self-Distillation of Language Models (2026) — Distills the detached final layer into an intermediate layer across model depth via advantage-weighted Jensen–Shannon alignment — needs no privileged context.
ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domains (2026) — Reflection-guided OPSD restricting self-teacher distillation to a rollout's erroneous span, targeting cross-domain reasoning generalization.
SGSD: Skill-Conditioned Gated Self-Distillation for LLM Reasoning (2026) — Skill-conditioned OPSD whose retrieved-skill teachers are outcome-validated before distillation, extending privileged self-distillation to unreliable experience-derived context. (Code)
Distilling LLM Feedback for Lean Theorem Proving (2026) — Distills a self-teacher conditioned on LLM-generated critique of the student's attempt, injecting external knowledge through natural-language feedback rather than logits or solutions.
CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO (2026) — Answer-free correctness-conditioned self-teacher bidirectionally flips GRPO token-advantage signs, unlike the privileged-context teachers of related self-distillation methods.
SC-SDPO: Restoring the Sweet Spot via Pass-Rate Weighted Self-Distillation (2026) — Reweights SDPO's self-distillation loss by an on-the-fly pass-rate term, restoring the difficulty sweet spot that pure self-distillation discards.
Self-Supervised On-Policy Distillation for Reasoning Language Models (2026) — Conditions a self-teacher on a successful peer completion to densely supervise failed on-policy prefixes within each GRPO group.
Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation (2026) — On-policy self-distillation for safety using a privileged-context self-teacher, with flip-rate prompt search selecting contexts that activate latent refusal.
Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning (2026) — Distills a temperature-scaled copy of the model's own logits to restore entropy in RL-collapsed policies before continued training.
SDPG: Self-Distilled Policy Gradient (2026) — Couples GRPO verifier rewards with exact full-vocabulary privileged self-distillation, casting its reverse-KL term as an equivalent centered log-ratio policy gradient. (Code)
Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning (2026) — Routes signed-KL self-distillation by student entropy, pulling low-entropy tokens toward the privileged self-teacher while pushing high-entropy tokens away to preserve exploration.
AR-OPD: Anchored Residual Guidance for Privileged On-Policy Distillation (2026) — Splits privileged supervision into a locally reachable partial-oracle anchor plus a scaled full-view residual, curbing hindsight leakage from monolithic full-privilege imitation.
The Role of Feedback Alignment in Self-Distillation (2026) — Compares self-teacher context forms, showing step-aligned critique matched to the solver's reasoning trace localizes distillation credit at errors unlike reference solutions.
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation (2026) — Contrasts the privileged-hint self-teacher gap under correct versus wrong hints, cancelling style-token drift to concentrate distillation signal on task-bearing tokens. (Code)
RGSD: Rubric-Guided Self-Distillation (2026) — Conditions the frozen self-teacher on per-prompt grading rubrics, distilling dense token-level targets into the prompt-only student to replace verifier-scored rubric reinforcement learning.
PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners (2026) — Privileged self-distillation re-scoring student rollouts under an adaptively masked reference solution, calibrating only high-entropy-mismatch tokens. (Code)
OPCoD: Be My Tutor — On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback (2026) — Bidirectional co-distillation where paired students condition each self-teacher on peer natural-language feedback, targeting mutual cross-domain Pareto improvement without external teachers.
Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation (2026) — Splices a student's own wrong rollouts with self-generated correct references into explicit error-correction trajectories, replacing logit-level self-distillation.
Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation (2026) — Conditions the privileged teacher on learned rubrics to give criterion-aware token-level guidance on student rollouts, replacing reference-trajectory supervision.
Self-Distillation as a Performance Recovery Mechanism for LLMs (2026) — Anchors a degraded model to a frozen earlier checkpoint via on-policy self-distillation, recovering capabilities lost to fine-tuning or compression.
On-Policy Self-Distillation Reduces Output Diversity (2026) — Sampled-demonstration teacher tilts the base distribution by pointwise conditional mutual information, collapsing rollout diversity unlike on-policy RL.
PHF: Privileged Hidden Flow for On-Policy Self-Distillation (2026) — Distills how the privileged teacher's hidden states move along the rollout, supervising internal computation that output-only self-distillation leaves untouched.
Learning from Partial Chain-of-Thought via Truncated-Reasoning Self-Distillation (2026) — Student matches a frozen self-teacher's answer distribution while conditioned only on a truncated reasoning prefix, building robustness to inference-time truncation.
Purified OPSD: On-Policy Self-Distillation Without Losing How to Think (2026) — Subtracts a reference-only teacher probe to isolate the transferable residual, distilling a closed-form PMI target that preserves long-CoT reflection.
DemoPSD: Disagreement-Modulated Policy Self-Distillation (2026) — Targets a disagreement-modulated reverse-KL barycenter of privileged teacher and student, pulling supervision toward the student where high divergence signals privileged-information leakage.
Neuron-OPSD: Neuron-Aware Data Selection for Annotation-Free LLM Self-Distillation (2026) — Derives both training-data selection and teacher context from internal neuron activations, driving annotation-free on-policy self-distillation without external oracles.
Denser ≠ Better: Limits of On-Policy Self-Distillation for Continual Post-Training (2026) — Empirical-theoretical critique showing dense on-policy self-distillation forgets more than GRPO during continual post-training, as teacher projection induces excess parameter drift. (Code)
Rethinking On-Policy Self-Distillation for Thinking Models (2026) — Diagnostic study showing privileged self-distillation degrades thinking models on long reasoning, as teacher context lowers fork rates at high-entropy positions.
CANON: Consensus as Privileged Context for Label-Free Self-Distillation (2026) — Turns agreement among a model's own samples into privileged context for a frozen self-teacher, densely supervising fresh rollouts without labels.
Enhancing Rubric-based RL via Self-Distillation (2026) — Distills a rubric-conditioned self-teacher into the policy via forward-KL on the model's own rollouts, rescuing criteria that scalar reward aggregation suppresses.
PromptSD: One Student, Many Teachers — Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context (2026) — On-policy self-distillation via a soft-prompt teacher, routing each example in a merged multi-task corpus to its own per-task prompt-teacher.
H²SD: Hybrid Hindsight Self-Distillation (2026) — Correctness-aware on-policy self-distillation that reweights updates on correct rollouts but applies reference-hint reverse-KL correction where the student's reasoning fails.
Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning (2026) — Downweights credit for tokens whose likelihoods shift most under opposite correctness assertions, recasting privileged self-teacher shifts as sensitivity rather than supervision.
Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation (2026) — Traces self-distillation's failure on hard tasks to privileged-information bias pulling per-token targets toward one reference solution rather than correctness.
On-Policy Self-Distillation without Any Supervision (2026) — Uses a majority-vote consensus rollout as privileged teacher context for distillation onto disagreeing rollouts, removing OPSD's last dependence on external supervision.
RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer (2026) — Routes multilingual token supervision by how much an English reference shifts the teacher view, targeting reasoning pivots rather than surface realization. (Code)
Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning (2026) — Replaces privileged solution traces with problem-structure guidance, preserving reasoning cues while reducing shortcut imitation.
Rubrics as Privileged Information for Open-Ended Generation (2026) — Conditions a self-teacher on task rubrics, transferring open-ended evaluation criteria through student-generated responses.
Learning from Consensus and Disagreement: Unsupervised On-Policy Self-Distillation with Minority-Trajectory Contrast (2026) — Builds privileged context from rollout consensus with minority-trajectory calibration to resist false consensus.
Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation (2026) — Conditions the self-teacher on other-problem solutions to isolate context-induced behavior from target-specific privilege. (Code)
PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation (2026) — Adapts a privileged teacher from completed student trajectories before distilling onto unchanged prefixes.
Adaptive Supervised Anchoring for On-Policy Self-Distillation (2026) — Adds alignment-weighted canonical supervision beside rollout-conditioned self-distillation to restore task signal after prefix drift.
Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation (2026) — Skill-conditioned self-teaching uses annealed tilted cross-entropy to internalize abstract guidance without privileged inference context.
I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization (2026) — Routes all-incorrect rollout groups to privileged self-distillation, withdrawing imitation once rewards become informative.

Context and Experience Internalization

OPCD: On-Policy Context Distillation for Language Models (2026) — Context-conditioned teacher on student rollouts; distills system prompts and experiential knowledge. (Code)
OEL: Online Experiential Learning for Language Models (2026) — Deployment loop using OPCD for consolidating interaction traces into weights. (Code)
Aligning Language Models from User Interactions (2026) — Hindsight self-distillation from user follow-ups; same model conditioned on the follow-up serves as the teacher.
MAIGO: Mitigating Lost-in-Conversation with History-Cleaned On-Policy Self-Distillation (2026) — History-cleaned OPSD distilling assistant-stripped reference distributions onto the student's own sharded rollouts, fixing lost-in-conversation self-contamination.
Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap (2026) — View-asymmetric self-distillation aligning on-policy multi-turn trajectories to the same model's single-turn behavior, needing no external teacher.
Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models (2026) — Canonical-context OPSD aligning multi-turn student trajectories to a full-context frozen self-teacher, countering self-anchored drift across turns.
Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight (2026) — Distills a weak-critic-conditioned self-teacher into a critique-free student, letting a weaker overseer improve a stronger model without test-time critiques.
Reasoning Compression with Mixed-Policy Distillation (2026) — A larger teacher rewrites student-sampled reasoning into concise traces for KL alignment, transferring brevity instead of enforcing length penalties.
When Context Returns: Toward Robust Internalization in On-Policy Distillation (2026) — Stop-gradient consistency regularizer anchoring the student's no-context output, making distilled privileged context safely removable when reintroduced at inference.
PMD: Procedural Memory Distillation — Online Reflection for Self-Improving Language Models (2026) — Accumulates cross-episode procedural memory conditioning a self-teacher over the policy's own rollouts, internalizing recurring strategies into a memory-free model beyond episode-local RLVR.
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories (2026) — Consolidates in-context memory into freshly added expert weights during a sleep phase, distilling a pre-update self over the model's own rollouts.
LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks (2026) — Repurposes the judge into a coach whose experiential knowledge conditions the teacher, replacing scalar rubric rewards on non-verifiable tasks.
Sample-Efficient Learning from Agent Experience (2026) — Distills an experience-conditioned self-teacher's one-step decisions branched from the agent's own recorded trajectories, requiring no additional environment interaction.
Masked Distillation: Internalizing the Chain-of-Thought in Language Models (2026) — Distills a CoT-conditioned teacher onto a student's answer-only rollouts, tuning a suffix scaffold that interpolates between full and no internalization.
Flux-OPD: On-Policy Distillation with Evolving Contexts (2026) — Distills contextual difference signals into a context-free teacher anchor, letting contexts re-extracted from student trajectories evolve as in-training supervision.
Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation (2026) — Turns failed trajectories into reflective feedback, enabling self-distillation despite rare task success.
Self-Improving Large Language Models via Progressive Experience Evolution (2026) — Evolves reusable experience from student trajectories, then distills an experience-conditioned self-teacher into the unassisted policy. (Code)
Latent On-Policy Self-Distillation (2026) — Learns privileged experience context as latent tokens to self-distill agent rollouts without designer-specified artifacts. (Code)
Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing (2026) — Injects privileged tokens into off-topic student rollouts to internalize composable knowledge edits. (Code)

Efficiency, Systems, and Privacy Variants

Prefix OPD: Fast and Effective On-policy Distillation from Reasoning Prefixes (2026) — Distills only reasoning prefixes, cutting training FLOPs 2×-47×.
OVD: On-policy Verbal Distillation (2026) — Trajectory-level verbal scoring instead of token-level logit matching; relaxes alignment requirements.
pi-Distill: Privileged Information Distillation for Language Models (2026) — Training-time privileged information in agentic settings where only actions are observable.
Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline OPD (2026) — Precomputes teacher log-probs once over SFT rollouts; 4× speedup via teacher-consistency condition.
DP-OPD: Differentially Private On-Policy Distillation for Language Models (2026) — Student-rollout OPD with DP-SGD on student updates; first OPD recipe with sample-level differential privacy.
TIP: Token Importance in On-Policy Distillation (2026) — Selective training on high-entropy and confidently-wrong low-entropy tokens; matches full-token baselines at lower memory.
Nitrobrew: Communication- and Memory-Efficient On-Policy Distillation (2026) — Hidden-state teacher→student transport plus tile-wise online divergence kernel; 1.5-3× throughput.
NPD: Near-Policy Distillation via Asynchronous Generation and Selective Packing (2026) — Decouples generation from training; sparse updates plus Δ-IFD filtering; 8.1× speedup over on-policy baselines.
Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning (2026) — Top-k overlap monitors prefix drift; attenuates unreliable rewards and truncates drifted rollouts.
EffOPD: Learning to Foresee — Unlocking Efficiency of On-Policy Distillation (2026) — Adaptively extrapolates along the current update step for ~3× training acceleration with no extra trainables.
Less is More: Early Stopping Rollout for On-Policy Distillation (2026) — Restricts rollout and reverse-KL loss to the first response tokens, where teacher supervision is strongest before it decays toward the student baseline.
ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation (2026) — Trains OPD on short teacher-anchored prefix windows whose horizon is adapted online by delayed full-rollout probes auditing prefix–full gradient alignment.
Are Full Rollouts Necessary for On-Policy Distillation? (2026) — Controls OPD rollout horizon by progressively expanding or permanently truncating student rollouts, distilling only reliable early segments to cut compute.
f-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control (2026) — Scores per-sample freshness from rollout–supervision drift to stabilize asynchronous on-policy distillation where generation and training are decoupled.
Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts (2026) — Prefix-overlap scores fixed-length student rollouts to allocate long-continuation budget only to teacher-compatible candidates, cutting long-horizon OPD compute.
AsyncOPD: How Stale Can On-Policy Distillation Be? (2026) — Asynchronous OPD pipeline decoupling rollout generation from learner updates, mapping how stale rollouts degrade reverse-KL while forward-KL stays robust.
LoRi: Low-Rank Distillation for Implicit Reasoning (2026) — Aligns the student's self-generated latent reasoning trajectory to the teacher's hidden states in a shared low-rank Tucker subspace, enabling length-invariant implicit-CoT distillation. (Code)
ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation (2026) — Recovers structurally pruned LLMs by distilling the frozen pre-pruning model on student rollouts, growing rollout length as teacher-confirmed repetition subsides.
Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation (2026) — Expands rollout horizons with training progress, concentrating early compute on short prefixes before longer reasoning becomes learnable.

Taxonomy

Cross-cutting views over the Core papers — representative examples per category, illustrative rather than exhaustive. For the full set, see the Core OPD Papers subsections. Many entries span multiple categories.

By Teacher Type

Teacher Type	Papers
External white-box	MiniLLM, GKD, DistiLLM, DistiLLM-2, Veto, Entropy-Aware OPD, ExOPD, Prefix OPD, Revisiting OPD, Rethinking OPD, TIP, TrOPD, TRD, SWITCH, OPD2, Demystifying OPD, ShortOPD, CADENCE, Distilled RL, Outcome-Confounded OPD, W2S-OPD, Lightning OPD 2.0, SAF-OPD, Counterfactual Recoverability, SPOT, Multilingual OPD2, RoCo-ACE, Relay-OPD, RSTG, SA-OPD, Adaptive FastOPD, TIDE, WDL-OPD, ReOrder-OPD, Simple-OPD, REOPD, CROP (+48 more)
External black-box	Black-Box OPD / GAD, OVD, SPoT, ROPD, OmniOPD, DistIL, ZPPO, CausalOPD
Self-teacher with privileged context	OPSD, SDFT, SDPO, OPSDC, GATES, pi-Distill, RLSD, SDZero, CREDIT, OPSA, DOPD, PHF, TRSD, Purified OPSD, AD-OPSD, CANON, BIRD, CriPO, PromptSD, H²SD, β-OPSD, CSCR, Privileged but Biased, DASH, U-OPSD, RP-OPSD, Woodpecker, RESD, PS-OPSD, DAPD, SPEE, RuPI, Privileged Likelihood, USD, SR-OPSD, CoDA, OP2SD, PAST, Adaptive Anchoring, SKALD, I-SDPO, Latent OPSD, HPSE (+39 more)
Internal self-teacher (cross-depth)	OISD
Self-teacher (non-privileged / answer-free)	CAST, TS-OPSD, SafeSteer
Context-conditioned	OPCD, OEL, FiC, CCOPD, NCA, PMD, Sleep, Coach, Agent Experience, Masked Distillation, Flux-OPD, Latent OPSD, HPSE (+2 more)
Multiple / lifecycle teachers	MAD-OPD, CaMOPD, CGTR, Multi-Teacher MOPD, TU-OPD (+8 more)

By Primary Goal

Goal	Papers
Compression / strong-to-weak transfer	MiniLLM, GKD, Prefix OPD, Rethinking OPD, Lightning OPD, TrOPD, TRD, SWITCH, KAT, PowerOPD, AsyncOPD, TOP-D, ShortOPD, CADENCE, Relay-OPD, Adaptive FastOPD (+23 more)
Stability / objective design	DistiLLM, DistiLLM-2, Veto, Entropy-Aware OPD, PACED, SCOPE, Uni-OPD, AOPD, vOPD, OPD+, AD-OPSD, OPD2, Demystifying OPD, Distilled RL, Outcome-Confounded OPD, β-OPSD, Lightning OPD 2.0, SAF-OPD, Counterfactual Recoverability, SPOT, DASH, RP-OPSD, RSTG, DAPD, SA-OPD, Privileged Likelihood, USD, TIDE, SR-OPSD, WDL-OPD, ReOrder-OPD, Simple-OPD, PAST, Adaptive Anchoring, REOPD, CROP (+ more)
Post-RL consolidation / skill integration	ExOPD, OPCritD, TS-OPSD, Multi-Teacher MOPD, W2S-OPD, TU-OPD, RoCo-ACE (+3 more)
Continual learning	SDFT, SPoT, OPCD, OEL, MixSD, CaMOPD, CCOPD, SD-Recovery, PromptSD, Sleep, Agent Experience, RESD, SPEE, Latent OPSD, HPSE (+4 more)
RL replacement / augmentation	SDPO, REOPOLD, RLSD, SDZero, OGLS-SD, PBSD, CREDIT, DA-SD, PGPO, Constitutional OPD, PHF, Direct-OPD, CANON, CriPO, H²SD, Coach, Flux-OPD, CSCR, Privileged but Biased, U-OPSD, Woodpecker, PS-OPSD, RuPI, CausalOPD, CoDA, OP2SD, SKALD, I-SDPO (+39 more)
Reasoning compression	OPSDC, MPD, TRSD, BIRD, Masked Distillation
Black-box distillation	GAD, OVD, ROPD, OmniOPD
Safety / alignment	SafeSteer, Constitutional OPD, Multilingual Safety Alignment

Adjacent and Enabling Work

Papers that are not canonical OPD but matter for understanding or deploying it.

Cross-Tokenizer and Model-Family Enablers

ULD: Towards Cross-Tokenizer Distillation (2024) — Universal Logit Distillation; foundational enabler for cross-family OPD.
Multi-Level OT for Universal Cross-Tokenizer KD (2024) — Token- and sequence-level optimal transport for cross-tokenizer KD.
CDM: Enhancing Cross-Tokenizer KD with Contextual Dynamical Mapping (2025) — Contextual dynamic mapping for vocabulary alignment.
Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching (2025) — Approximate likelihood matching across fundamentally different tokenizers.
Cross-Tokenizer Likelihood Scoring Algorithms (2025) — Exact and approximate sequence likelihood scoring across BPE vocabularies.
DSKD: A Dual-Space Framework for General KD (2025) — Unifies output spaces; supports on- and off-policy KD between any two LLMs. (Code)
GOLD: Unlocking On-Policy Distillation for Any Model Family (2025) — Cross-tokenizer OPD with TRL integration.
CTPD: Cross Tokenizer Preference Distillation (2026) — Aligned-span projection plus teacher-anchored DPO with cross-tokenizer importance sampling.
DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer KD (2026) — Dual-space token weighting plus Soft-DTW differentiable sequence alignment.
Cross-Tokenizer LLM Distillation through a Byte-Level Interface (2026) — Byte-level conversion of teacher distributions plus byte-level student decoder for mismatched tokenizers.
SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation (2026) — Short multi-token continuations replace exact matching; recovers teacher signal at mismatched positions.
Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families (2026) — Maps tokens across mismatched tokenizers so on-policy distillation transfers teacher signal across model families, not only within one series.
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation (2026) — Maps student vocabulary into a different teacher's via a rule-built sparse projection matrix, rescuing critical tokens that strict string-matching alignment drops.
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization (2026) — Re-expresses the teacher's distribution over student tokens in shared byte space, preserving probability mass that prior cross-tokenizer alignments discard.
Vocabulary Expansion of Large Language Models via Kullback-Leibler-Based Self-Distillation (2025) — Transfers old-token probabilities into an expanded vocabulary, enabling distillation across tokenizer changes without paired corpora.
Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatch (2026) — Aligns hidden states in dual vocabulary spaces through key-query matching, supporting distillation between mismatched tokenizers. (Code)
Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models (2026) — Aligns autoregressive teachers with diffusion students through architecture-aware token distributions, enabling transfer across incompatible generation processes.

Mismatch Mitigation and Student Quality

Exploring and Enhancing Distribution Transfer in KD (2024) — Analyzes reverse-KL with student-generated output; proposes OKD.
FIRST: Efficient Trustworthy Distillation (2024) — Teacher recalibration for trustworthy offline KD.
Multi-Granularity Semantic Revision (2024) — Sequence correction for low-quality student-generated outputs.
Warmup-Distill (2025) — Bridges distribution mismatch before distillation begins.
TAID: Temporally Adaptive Interpolated Distillation (2025) — Addresses teacher-student mismatch via adaptive interpolation.
SpecKD: Speculative Decoding for Effective KD (2025) — Speculative-decoding-inspired selective token-level losses.
Knowledge Distillation with Training Wheels (2025) — Entropy-regularized value optimization with on-/off-policy demonstrations.
Revealing the Power of Post-Training via KD (2025) — Offline on-policy KD: student generates, then teacher labels.
TSD-KD: Explain in Your Own Words (2026) — Student proposes candidates, teacher reranks, selective token distillation. (Code)
SSD: Embarrassingly Simple Self-Distillation Improves Code Generation (2026) — Temperature-shifted self-sampling plus SFT; identifies precision-exploration conflict.
AdaSwitch: Balancing Exploration and Guidance in KD via Adaptive Switching (2025) — Switches between on-policy rollouts and off-policy teacher data via context-aware divergence threshold.
DDT: Towards On-Policy SFT via Distribution Discriminant Theory (2026) — In-Distribution Finetuning and Hinted Decoding realign training data to the student's distribution. (Code)
DASD: Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning (2026) — On-policy correction pipeline for distribution mismatch and exposure bias in sequence-level CoT distillation.
Distillation Traps and Guards: A Calibration Knob for LLM Distillability (2026) — Post-hoc calibrates teachers via RFT to control distillability against tail noise and instability.
A Predictive Law for On-Policy Self-Distillation From World Feedback (2026) — Predictive law: a linear relation between the initial student–self-teacher gap and final OPSD improvement, estimable before training.
Rethinking Selective Knowledge Distillation (2026) — Disentangles selective distillation along position, class, and sample axes, spanning both on-policy and off-policy regimes.
RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting (2026) — Pairs data refinement with on-policy distillation, using a gold-answer-conditioned frozen teacher to supervise student trajectories against domain-fine-tuning forgetting.
DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer (2026) — Scores student rollouts with a teacher routed through a cross-lingual verbalizer, repairing teacher-student transfer for multilingual students.
PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning (2024) — Prompt-tunes the teacher to emit student-friendly targets, distilled on the student's own generations to ease exposure bias.
AllMem (2026) — Offline on-policy distillation for long-context modeling.
On Teacher Hacking in Language Model Distillation (2025) — Diagnoses students exploiting fixed teacher data, showing fresh student-generated outputs prevent the resulting objective failure.

Preference, Reward-Guided, and Hybrid RL+KD

Direct Preference Knowledge Distillation (2024) — Preference-aware KD combining reverse-KL with implicit reward objectives.
PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs (2024) — Ranking loss over teacher-preferred-over-student pseudo-preference pairs recalibrates student sequence likelihood without teacher logits.
Online Knowledge Distillation with Reward Guidance (2025) — Sequential KD via preference optimization; offline and online variants.
KDRL (2025) — Unified reverse-KL KD with RL in a single post-training objective.
RLTF-SD: Expanding RL via Text Feedback (2026) — Internalizes text feedback via self-distillation.
RLAD: Reinforcement-aware KD for LLM Reasoning (2026) — Trust-region ratio distillation on student rollouts.
Multi-Token Prediction via Self-Distillation (2026) — Online self-distillation for multi-token prediction and faster inference. (Code)
ORPO-Distill: Mixed-Policy Preference Optimization for Cross-Architecture LLM Distillation (2025) — Mixed-policy preference distillation with student-generated outputs; black-box cross-architecture transfer.
SODA: Semi On-Policy Black-Box Distillation for Large Language Models (2026) — Pairs teacher responses with a one-time static snapshot of student generations as preference pairs, avoiding fully on-policy distillation's dynamic rollouts.
SRPO: Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing (2026) — Routes correct student rollouts to reward-based RL and failed ones to self-distillation.
KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation (2025) — K-step Bellman return replaces high-variance single-step REINFORCE in sequence-level OPD.
Rethinking LLM Distillation: A Constrained MDP Perspective (2025) — Maximizes task reward under hard KL constraint against the teacher; avoids manual Lagrangian tuning.
RLKD: Distilling LLMs' Reasoning via Reinforcement Learning (2025) — Generative Structure Reward Model on student rollouts.
LUFFY: Learning to Reason under Off-Policy Guidance (2025) — Mixed-policy GRPO combining on-policy rollouts with off-policy teacher traces via regularized importance sampling. (Code)
BOND: Aligning LLMs with Best-of-N Distillation (2024) — RL mimicking best-of-N via Jeffreys-divergence matching; eliminates inference-time BoN cost.
Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignment (2024) — Game-theoretic iterative BoN as self-play; win-rate dominance optimization.
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation (2025) — Casts RLHF as token-level distillation by injecting DPO rewards. (Code)
KEPO: Knowledge-Enhanced Preference Optimization for Reinforcement Learning with Reasoning (2026) — Quality-gated OPD on high-quality trajectories plus knowledge-enhanced exploration via teacher hints. (Code)
𝒳-KD: General Experiential Knowledge Distillation for Large Language Models (2026) — Jointly models teacher reward and policy-distills so the student learns inside the teacher's original environment.
ExGRPO: Probing to Refine — Reinforcement Distillation of LLMs via Explanatory Inversion (2026) — Explanatory probes plus dialogue-structure utility bonus reward coherent reasoning over memorized answers.
NPO: Near-Future Policy Optimization (2026) — Later checkpoint of same policy as teacher; AutoNPO adaptively triggers switch to maximize RLVR signal.
CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization (2026) — Dual GRPO with on-policy KD reward between large/small models; matches standard GRPO with 18% speedup.
Combining On-Policy Optimization and Distillation for Long-Context Reasoning in Large Language Models (2026) — dGRPO augments GRPO with dense teacher-KL guidance on student rollouts in one objective, for long-context reasoning.
CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization (2026) — Contrasts correct- and wrong-answer self-teachers from RLVR rollouts into per-token evidence sharpening credit at decisive reasoning steps.
LLMR: Knowledge Distillation with a Large Language Model-Induced Reward (2024) — Scores student-sampled sequences with a dense reward induced from teacher probabilities, training via REINFORCE to alleviate exposure bias.
Extreme Region Policy Distillation (2026) — Two-stage RL distilling token-level policy signals into the base model under trust-region constraints, decoupling sample efficiency from KL drift.
Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones? (2025) — Distills self-supervised pseudo-rewards from teacher and student responses, then guides reinforcement learning on student rollouts beyond response-only distillation.
DRIFT: Difficulty Routing Self-Distillation with Rhythm-Gated Exploration (2026) — Allocates self-distillation versus reinforcement-learning signals per problem by tracked learning progress, unlike methods that optimize all problems uniformly.
Compete Then Collaborate: Frontier AI Teachers Build a Verifiable Curriculum to Improve a Coding Student Beyond Imitation (2026) — Teachers compete then collaborate to build an execution-verified curriculum used as an RLVR environment where the student learns by doing not imitation.
SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation (2026) — Distills reward-specialized policies into one online student, preventing conflicting objectives from destabilizing joint reinforcement learning.
Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models (2026) — Mixes target-policy and length-controlled external rollouts with source-specific reward processing to stabilize diffusion-language-model reinforcement learning.
TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models (2024) — Iteratively mines preference pairs from policy rollouts using teacher-ranked student reward models for scalable alignment.
Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration (2025) — Dynamically routes small-model training between offline teacher traces versus on-policy reinforcement learning based on exploration state.

Self-Play and Iterative Bootstrapping

SPIN: Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024) — Self-play distinguishing own generations from human references. (Code)
Self-Rewarding Language Models (2024) — Iterative DPO with model-as-judge self-rewards on own generations.
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025) — MCTS-guided self-evolution; policy and PRM co-improve via code-augmented reasoning. (Code)
rStar2-Agent: Agentic Reasoning Technical Report (2025) — GRPO with Resample-on-Correct rollouts plus multi-stage SFT→RL recipe for 14B agentic reasoner. (Code)
π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data (2026) — Examiner-generated tasks plus question-construction-paths as privileged context for dense student supervision.
SPHERE: Self-Evolved Preference Optimization for Mathematical Reasoning in SLMs (2025) — PRM/ORM-scored MCTS rollouts plus self-correction yield preference pairs for iterative DPO.
SGS: Scaling Self-Play with Self-Guidance (2026) — Three-role self-play (Solver, Generator, Reviewer) for theorem proving.
Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline (2026) — Samples candidate solutions to unlabeled questions, filters them through a multi-stage self-verification cascade, then SFTs on accepted ones — no teacher or ground-truth.
IRIS: Interpolative Rényi Iterative Self-play for Large Language Model Fine-Tuning (2026) — Generalizes SPIN-style self-play with an adaptively scheduled Rényi-family objective over annotated versus self-generated responses, unifying several self-play variants.
Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs (2026) — Replaces SPIN's pairwise objective with a triplet over annotated, current-synthetic, and initial-policy responses to stabilize iterative self-play.
SkillFactory: Self-Distillation for Learning Cognitive Behaviors (2025) — Rearranges the base model's own sampled solutions and reflections into structured retry-verify traces, instilling cognitive skills without a stronger teacher.
The Verifier is the Curriculum: Execution-Gated Self-Distillation for Cross-Family Game Generation (2026) — Iterates rejection-sampling self-distillation gated by a deterministic clean-launch verifier, showing verifier precision drives out-of-family generalization.
Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation (2026) — Bootstraps from feedback-conditioned second attempts, distilling the agent's own successful retries back into its base policy while carrying lessons across episodes.

Precursors

Autoregressive KD through Imitation Learning (2020) — Early precursor framing sequence-model KD as imitation learning.
Learning by Distilling Context (2022) — Context distillation; key precursor to OPCD and OEL.
Kickstarting Deep Reinforcement Learning (2018) — Auxiliary distillation loss steers a student RL agent on its own rollouts toward teacher policies, the canonical precursor to on-policy distillation.
Proximal Policy Distillation (2024) — Distills a teacher into a student RL agent on the student's own PPO rollouts, a non-LLM precursor to on-policy distillation.
Online Policy Distillation with Decision-Attention (2024) — Derives group teachers from concurrently learning policies, focusing online distillation on decisions where their action preferences diverge.

Domain Extensions

OPD applied to non-text-reasoning settings — agents, multimodal models, diffusion, audio, robotics — and to inference acceleration via speculative decoding. These pass the inclusion criterion (student rollouts central to the learning signal) but on substrates beyond LLM text reasoning.

Agents and Tool-Use

Structured Agent Distillation (2025) — Queries teacher online to avoid distribution drift in agent settings.
SCoRe: From Correction to Mastery via Reinforced Distillation of LLM Agents (2025) — Teacher intervenes at first critical error in student agent trajectories for corrective distillation.
TCOD: Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents (2026) — Short-to-long trajectory-depth curriculum mitigating multi-turn KL instability.
Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (2026) — Plain-prompt student; skill-augmented same model as token-level self-teacher for multi-turn agent training.
SDAR: Self-Distilled Agentic Reinforcement Learning (2026) — Sigmoid-gated OPSD auxiliary on top of GRPO for multi-turn agents; amplifies positive-gap, attenuates negative-gap tokens. (Code)
GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation (2026) — Reverse-KL between student and ground-truth-conditioned same-model teacher gives token saliency; KL-initiated entropy-terminated segmentation propagates credit within segments to sign-aware reweight GRPO advantages.
Revisiting DAgger in the Era of LLM-Agents (2026) — Turn-level (DAgger) and trajectory-prefix (AggreVaTe) student/teacher rollout mixtures with teacher actions queried at every visited state.
Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning (2026) — Alternates GRPO with offline self-distillation on correct-fewest-search / divergent-sibling pairs mined from the converged rollout pool.
Healthcare AI GYM for Medical Agents (2026) — Clinical-agent gymnasium plus Turn-level Truncated OPD: an EMA teacher conditioned on outcome-privileged hints that are stripped before logprob comparison.
HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents (2026) — Parallel multimodal search agent that applies OPD only to failed rollouts to salvage correct intermediate tool calls from GRPO's uniform negative advantage. (Code)
DGPO: Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities (2025) — Selective reverse-KL teacher guidance on student-generated outputs within PPO; teacher intervenes only when the compact agent's autonomous attempts fail.
EMPO²: Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization (2026) — Hybrid agent RL whose off-policy mode distills memory-tip-conditioned rollouts into the tips-free policy, internalizing memory-driven exploration without tips at inference.
StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning (2026) — Converts teacher–student log-probability gaps into sign-preserving GRPO advantage shaping localized to action-centered step spans rather than whole agent trajectories.
HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents (2026) — Distills reverse-KL from a feedback-conditioned self-teacher onto only the failure-relevant action spans the agent selects from its own failed rollouts.
What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents (2026) — An environment-conditioned teacher's hindsight log-probability gap on student action tokens becomes a bounded, sign-aware GRPO advantage reweight anchored to task reward.
GRAFT: Graph-Tokenized LLMs for Tool Planning (2026) — On-policy tool-context distillation where a subtask-privileged same-model teacher supervises the student's own tool-token trajectories, curing exposure bias in graph-tokenized tool planning.
GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering (2026) — Agentic-KBQA OPD distilling a gold-action-conditioned self-teacher onto entity-anchored student action spans, densifying sparse outcome rewards.
COMAP: Co-Evolving World Models and Agent Policies for LLM Agents (2026) — Co-evolves a textual world model and agent policy by self-distilling the world model on its own on-policy next-state rollouts.
HERA: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation (2026) — Agentic self-distillation reflecting each environment observation into a turn-level diagnosis, supplying locally aligned dense feedback for multi-turn credit assignment.
SGCD: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents (2026) — Turns teacher/student divergence over mixed sibling rollouts into a detached credit signal reshaping GRPO token advantages, rather than a tool-use-destroying actor loss.
Guided-OPD: On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents (2026) — Mixes teacher- and student-generated turns within each agent rollout, decaying the teacher-intervention probability along a curriculum to counter compounding multi-turn distribution drift. (Code)
Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding (2026) — Down-weights GUI-grounding self-distillation's coordinate-token teacher signals when the student prefix can no longer complete into the ground-truth box.
OPD-Evolver: Cultivating a Holistic Agent Evolver via On-Policy Distillation (2026) — Distills a hindsight-privileged self-teacher into a self-evolving memory agent across the full experience lifecycle rather than execution alone. (Code)
SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation (2026) — Verifier-free multi-turn agent OPD where a teacher judges each student turn, allocating distillation only where intervention is necessary and reliable.
OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning (2026) — Extracts hierarchical episode- and step-level hindsight skills from agent rollouts, routing them critical-first to score responses into token-level self-distillation advantages. (Code)
ATOD: Annealed Turn-aware On-policy Distillation for Multi-turn Autonomous Agents (2026) — Anneals one hybrid advantage from on-policy distillation toward reinforcement learning, letting multi-turn agents imitate the teacher early then exceed its ceiling.
CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic RL (2026) — Reuses GRPO's free sibling rollouts to form signed counterfactual token credit for self-distilled agentic RL, replacing the retrospective sign-blind teacher-student gap.
KbSD: Knowledge Boundary aware Self-Distillation for Agentic Search (2026) — Builds a knowledge-boundary-hinted self-teacher giving dense token-level supervision for agentic-search calibration, replacing sparse binary rewards that underspecify the reasoning.
Building Multi-Task Agentic LLMs via Two-Phase Distillation (2026) — Consolidates per-task RL experts into one agentic model via off-policy distillation followed by on-policy refinement, avoiding each phase's standalone failure.
Multi-Turn On-Policy Distillation with Prefix Replay (2026) — Off-environment multi-turn agentic OPD that replays pre-collected teacher trajectory prefixes for student action, eliminating fresh environment rollouts during training.
UI-MOPD: Multi-Platform On-Policy Distillation for Continual GUI Agent Learning (2026) — Routes student GUI rollouts to platform-specific teachers for on-policy reverse-KL distillation, unifying continual multi-platform agent learning in one policy.
TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training (2026) — Turn-aware on-policy distillation for long-horizon agents, adaptively budgeting rollout depth to concentrate supervision on decision-critical turns.
SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning (2026) — Converts the policy's own trajectories into hindsight skills whose probability shifts on sampled actions form a self-evolving token-level distillation signal. (Code)
Diagnosing and Calibrating Tool-Call Boundary Drift in Multi-Teacher On-Policy Distillation (2026) — Multi-teacher on-policy distillation for tool-use agents, soft-clamping extreme per-token divergence at tool-call boundaries to curb drift while preserving gradients.
Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents? (2026) — Analyzes on-policy self-distillation on a search agent's own rollouts, diagnosing why decoding collapse makes the KL signal uninformative.
DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment (2026) — Iteratively self-distills a search agent on its own rollouts verified by a deterministic offline tool environment, needing no stronger teacher.
MAPD: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search (2026) — Feeds a style-normalized protocol to a privileged student branch whose token distributions densify sparse agentic-search rewards, bypassing proprietary teachers' hidden logits.
The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation (2026) — Isolates long-horizon planning in a controlled multi-turn environment, finding on-policy distillation has a broader effective region than GRPO.
CRPO: Contrastive Reinforced Policy Optimization via Privileged Self-Distillation (2026) — Ranks rollout positions by student-teacher entropy gap into a contrastive objective, pushing away from exposure-biased positions instead of imitating all.
OVCSD: Outcome-Verified Comparative Self-Distillation for LLM Agents (2026) — Distills only environment-verified teacher continuations branched from shared states of all-failed student rollouts, replacing unvalidated action-level teacher scores. (Code)
GRSD: Group-Reflective Self-Distillation for Agentic Reinforcement Learning (2026) — Contrasts the policy's own reflections on successful versus failed group rollouts into guidance conditioning a self-teacher's turn-level credit. (Code)
SKILL-KD: Contrastive Skill Distillation for LLM Agents (2026) — Distills teacher-student trajectory discrepancies into textual skill patches validated by rerunning the frozen student, updating a skill library instead of weights.
DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation (2026) — Switches the rollout executor by accumulated turn-level student-teacher discrepancy, deciding when teacher support is needed rather than scheduling it by progress.
EvoReason: Self-Evolving Reasoning Primitive-Guided On-Policy Distillation for Latent Reasoning in Generative Recommendation (2026) — Refines a generative recommender's own latent reasoning rollouts with a reasoning-primitive-conditioned self-teacher, replacing raw chain-of-thought traces as latent supervision.
OCSD: Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation (2026) — Contrasts full and observation-ablated replay views to isolate an observation residual that modulates token-level agent updates, removing the replay-scaffold confound. (Code)
When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents (2026) — Applies privileged distillation only at turns whose execution state matches the reference trajectory, replacing full-path guidance that misleads at uncovered states. (Code)
The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents (2026) — Gives a GUI agent's self-teacher the next screenshot as privileged evidence, distilling only where the student fails but hindsight recovers the action.
MENTOR: A Reinforcement Learning Framework for Enabling Tool Use in Small Models via Teacher-Optimized Rewards (2025) — Converts teacher references into flexible rewards on student tool-use rollouts, transferring agent behavior without direct trajectory imitation.
EviBack: Search-Agent Reinforcement Learning via Evidence-Constrained Teacher Backoff (2026) — Backs off zero-reward search rollouts to an evidence-constrained teacher, recovering supervision without overriding successful agent behavior.
CAST: Game Solvers as Turn-Level Teachers for LLM Agents (2026) — Converts solver values into turn-level advantages on student game rollouts, providing dense guidance without teacher logits. (Code)
Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations (2026) — Distills teacher action recommendations during live cyber interactions, giving lightweight agents dense guidance in sparse-reward environments. (Code)
MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation (2026) — Routes student GUI rollouts to platform experts, distilling only structured action tokens into one cross-platform agent.
SERL-SQL: Selective Hindsight Distillation for Text-to-SQL Reinforcement Agentic Learning (2026) — Reweights teacher-rescored SQL rollouts with execution feedback, targeting agent failures while bounding noisy hindsight guidance. (Code)
AlphaG-OPD: Reliability-Gated Sibling Counterfactuals for On-Policy Distillation in Symbolic Alpha Factor Discovery (2026) — Distills sibling counterfactuals at student-visited symbolic states, gating teacher targets by reliability before policy updates.
EviSD: Evidence-Conditioned Self-Distillation for Search-Augmented Agents (2026) — Rescores student search actions with an evidence-conditioned self-teacher, bounding corrections to preserve on-policy exploration. (Code)
PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning (2026) — Persists privileged-teacher disagreement across later turns, assigning credit beyond the token where agent trajectories first diverge.
Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation (2026) — Validates teacher interventions through student continuations, rejecting locally plausible guidance that harms downstream agent outcomes. (Code)
Agentic Reinforcement Learning with Self-Distilled Reward Shaping (2026) — Rescores student trajectories with a privileged skill-conditioned snapshot, assigning return-aware credit to agent decisions. (Code)
TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning (2026) — Replays tool trajectories with execution hindsight, distilling turn-level credit into the unassisted agent. (Code)
AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning (2026) — Aggregates privileged self-teacher gaps into recursive turn-level credit, propagating sparse outcomes through long agent trajectories. (Code)
Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents (2026) — Rescores skill-conditioned rollouts under augmented and pruned contexts to modulate agent credit while retaining runtime skills.
MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents (2026) — Reconstructs compact-memory rollout states before teacher scoring to preserve valid supervision across rewritten agent contexts. (Code)
Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning (2026) — Normalizes hindsight-conditioned token gaps across rollout turns to reallocate dense supervision without changing trajectory-average weight. (Code)
BOUND: Brief-Guided Corrective Preference Distillation at Search-Control Boundaries (2026) — Builds brief-guided corrective preferences at student-induced search states to target drift without runtime teacher context. (Code)
Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation (2026) — Conditions a self-teacher on GUI rollout reflections to suppress corrupted supervision after erroneous coordinate prefixes.
SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation (2026) — Visual skill cards privilege the teacher on student action prefixes to internalize GUI procedures without runtime retrieval.
LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation (2026) — Distills privileged reflection on student agent trajectories through a fast channel calibrated by outcome-based reinforcement learning.
Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents (2026) — Converts privileged self-teacher disagreement into step-level advantages on failed search trajectories, preserving reward direction. (Code)
Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents (2026) — Uses a privileged self-teacher to modulate verifier-directed token credit across multi-turn agent rollouts.

Multimodal and Vision-Language

Video-OPD (2026) — OPD for temporal video grounding in multimodal LLMs.
Reinforced Attention Learning (2026) — On-policy attention distillation for multimodal models.
VOLD: Reasoning Transfer from LLMs to Vision-Language Models via OPD (2025) — Text-only teacher distills reasoning into VLM via student-generated traces with combined GRPO and OPD.
PRISM: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL (2026) — Black-box OPD pre-alignment between SFT and RLVR for VLMs; MoE discriminator supplies adversarial signals.
VISD: Enhancing Video Reasoning via Structured Self-Distillation (2026) — Structured video judge feeds EMA teacher with privileged feedback; direction-magnitude decoupling stabilizes RL+supervision.
Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation (2026) — Distills a crop-conditioned privileged self-teacher into the full-image student along the student's own multimodal rollouts, internalizing regional-to-global visual zooming.
Visual-Advantage On-Policy Distillation for Vision-Language Models (2026) — Reweights VLM on-policy distillation by "visual advantage," the teacher's log-prob gain from fine-grained image detail, so supervision targets vision-critical tokens.
Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding (2026) — Decomposes vision-language distillation into language-prior and visual-grounding gradients, steering student-rollout updates toward the visual subspace to fix perceptual bottlenecks.
Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization (2026) — Distills failed multimodal RLVR rollouts toward a frozen reference conditioned on answer-free privileged hints, providing dense guidance without answer-revealing shortcuts. (Code)
GNDPO: Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization (2026) — Normalizes token-level reverse-KL distillation rewards into batch-relative advantages, suppressing gradient spikes from out-of-distribution teacher-student logit mismatches. (Code)
Visual-SDPO: Self-Distillation Policy Optimization via Visual Feedback (2026) — Distills rendered-artifact visual feedback from a weight-sharing privileged teacher into a code-generating student, weighting tokens by defect-to-statement code credit.
ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation (2026) — Multimodal OPD where a visual-cue-privileged teacher supervises student rollouts, using cues recoverable from the image to avoid answer-leakage shortcuts. (Code)
OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing (2026) — Distills a frontier-rationale-privileged self-teacher into the multimodal student along its own rollouts, using rationales as teacher-side evidence rather than imitation targets.
Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning (2026) — Distills a unified model's visual-generation reasoning into its text pathway by scoring student rollouts against a privileged-visual-thought teacher.
Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation (2026) — Routes separate perception and reasoning teachers to distinct segments of a student's multimodal rollouts, curbing visual-grounding shortcuts.
V-Zero: Answer-Label-Free On-Policy Distillation for Fine-Grained Visual Reasoning (2026) — Gates dense token-level distillation of student trajectories using a contrastive regional-crop-versus-negative-view signal, requiring no answer labels. (Code)
H-OPD: Confidence-Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation (2026) — Confidence-aware heterogeneous multi-teacher OPD arbitrating vision-language and text-only teachers per token along the student trajectory, replacing static per-sample teacher routing. (Code)
Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillation (2026) — Weights on-policy distillation of a student VLM's rollouts by each token's reliance on visual evidence, sharpening medical diagnostic reasoning.
COPD: Contrastive On-Policy Distillation (2026) — Scores student tokens under paired light- and heavy-thinking teacher instructions, using the score gap as a token-level advantage for adaptive reasoning compression.
OPOD: On-Policy Omni Distillation (2026) — Routes each omni-model rollout to a modality-matched specialist teacher, tuning guidance strength per modality to prevent cross-modal interference.
Visual Contrastive Self-Distillation (2026) — Shapes on-policy self-distillation targets by contrasting an EMA teacher's image-conditioned and blank-image predictions, needing no privileged answers.
NOPD: Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation (2026) — Trains a VLM on corrupted inputs against its own clean-input predictions, needing no external teacher or ground-truth answers.
RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models (2026) — Uses the high-versus-low-resolution image gap as the teacher's privileged information, supervising low-resolution student rollouts without annotations or external models.
Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detection (2026) — Grounds AI-generated image detection in on-policy self-distillation that reweights each token by the privileged teacher's corrective value. (Code)
VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation (2026) — Projects the teacher's correction onto the evidence-present-minus-removed direction, shifting the student's own logits instead of matching the privileged teacher. (Code)
Correcting What You Cannot See: Credit Assignment for Perception Distillation in Multimodal Reasoners (2026) — Weights perception-span distillation by a soft-AND of downstream failure and teacher disagreement, separating correctable perception errors from reasoning difficulty.
Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models (2026) — Keeps only the teacher correction lying in the student's visual tangent space, replacing the full teacher target with a capacity-aware one.
OPD-V: Visual On-Policy Self-Distillation with Modality Balance (2026) — Selects on-policy tokens via a trust region between zoom-in and masked-image teachers, treating modality balance itself as the privileged signal. (Code)
OPLD: On-Policy Latent Distillation for Multimodal Reasoning (2026) — Aligns student-generated reasoning with a chain-of-thought teacher across token-plus-latent spaces, strengthening multimodal inference without longer outputs.

Speech and Audio

X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs (2026) — Student on-policy rollouts with token-level teacher feedback for cross-modal speech-LLM distillation.
Step-Audio-R1 Technical Report (2025) — Modality-Grounded Reasoning Distillation produces audio reasoning grounded in acoustic features. (Code)
CORD: Bridging the Audio–Text Reasoning Gap via Weighted On-policy Cross-modal Distillation (2026) — Audio-conditioned rollouts; text-conditioned same model as teacher; importance-weighted reverse KL plus GRPO.
Data-Efficient On-Policy Distillation for Automatic Speech Recognition (2026) — Cross-modal OPD for speech recognition: a compact audio-conditioned student learns from a frozen ASR teacher scoring its own transcript rollouts.
ParaBridge: Bridging Paralinguistic Perception and Dialogue Behavior in Speech Language Models (2026) — Distills a paralinguistic-scaffold-conditioned self-teacher into a scaffold-free speech student along its own dialogue rollouts, internalizing acoustic cues without labels.
X³-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment (2026) — Scores audio-conditioned student rollouts token-by-token with a text teacher granted the gold answer, adding answer privilege to cross-modal alignment.

Diffusion, Flow, and Generative Media

π-Flow: Policy-Based Few-Step Generation via Imitation Distillation (2025) — Imitation distillation aligns student flow-model trajectories with teacher under standard flow matching. (Code)
D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models (2026) — OPSD ported to few-step T2I diffusion; text-only student vs. text+image teacher with velocity-MSE on rollouts.
Flow-OPD: On-Policy Distillation for Flow Matching Models (2026) — Per-domain Flow-GRPO experts supervise student SDE rollouts via reverse-KL with Manifold Anchor Regularization. (Code)
TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM (2026) — Partitions masked positions by remaining decoding steps into near (CE) and distant (KL) subsets. (Code)
DiMO: Distilling Masked Diffusion Models into One-step Generator (2025) — First OPD for masked discrete diffusion image generation; Generalized Jeffrey divergence with DMD-style auxiliary.
DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models (2026) — Lifts OPD from autoregressive tokens to diffusion denoising via closed-form reverse-KL along student rollouts; unifies SDE and ODE samplers.
LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation (2025) — On-policy distillation for real-time interactive video diffusion, extending the Self-Forcing few-step student-rollout recipe.
AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map Distillation (2026) — On-policy flow-map distillation along the student's own Euler rollout; supports any-step video generation.
AFD: On-Policy Adversarial Flow Distillation for Autoregressive Video Generation (2026) — On-policy distillation of a black-box video teacher into a causal student, scoring the student's own rollouts with an adversarial flow-matching discriminator.
OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generators (2026) — On-policy self-distillation for few-step autoregressive video generators, supervising the student's own rollout with real long-video context to reduce long-horizon error accumulation.
GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models (2026) — Diffusion-LLM RL-as-self-distillation matching denoiser logits to an advantage-guided self-teacher, bypassing ELBO surrogate likelihood bias.
CollectionLoRA: Collecting 50 Effects in 1 LoRA via Multi-Teacher On-Policy Distillation (2026) — DMD-based multi-teacher distillation consolidating many effect LoRAs into one student LoRA for few-step image editing.
Knowledge Distillation for Visual Autoregressive Models (2026) — Distills visual autoregressive image models along student-generated token rollouts with teacher-scored suffixes, using confidence reweighting for ambiguous visual tokens.
OPDLM: Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation (2026) — Converts an autoregressive LM into a diffusion LM along the student's own reverse-diffusion rollouts, supervised by the frozen original model. (Code)
Diffusion-GR2: Diffusion Generative Reasoning Re-ranker (2026) — Converts an autoregressive reasoning re-ranker to block-diffusion, on-policy-distilling its own parallel-decoded trajectories against the frozen autoregressive teacher's dense per-token targets.
T3D: Few-Step Diffusion Language Models via Trajectory Self-Distillation (2026) — Self-distills a few-step diffusion-LM student onto the full-step teacher's own generative trajectory via a reverse-KL objective curbing factorization error.
Learning from the Self-future: On-Policy Self-Distillation for dLLMs (2026) — First on-policy self-distillation tailored to diffusion LLMs, conditioning the self-teacher on self-generated answers as suffix context rather than autoregressive prefixes. (Code)
Causal-rCM: Teacher-Forcing and Self-Forcing Distillation for Autoregressive Video Diffusion (2026) — Self-forcing reverse-KL distillation on student-generated autoregressive video chunks, unified with teacher-forcing consistency-model initialization.
DanceOPD: On-Policy Generative Field Distillation (2026) — On-policy velocity-field matching at student-generated rollout states, hard-routed across capability fields to compose one image generator.
RMMD: Rewarded Moment Matching Distillation for Diffusion Fine-tuning (2026) — Couples diffusion moment-matching distillation with on-policy reward maximization, repurposing the distillation loss as KL regularization to preserve generative fidelity.
dOPSD: On-Policy Self-Distillation for Diffusion Language Models (2026) — Draws the teacher's privileged signal from later, more-decoded steps of the student's own denoising trajectory, replacing the external reference label. (Code)
Trace-Based On-Policy Distillation for Masked Diffusion Language Models (2026) — Supervises a masked diffusion language model on its own sampled denoising trajectories via token-level reverse-KL, replacing off-policy masked-state supervision.
DLM-One: Diffusion Language Models for One-Step Sequence Generation (2025) — Score-distills a continuous diffusion language model into a one-step generator by matching teacher scores at the student's own embedding samples.
FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models (2026) — Matches student and reference flow trajectories continuing from the same student-visited state, replacing vanilla OPD's temporally mismatched pointwise KL.
Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation (2026) — Constrains the positive prediction and guidance direction separately, fixing the branch-error asymmetry that composed classifier-free-guidance matching induces.
OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models (2026) — Compresses the longest outcome-preserving future prefix into one transition along the few-step student's own partial decoding states, replacing off-policy supervision.
Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Models (2026) — Re-encodes student-generated images through each teacher's own encoder, enabling on-policy distillation from heterogeneous flow teachers whose latent spaces are incompatible.
STEP-OPD: Rethinking Output Targets and Internal Dynamics in On-Policy Distillation for Diffusion Models (2026) — Extrapolates the on-policy distillation target past the teacher using the teacher-minus-base velocity direction, lifting the student above its task-specific teachers.
DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models (2026) — Extrapolates specialized-teacher velocity targets on student trajectories using a degraded reference for reward-aligned directions.
FlowErase-OPD: Multi-Concept Erasure via Anchored On-Policy Distillation in Flow Matching Models (2026) — Distills concept-erasure teachers on student flow trajectories while balancing difficult erasures against a retention anchor.
TRACE-GS: On-Policy Trajectory Distillation with Privileged Geometric Conditioning for Sparse-View 3DGS Restoration (2026) — Queries a dense-geometry teacher along sparse-view student denoising trajectories to target states missed by independent noise.
DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation (2026) — Combines DMD on autoregressive video rollouts with consistency distillation to preserve teacher-mode coverage. (Code)
Distilling Physical Priors into Streaming World Models (2026) — Matches causal self-rollouts to a physics-aware diffusion teacher before routing local physical rewards to aligned denoising actions.
HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models (2026) — Anchors student video sub-trajectories to privileged teacher states for condition-matched hybrid-policy self-distillation. (Code)
Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillation (2026) — Matches causal-teacher scores to student-generated video prefixes for context-consistent few-step distillation.
Alaya-EVOKE: From Linear-Scaling Supervision to Endless World (2026) — Matches long-horizon teacher scores across self-forced video rollouts to prevent drift in a few-step world model. (Code)

Embodied, Robotics, and Control

VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via OPD (2026) — Reverse-KL OPD bridging offline SFT and online RL for robotic manipulation.
LLM4Teach: Large Language Model as a Policy Teacher for Training RL Agents (2023) — LLM teacher distills into small RL agent that surpasses teacher through environment interaction. (Code)
RPD: Refined Policy Distillation — From VLA Generalists to RL Experts (2025) — Teacher VLA actions guide student during RL exploration; combines RL with behavioral cloning. (Code)
OPD-AVMP: On-Policy Distillation of Language Models for Autonomous Vehicle Motion Planning (2026) — Generalized OPD for LLM-based driving planners; 5× compression at near-teacher performance.
Adversarial Dual On-Policy Distillation from Expressive Flow-based Teacher (2026) — Co-trains a demonstration-learned flow-matching teacher that supplies reward and action signals on embodied student rollouts, enabling OPD without a fixed strong teacher.
World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning (2026) — Distills a privileged-future evaluator that sees ground-truth outcomes into a multimodal controller along its own on-policy world-model rollouts.
GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Driving (2026) — Distills a handbook-conditioned teacher into a driving VLM along the student's own rule-free rollouts, removing inference-time handbook lookups.
World Model Self-Distillation: Training World Models to Solve General Tasks (2026) — Distills a caption-conditioned video-diffusion demonstrator into an instruction-only executor along the student's own rollouts, eliciting task-solving without curated task-execution video supervision.
ROAD-VLA: Robust Online Adaptation via Self-Distillation for Vision-Language-Action Models (2026) — Builds an action-space proximal teacher by perturbing the policy's own action logits with advantage estimates, converting sparse rewards into dense supervision.
Distilling Realizable Students from Unrealizable Teachers (2025) — Queries privileged teachers only at critical student-visited states, combining realizable guidance with autonomous recovery rollouts.
LeAct: Learning to Reason from Expert Actions (2026) — Selects student-generated reasoning by its recovery of expert actions, transferring latent decision logic without expert rationales.
FIRE-VLA: Failure-Informed Self-Evolution for Vision-Language-Action Models in Autonomous Driving (2026) — Routes failed driving rollouts to a hidden-future self-teacher for answer-token supervision. (Code)

Speculative Decoding (Draft-Model Training)

Draft-model training for speculative decoding shares OPD's core loop: the draft (student) generates, the target (teacher) verifies, and the draft is updated to match. Included for breadth even though the goal is inference acceleration rather than student capability.

Online Speculative Decoding (2023) — Continuously updates draft on observed queries via KD; 1.42×-2.17× latency gains. (Code)
DistillSpec: Improving Speculative Decoding via Knowledge Distillation (2023) — Aligns draft with target via on-policy data and task-tailored divergence.
HASS: Learning Harmonized Representations for Speculative Sampling (2024) — Harmonized objective and context distillation fixes train-decoding inconsistency. (Code)
Falcon: Faster and Parallel Inference through Enhanced Semi-Autoregressive Drafting (2024) — Coupled Sequential Glancing Distillation strengthens inter-token dependencies in semi-AR drafters. (Code)
CORAL: Consistent Representations across Multi-step Training with Lighter Speculative Drafter (2025) — Cross-step representation alignment for multi-step drafter training.
EAGLE-3: Scaling up Inference Acceleration via Training-Time Test (2025) — Direct token prediction with multi-layer feature fusion under on-policy training-time test; up to 6.5×. (Code)
MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of VLMs (2025) — Adapts SLM into VLM drafter via self-distilled visual instruction tuning.
DVI: Draft, Verify, and Improve — Toward Training-Aware Speculative Decoding (2025) — Self-speculative drafter trained online from verifier decisions via KL→RL schedule.
ReSpec: Optimizing Speculative Decoding in Reinforcement Learning Systems (2025) — Evolves drafter during RL via reward-weighted distillation on rollouts.
DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting (2026) — Multimodal speculative-reasoning drafter with verifier-gated parallel execution.
MSD: Speculative Decoding Reimagined for Multimodal Large Language Models (2025) — Decouples text/visual tokens in draft; two-stage training lifts MLLM speedups to 2.29–2.46×.
SpecVLM: Fast Speculative Decoding in Vision-Language Models (2025) — Elastic visual compressor plus online-logit distillation; 2.5–2.9× end-to-end VLM speedups.
ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding (2025) — Lightweight vision adaptor compresses image tokens; trained on target-generated long responses.
Aurora: When RL Meets Adaptive Speculative Training (2026) — Online continual draft training; target verifications stream into FKL/RKL fine-tuning then hot-swap into serving.
SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting (2026) — Block-iterative drafter with layer-wise shift; valid-prefix masking and cost-aware bandit adaptation.
SFDD: Flatter Tokens are More Valuable for Speculative Draft Model Training (2026) — Sample-level flatness filters EAGLE training data; 2× speedup at 50% data with <4% inference-speedup loss.
OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decoding (2025) — Online on-policy distillation on the draft's own generated tokens; cross-vocabulary n-gram cache lets one drafter serve any target.
Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs (2026) — Trains an OPD-aligned confidence head whose acceptance decision replaces speculative decoding's verifier pass, unifying latent input compression with multi-token-prediction output.
LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding (2026) — Replaces KL-proxy draft training with objectives directly targeting acceptance rate, since capacity-limited drafters minimizing KL converge to low-acceptance solutions.
Draft-OPD: On-Policy Distillation for Speculative Draft Models (2026) — Draft-model OPD that replays drafting from verification-exposed error positions, training on target feedback over both accepted and rejected proposals. (Code)
Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding (2025) — Trains the draft model on its own verified tree rollouts via a group-standardized acceptance-length reward, aligning training with tree-based decoding.
JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting (2026) — Trains a causal parallel draft head over the frozen target's hidden states, producing branch-conditioned speculative trees that resolve tree drafting's causality-efficiency dilemma. (Code)
FastEagle: Cascaded Drafting for Accelerating Speculative Decoding (2025) — Single-pass cascaded drafter trained without teacher forcing, so each layer learns under its own same-pass prediction errors against target supervision.
TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decoding (2026) — Trains a multimodal speculative drafter on its own draft blocks using verifier acceptance-length rewards under a distillation-derived KL anchor.
Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding (2026) — Adapts a drafter online to the current policy so speculative decoding accelerates training-loop rollout generation without perturbing the sampling distribution.

Technical Reports and Industrial Recipes

Production training pipelines that use OPD as a post-training stage.

Year	System	OPD Usage	Link
2024	Gemma 2	KD as alternative to next-token prediction for 2B and 9B students	arXiv
2025	Qwen3	Strong-to-weak; off-policy then on-policy distillation	arXiv
2025	Qwen3-Omni	Off-policy then on-policy distillation before GSPO	arXiv
2025	GLM-4.5 / 4.6	Multi-stage post-training with expert model iteration and RL	arXiv
2025	HY-MT1.5	Multi-stage translation: SFT + OPD + RL	arXiv
2026	MiMo-V2-Flash	Multi-Teacher OPD (MOPD) as post-training stage	arXiv
2026	GLM-5	On-policy cross-stage distillation to recover earlier skills	arXiv
2026	Typhoon-S	Minimal sovereign recipe: SFT + OPD + small-scale RFT	arXiv
2026	Nemotron-Cascade 2	Cascade RL + multi-domain on-policy distillation	arXiv
2026	Baichuan-M3	Task RL → offline policy distillation → multi-teacher OPD	arXiv
2026	MobileLLM-R1.5	Final-stage on-policy KD as primary improvement over R1	model card
2026	Nanbeige4-3B-Thinking	OPD preferred over off-policy for math reasoning	model card
2026	DeepSeek-V4	Domain-expert SFT+GRPO → unified model consolidation via OPD	report
2026	Qwen3.5-Omni	Specialist distillation → privileged-input self-distillation aligning audio to text	arXiv
2026	HY-Embodied-0.5	32B → 2B on-policy distillation; student rollouts, teacher token-level supervision	arXiv
2026	KAT-Coder-V2	Specialize-then-Unify: 5 domain-expert agents → unified via OPD on student trajectories	arXiv
2026	KAT-Coder-V2.5	Multi-Teacher OPD (MOPD): reverse-KL on student trajectories fuses 5 experts, stabilized by drift-aware truncation	arXiv
2026	Cursor Composer 2.5	Hint-conditioned self-teacher OPD KL added to RL for targeted behaviors (tool calls, style); built on Kimi K2.5	blog
2026	MAI-Thinking-1	Self-distillation on own RL rollouts to resume climbs after collapses or base-policy refreshes	report
2026	Nemotron 3 Ultra	Iterative MOPD: 10+ domain-specialized teachers consolidated into the student, refreshed from it each cycle	report
2026	InternVideo3	Final-stage reverse-KL distillation on student-sampled video rollouts from a stronger teacher	arXiv
2026	Kwai Keye-VL-2.0	Cross-modal multi-teacher OPD: dense token-level teacher feedback on student multimodal rollouts	arXiv
2026	NebulaExp-8B	Single- and multi-teacher OPD (MOPD), finding teacher capability outweighs scale	arXiv
2026	Mach-Mind-4-Flash	Multi-Teacher OPD (MOPD): routed reverse-KL on student rollouts fuses 10+ specialists, ending mixed-reward see-saw	arXiv
2026	Agents-A1	Final-stage multi-teacher OPD: routed teachers supervise student rollouts via salient vocabulary alignment	arXiv
2026	Audex	Cascade RL + multi-domain OPD (Nemotron-Cascade lineage), preserving text intelligence in unified audio-text model	arXiv
2026	OvisOCR2	0.8B OCR student distilled from a 4B RL teacher via top-k reverse-KL on its own page-output rollouts	arXiv
2026	Gryphon-v2	Generate-and-rank recommender replacing a cascade through teacher-ranked student candidate rollouts	arXiv

Frameworks and Implementations

Training Frameworks

Framework	Description	Link
TRL	GKD, GOLD, and MiniLLM trainers; most accessible starting point	docs
NeMo-RL	Multi-teacher and cross-tokenizer OPD at scale	docs, repo
veRL	Async on-policy KD trading strict on-policy guarantees for throughput	docs
MS-Swift	GKD and OPSD sections in the ModelScope ecosystem	docs
EasyDistill	Comprehensive KD toolkit for black-box and white-box LLM distillation	arXiv
KDFlow	Off-policy, on-policy, and cross-tokenizer distillation via decoupled backends	arXiv, repo
slime	Unified RL stack supporting on-policy distillation and hindsight hints	repo
OpenClaw-RL	Agentic RL stack with hindsight-guided OPD	arXiv
NexRL	Dedicated on-policy distillation recipes	repo
SkyRL	OPD examples and blog resources	repo
ATLAS	Continual-learning framework using GKD/GRPO from runtime traces	docs
AReaL	OPD and KDRL over student-sampled trajectories with teacher log-prob guidance	docs
rLLM	Agent RL framework (UC Berkeley Sky) with first-class OPD: `examples/math_distill/` (DeepMath OPSD + `train_deepmath_distill_tinker.{py,sh}`) and `rllm/trainer/distill/` modules over verl or tinker backends	docs, repo
SpecForge	Speculative draft training with EAGLE-3 support and hybrid parallelism	arXiv, repo
TorchSpec	Torch-native speculative draft training with disaggregated inference/training; streams target hidden states via Mooncake store; Kimi-K2.5/MiniMax-M2.5/Qwen3-Coder-Next examples	blog, repo
Tinker Cookbook	Thinking Machines' Tinker SDK recipes for off-policy KD, single/multi-teacher OPD, multi-turn tool use	recipes, repo
ROLL	Alibaba's scalable RL library for LLMs/VLMs with an OPD pipeline	repo
EasyOPD	Unified verl-based OPD with method-local supervision hooks for cross-tokenizer, self-distillation, and step-wise settings	arXiv, repo
EasyDeL	JAX/Flax framework whose GKDTrainer distills onto student-generated completions via a generalized Jensen-Shannon objective	repo
SpecRoll	Exact speculative rollout serving adapts lightweight proposers at two timescales without changing the target policy distribution	arXiv
Mixed Rollout Scheduling	Mix-aware session admission allocates cache by workload demand while preserving trainer-defined rollout mixtures across OPD plus reinforcement learning	arXiv

Implementations

OPSD — Official code for Self-Distilled Reasoner / OPSD.
SCOPE — Dual-path OPD: student-PPL-weighted MLE for correct rollouts, teacher-PPL-weighted KL for incorrect.
CaOPD — K student rollouts → empirical success rate → confidence target replacement → reverse-KL OPD.
OPSD-OnPolicyDistillation — verl-based OPD with separate teacher, agent-loop rollouts, and memory-efficient execution.
nano-opd — Hackable OPD library decoupling vLLM rollout, FSDP training, and teacher forwards across independent GPU groups.
Rethinking OPD — Official code for Rethinking OPD, with verl-based scripts and top-k teacher–student overlap diagnostics merged upstream.
DiffusionOPD — Official implementation of round-robin multi-task diffusion OPD distilling task-specialized teachers into one student along its own rollout trajectories.

Acknowledgments

This list draws on the parallel curation effort at thinkwee/AwesomeOPD, which provided pointers to several papers (notably speculative-decoding draft training, BoN distillation, self-play, multilingual and crosslingual self-distillation, clinical and multimodal agentic OPD, additional industrial reports, and several training frameworks). The two lists organize differently — thinkwee/AwesomeOPD groups by feedback signal and access mode; this list groups by methodological role — and are best read together.

Contributing

Contributions welcome. See CONTRIBUTING.md for criteria, section placement, and formatting.

Inclusion criteria: the work should involve student rollouts as central to the learning signal, or directly enable OPD deployment (cross-tokenizer, frameworks, etc.).
Entry format: [Title](url) *(Year)* — One-line description. See CONTRIBUTING.md for full examples.

Citation

@software{awesome-on-policy-distillation,
  title = {{Awesome On-Policy Distillation}},
  author = {Liu, Chris Yuhao and others},
  year = {2026},
  doi = {10.5281/zenodo.19411493},
  url = {https://github.com/chrisliu298/awesome-on-policy-distillation},
  version = {v1.0.0}
}

Name		Name	Last commit message	Last commit date
.claude/skills/add-paper		.claude/skills/add-paper
CITATION.cff		CITATION.cff
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation

⚗️ Awesome On-Policy Distillation

Contents

Start Here

Surveys and Essays

Surveys and Position Papers

Essays, Blog Posts, and Walkthroughs

Core OPD Papers

Foundations

Gap-Bridging

Stability and Objective Design

Self-Distillation

Context and Experience Internalization

Efficiency, Systems, and Privacy Variants

Taxonomy

By Teacher Type

By Primary Goal

Adjacent and Enabling Work

Cross-Tokenizer and Model-Family Enablers

Mismatch Mitigation and Student Quality

Preference, Reward-Guided, and Hybrid RL+KD

Self-Play and Iterative Bootstrapping

Precursors

Domain Extensions

Agents and Tool-Use

Multimodal and Vision-Language

Speech and Audio

Diffusion, Flow, and Generative Media

Embodied, Robotics, and Control

Speculative Decoding (Draft-Model Training)

Technical Reports and Industrial Recipes

Frameworks and Implementations

Training Frameworks

Implementations

Acknowledgments

Contributing

Citation

About

Topics

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors