Skip to content

Hybrid Distillation

Jump to bottom
Raul Montoya Cardenas edited this page Jul 29, 2026 · 2 revisions

Hybrid Distillation

Wiki hero

Generated with Grok Build: Grok 4.5 · xAI Imagine (/imagine)

SynapticDistill does not ship teacher models or MoE routing. Hybrid teacher–student training is a caller pattern:

  1. Run the teacher offline / out-of-process; obtain teacher_targets.
  2. Inject an SNN-only model_step.
  3. Close over teacher targets in loss_fn (e.g. cross-entropy).
  4. Call train_step! so Zygote updates the SNN model graph only.
teacher_targets = get_teacher_targets(batch_id)  # app-specific

model_step(model, spikes::SpikeBatch) = (logits = run_snn(model, spikes.spikes),)
loss_fn = output -> cross_entropy(output.logits, teacher_targets)

model, state = train_step!(model, batch, loss_fn; forward_fn=model_step, rule=:eprop)

Runnable sketch: examples/hybrid_moe_training.jl.

Related

Last updated: July 29, 2026 Updated by: Grok Build: Grok 4.5 Package tip reference: e986a66 (main, v0.2.0) Devin DeepWiki: commit e986a66 · SynapticDistill.jl

Clone this wiki locally