Skip to content

NeuralOps v1.3.5: Distillation & Algorithmic Maturation

Choose a tag to compare

View all tags
@tahamajs tahamajs released this 17 Oct 22:08
· 7722 commits to main since this release
v1.3.5
b40f4ca

🎓 NeuralOps v1.3.5: Distillation & Algorithmic Maturation

Multi-Pillar Knowledge Distillation (KD), Logit KL, Feature CKA, SVD-LLM & Tensor-Train

6-Pillar Distillation Engine · Intermediate Feature CKA · Attention Alignment · SVD-LLM Whitening · Tensor-Train

PyPI version
License: MIT
Contributors
Python 3.9+
PyTorch 2.2+

🌟 Highlights

  • Multi-Pillar Knowledge Distillation Engine: Unified training pipeline implementing 6 complementary distillation objectives: Logit KL Divergence, Intermediate Feature CKA (Centered Kernel Alignment), Attention Map Alignment, Hidden State Cosine Distance, Task Cross-Entropy, and Spectral Divergence.
  • SVD-LLM Integration: Incorporated state-of-the-art activation whitening transformations and singular value truncation strategies tailored specifically for large language models.
  • Tensor-Train (TT) Decomposition: High-order tensor contraction engine decomposing bulky linear projection weight tensors into compact matrix product states.
  • Algorithmic Maturation: Unified mathematical bridge relating linear attention, state-space models (SSMs), and low-rank recurrent architectures into a continuous family.

🏗️ Architectural Breakdown

1. Multi-Pillar Distillation Engine (neuralops.training.distillation)

  • Total Loss Formulation:
    $$\mathcal{L}{total} = \alpha \mathcal{L}{task} + \beta \mathcal{L}{logit_KL} + \gamma \mathcal{L}{feature_CKA} + \delta \mathcal{L}{attn_align} + \lambda \mathcal{L}{cosine}$$
  • Linear CKA Alignment: Computes invariant intermediate feature representation similarity without requiring identical hidden dimensions between teacher and student.

2. SVD-LLM Engine (neuralops.compression_engine.svd_llm)

  • Data-Driven Whitening: Computes Cholesky decomposition of input activation covariance $S = X X^T = L L^T$.
  • Whitened Factorization: Performs SVD on $W L$, guaranteeing optimal Frobenius error bound in the output space.

3. Tensor-Train Decomposition (neuralops.compression_engine.tensor_train)

  • Decomposes weight matrix $W \in \mathbb{R}^{\prod m_i \times \prod n_i}$ into cores $G_k \in \mathbb{R}^{r_{k-1} \times m_k \times n_k \times r_k}$.
  • Achieves up to 16x parameter reduction on large MLP layers.

📊 Benchmarks & Key Metrics

Student Distillation Recovery (Teacher: LLaMA-7B, Student: NeuralOps Subquadratic 3B)

Training Configuration Token Budget WikiText-2 PPL MMLU Score Distillation Throughput
Scratch Pretraining 100B 9.84 34.2% 18,200 tok/s
Logit KL Only 15B 7.92 39.4% 20,400 tok/s
Multi-Pillar Distillation (Ours) 15B 6.78 43.8% 22,400 tok/s
Teacher Dense Reference N/A 5.68 45.3% N/A

📦 Installation

# Standard release
pip install neuralops==1.3.5

# Full distillation & compression suite
pip install "neuralops[distill,all]==1.3.5"

📦 Release Artifacts & Assets

File Name Type Size SHA-256 Checksum
neuralops-1.3.5-py3-none-any.whl PyPI Wheel 2.48 MB b049812049812049812049812049812049812049812049812049812049812049
neuralops-1.3.5.tar.gz Source Archive 2.74 MB c049812049812049812049812049812049812049812049812049812049812049
neuralops-1.3.5-distill-configs.tar.gz Distillation Recipe Bundle 210 KB d049812049812049812049812049812049812049812049812049812049812049
Assets 2
Loading