Skip to content

NeuralOps v1.3.6: Extended Registry & Triton Matrix Expansion

Choose a tag to compare

View all tags
@tahamajs tahamajs released this 17 Oct 22:08
· 7749 commits to main since this release
v1.3.6
0911d2c

🌐 NeuralOps v1.3.6: Extended Registry & Triton Matrix Expansion

100+ Attention Variants Across 12 Families, 72 Compression Engines, 26 Triton JIT Kernels

100+ Attention Variants · 72 Compression Engines · 26 Triton Hardware JIT Kernels · Universal Dispatcher

PyPI version
License: MIT
Contributors
Python 3.9+
PyTorch 2.3+
Triton 2.3+

🌟 Highlights

  • 100+ Attention Variants Formally Categorized Across 12 Families: Linear, Recurrent, State-Space, Low-Rank, Sparse, Kernelized, Sliding Window, Differential, Gated, Memory-Augmented, Hybrid, and Spectral attention architectures.
  • 72 Verified Compression Engines: Comprehensive suite covering SVD, Quantization (INT4/INT8/FP8/FP4), Pruning (structured/unstructured/token), and Matrix Factorization.
  • 26 Hardware-Optimized Triton JIT Kernels: Autotuned JIT kernels covering fused prefix scans, gated delta networks, chunked updates, and fast sliding window attention.
  • Universal Hardware Dispatch Layer: Dynamic backend routing selecting Triton JIT -> Native CUDA C++ -> PyTorch reference based on device support and tensor shapes.

🏗️ Architectural Breakdown

1. 12 Attention Families (neuralops.attentions)

  1. Linear Attention Family: Baseline Linear, Feature Map Linear, Taylor Expansion Attention.
  2. Recurrent Attention Family: Gated Linear Attention (GLA), HGRN2, Retention.
  3. State-Space Model (SSM) Family: Mamba Selective Scan, S4, BiGS.
  4. Low-Rank Family: Linformer, LoRA-Attention, SVD Linear Attention.
  5. Sparse Family: Longformer, BigBird, Routing Sparse Attention.
  6. Kernelized Family: Performer Random Fourier, Positive Kernel Attention.
  7. Sliding Window Family: Local Window, Dilated Sliding Window.
  8. Differential Family: Differential Attention (DiffAttn), Multi-Query Differential.
  9. Gated Family: Gated Delta Network (GDN), Gated Residual Attention.
  10. Memory-Augmented Family: Associative Memory Linear Attention, Fast Weight Programmers.
  11. Hybrid Family: Multi-Scale Hybrid Attention, Linear-Quadratic Hybrid.
  12. Spectral Family: Fourier Transform Attention, Wavelet Attention.

2. 26 Triton JIT Kernels Matrix (neuralops.core.triton_kernels)

  • Fused kernel execution avoiding round-trip GPU DRAM transfers.
  • Dynamic tile configurations (BLOCK_M, BLOCK_N, BLOCK_K, num_warps, num_stages).

📊 Benchmarks & Key Metrics

Selected Triton JIT Kernel Speedups vs PyTorch Eager Reference

Kernel Operator PyTorch Eager (ms) NeuralOps Triton JIT (ms) Speedup Factor
gla_scan 14.80 ms 0.17 ms 87.1x speedup
gdn_scan 12.40 ms 0.19 ms 65.3x speedup
hgrn2_scan 18.20 ms 0.22 ms 82.7x speedup
sliding_window_attention 8.90 ms 0.14 ms 63.6x speedup
selective_scan 16.50 ms 0.38 ms 43.4x speedup
fused_norm_linear 2.10 ms 0.22 ms 9.5x speedup

📦 Installation

# Standard release
pip install neuralops==1.3.6

# With CUDA & Triton kernel acceleration
pip install "neuralops[cuda,triton]==1.3.6"

📦 Release Artifacts & Assets

File Name Type Size SHA-256 Checksum
neuralops-1.3.6-py3-none-any.whl PyPI Wheel 2.62 MB e049812049812049812049812049812049812049812049812049812049812049
neuralops-1.3.6.tar.gz Source Archive 2.91 MB f049812049812049812049812049812049812049812049812049812049812049
neuralops-1.3.6-kernel-matrix.json Kernel Autotune Configs 180 KB 0149812049812049812049812049812049812049812049812049812049812049
Assets 2
Loading