Repository navigation
torch-vulkan 0.2.0
The first public release: a native PyTorch device ("vk") on portable Vulkan
compute, aimed at scientific workloads (float64, complex numbers, FFTs, dense
and sparse linear algebra, special functions) on integrated, discrete and
phone GPUs, without vendor libraries.
Highlights
- No CPU fallbacks in PyTorch's tests. Across 36 of PyTorch's
device-generic test files (about 160,000 tests each on lavapipe and an NVIDIA
GPU), no op computes on the CPU or calls it to raise an error; every
remaining failure is shared with the CPU or listed with its reason. Every
OpInfo op runs on the GPU and matches the CPU. - Results that match the CPU, independent of the driver: Cephes-style
float32 transcendentals, a float64 math library, correctly rounded float32
division, sqrt and rsqrt and float64 division without fused multiply-adds, exact float8 conversions, and fixes
for driver quirks found on Intel (Windows and Mesa), NVIDIA, Arm Mali and
lavapipe. - Faster kernels: subgroup-tiled matrix products (float32 level with MKL on
an Intel UHD 620), float16 on tensor cores through cooperative matrices
and bfloat16 on tensor cores (155-165 TFLOP/s on an H100 NVL or L40S, about
95% of NVIDIA's own Vulkan sample on the H100), FFTs in register-resident passes
with the real-to-complex step fused in (within 2x of CUDA, except
transforms of 2^16-2^20 points at up to about 2.3x, from per-call latency),
a matrix-vector kernel,eighby blocked tridiagonal reduction and divide
and conquer (float64 at 1.5-3.4x CUDA's time, from 50-110x),svdby
bidiagonalisation (0.2-0.8x: faster than CUDA in all our measurements), and
solveandqrwith their panels spread over many workgroups (solve
1.4-3.4x;qrof a 1000x64 matrix about 4-5x, from 20-40x). - Training: fused and foreach Adam, AdamW, SGD and Adagrad, and
torch.amp.GradScaler, all on the GPU. - New coverage: quantisation (int8 weight-only and dynamic 4-bit matrix
products, float8_scaled_mm, per-tensor affine quantised tensors, fake
quantisation), nested tensors,nn.MultiheadAttention's fast path, Lanczos
upsampling, complex32 and bcomplex32,segment_reduce.
Installing
Wheels for Windows and Linux (x86_64), Python 3.10 or newer, PyTorch 2.14:
pip install "torch==2.14.*"
pip install <wheel URL from the assets below>
On Android (Termux), build from source with the prebuilt kernels: extract
torch-vulkan-spirv-v0.2.0.tar.gz and run tools/build-termux.sh with that
directory.
Tested on
Intel UHD Graphics 620 (Windows), Intel HD Graphics 530 (Mesa ANV, Linux),
NVIDIA H100 NVL and L40S (Linux), Arm Mali-G57 (Android, Termux), Mesa
lavapipe (CI, with the validation layer).
Known limitations
torch.compile has no fusing backend yet (planned for 0.3.0). One device;
tensors under 2^31 elements;
per-tensor affine quantisation only. See the README's Limitations section and
benchmarks against CUDA.
The development was directed by Asem Wardak, with the code written by Claude
Opus 5.5 (Anthropic); every change is checked against PyTorch's CPU results.