numr 0.8.0

Destination-passing ops and a capture-safe CUDA stream for graph replay. CUDA kernels ship as multi-arch fatbins inside the binary. New activation, padding, grouped-matmul, and transform ops.

Breaking

• CapturedGraph::new_with_arena takes a fifth parameter, arena_bytes_used: usize: the arena's peak footprint at the end of capture.
• CudaClient::stream() returns &GuardedStream instead of &CudaStream. Build launches with GuardedStream::launch_builder. GuardedStream::raw() returns the bare CudaStream.
• New required trait methods — BinaryOps::copy_into, ShapeOps::pad_mode, and MatmulOps::matmul_wide have no default body. A downstream implementor of these traits must add them.
• numr::runtime::cuda::kernels::launch_gemv_kernel_bt is removed. F32 and integer small-M matmul go through MatmulOps::matmul.
• kernel_names::NORM_MODULE and FUSED_ADD_NORM_MODULE are removed. Norm kernels ship as one module per op: NORM_RMS_MODULE, NORM_LAYER_MODULE, NORM_GROUP_MODULE, FUSED_ADD_RMS_NORM_MODULE, FUSED_ADD_LAYER_NORM_MODULE.
• kernel_names::GEMV_INT_MODULE is removed with the integer GEMV kernel.

Added

• Destination-passing ops — BinaryOps::copy_into, HostCopyOps::write_host_slice, and RandomOps::{rand_into, rand_seeded_into, randn_into, randn_seeded_into} write into a caller-owned tensor. The buffer keeps its device address, so a captured CUDA graph replays against it. On CUDA each one records as a graph node.
• CUDA graph capture lock — numr::runtime::cuda::capture adds DeviceCaptureLock, CapturePermit, EnqueuePermit, thread_is_capturing, GuardedStream, and GuardedLaunchBuilder. A capture holds the device's write lock for its whole region. Each enqueue holds the read side for that one enqueue. Threads that share a device no longer leak launches into another thread's graph.
• Capture queries — CudaClient::is_capturing() answers for the calling thread. CudaClient::stream_capture_active() reports the driver's capture state on the stream.
• Typed arena exhaustion — Error::ArenaExhausted { requested, used, capacity } replaces a formatted string. Error::as_arena_exhausted() returns the three byte counts, unwrapping one AllocFailed layer. CapturedGraph::arena_bytes_used() reports the arena's peak, so a caller can size the next capture.
• Activations — ActivationOps::gelu_erf, gelu_erf_mul, and gelu_erf_mul_bwd compute exact erf GELU, matching PyTorch F.gelu(approximate="none"). snake_beta and snake_beta_bwd compute Snake with per-channel alpha and beta. All five run on CPU, CUDA, and WebGPU, with autograd wrappers.
• Padding — PadMode::{Constant, Reflect} and ShapeOps::pad_mode. Reflect matches PyTorch F.pad(mode="reflect"). CPU, CUDA, and WebGPU each have a native reflect kernel.
• Matmul — MatmulOps::matmul_wide returns the F32 accumulator for F16 and BF16 inputs, so a caller that keeps summing rounds once. GroupedMatmulOps::{grouped_matmul, grouped_matmul_activation} run one matmul per row group, with group offsets held on the device. Grouped matmul covers F32, F16, and BF16 on CPU and CUDA.
• Transforms and norms — FwhtOps::fwht is a block-wise, normalized Walsh-Hadamard transform on CPU, CUDA, and WebGPU. NormalizationOps::l2_normalize divides by max(‖x‖₂, eps) along one dim.
• Device profiles — DeviceProfile, DeviceArch, and DeviceCaps describe SM count, shared memory, and tensor-core support (bf16, int8, fp8, and the int8 MMA shape). Device::profile() defaults to a conservative unknown profile. CudaDevice queries the driver once per device and caches the result.
• Context-free CUDA queries — CudaDevice::count(), CudaDevice::product_name(), and CudaDevice::total_memory() need no CUDA context. count() returns 0 when the driver library is absent.
• CUDA schedule tuning — numr::runtime::cuda::tune (tuned, time_launches) picks between launch alternatives that produce identical bits. It probes once per device and key, then caches the winner. Probes never run inside a graph capture. Set NUMR_CUDA_TUNE to 0, false, or off to use the fixed defaults.
• Embedded multi-arch fatbins — build.rs compiles every kernel to a fatbin with native SASS per arch, plus compute_75 and compute_120 PTX for JIT. The fatbins are embedded with include_bytes!, so the binary reads no kernel file at run time.
• NUMR_CUDA_ARCH — selects the SASS archs: unset builds for the host GPU, a list such as 86,89 builds those archs, and all builds every supported arch. Use all for redistributable binaries.
• CUDA kernels — tensor-core WMMA GEMM with fused bias, activation, and residual epilogues, at several tile sizes. im2col + GEMM paths for conv1d, conv2d, and conv_transpose1d. A column-blocked depthwise_conv2d. A tiled transpose for strided copies. Serial and split kernels for dim reductions. A small-M F32 kernel for transposed weights.

Changed

• CUDA requirement — the cuda feature needs CUDA 12.8 or newer, because the build always emits compute_120 PTX. CUDA 12.8+ and 13.x both work.
• CPU transcendental accuracy — f64 exp, exp2, expm1, sin, cos, tan, and the hyperbolic family reach double precision. The f64 log family stays within 2 ulps. f32 exp, exp2, expm1, cbrt, log, and the hyperbolic family reach single precision. f32 sin, cos, tan, atan, asin, and acos stay within 2 ulps. f64 exp, sinh, and cosh cover the full representable domain.
• CPU performance — Rayon dispatch is gated on work size. Broadcast binary ops take a row-wise SIMD path. Strided copies coalesce dims and walk rows. FFT twiddle tables are cached. Strided conv1d is vectorized. The tiled F32 matmul reads a transposed B in place instead of copying it.
• Dependencies — the archived paste macro crate is replaced by its maintained fork pastey. The benchmarks use fluxbench 0.2.
• CUDA performance — RMSNorm uses a register-cached single pass with quad-packed loads. Norm kernels compile one fatbin per op. Softmax uses a flattened grid. Batched F32 GEMM and grouped F16/BF16 GEMM run on tiled and tensor-core kernels.

Fixed

• Determinism — CPU element-wise kernels run per row, so a row's result no longer depends on how many rows share a call. The CPU matmul column split depends on shape only, never on thread count. CUDA conv kernel selection no longer depends on batch size.
• Graph capture — the allocator routes only the capturing thread to the arena. A failed begin_capture unfreezes the allocator. The CUDA index-bounds check and the Sobol warmup check no longer synchronize or misread state during another thread's capture. A missing Sobol cache entry returns an error instead of panicking.
• Tuning races — concurrent probes for one key run once, and the first cached value stays.
• CUDA launch limits — element-wise launches that overflow the grid and matmul launches over the shared-memory budget return an error instead of computing wrong elements.
• Batch broadcasting — matmul_bias, matmul_bias_residual, fp8_matmul, and semiring_matmul broadcast each operand's batch dims per index instead of wrapping.
• Narrow-float accumulation — CPU fused GEMM epilogues and every direct convolution kernel accumulate F16 and BF16 in F32, and F64 in F64.
• Norm stability — layer_norm, group_norm, and fused_add_layer_norm shift by a per-row reference before accumulating, on every backend. The WebGPU shaders keep that shift through compilation.
• Special functions — bessel_i0 and bessel_i1 no longer overflow early in the asymptotic branch. SIMD pow honors IEEE 754 identities for non-finite exponent products. f64 pow falls back to scalar for non-positive bases. CPU f64 softmax and logsumexp, and the CUDA softmax merge, handle all-masked rows. NEON f64 gelu_mul saturates tanh instead of dividing exp(2x).
• SIMD dispatch — f64 AVX-512 kernels that use bitwise ops require AVX-512DQ, so AVX-512F-only CPUs no longer trap.
• Autograd — conv1d and conv2d input gradients are correct when input and output channel counts differ.
• WMMA selection — BF16 WMMA runs only on devices with native BF16 tensor cores (sm_80+). Padding and launch read the same predicate.
• dispatch_dtype! — a wildcard arm returns Error::UnsupportedDType, so downstream crates compile against the #[non_exhaustive] DType.
• Feature gating — builds without f16 keep the non-half code in the clamp and where kernels.

Full diff: v0.7.0...v0.8.0