NVIDIA and CUDA

NVIDIA and CUDA

The current repository contains a real, tested CUDA vertical slice for Linux x86-64 with NVIDIA hardware. It uses Hugging Face Candle and cudarc, exposes device("cuda"), supports GPU forward operations, and accelerates recognized LoRA and MLP training shapes. Older docs/using-cuda.md text predates this implementation and should be read as historical design, not current status.

Platform and build gate

CUDA crates are gated on all three conditions:

• Cargo feature cuda
• target_os = "linux"
• target_arch = "x86_64"

The foundation was proven on an RTX 5060 Ti with CUDA 13.2 and compute capability sm_120. Other NVIDIA/toolkit combinations need their own compatible CUDA_COMPUTE_CAP, driver, and Candle/cudarc support.

Representative environment and test command from the CUDA training component:

export PATH="/opt/cuda/bin:$PATH"
export CUDA_ROOT=/opt/cuda
export CUDA_PATH=/opt/cuda
export CUDA_COMPUTE_CAP=120
export LD_LIBRARY_PATH="/opt/cuda/lib64:$LD_LIBRARY_PATH"

cd sw-mlpl/components/cuda-rt
cargo test -p mlpl-cuda-train --features cuda -- --test-threads=1

Adjust the paths and compute capability to the host. For a connected UI, build/run mlpl-serve with the CUDA feature from its owning workspace and serve the web assets.

CUDA flow

flowchart LR
    Client[Browser or terminal client] --> Serve[CUDA-enabled mlpl-serve]
    Serve --> Eval[MLPL evaluator]
    Eval --> Scope[device cuda scope]
    Scope --> Ops[CUDA dispatch]
    Ops --> Candle[Candle tensors and autograd]
    Candle --> GPU[NVIDIA GPU]
    GPU --> Metrics[Loss and device telemetry]
    Metrics --> Client

Loading

Shipped capability

• CUDA runtime crates for conversion, matmul, reshape, transpose, arithmetic, unary activations, reductions, normalization, and cross-entropy-related operations.
• Traceable CUDA forward primitives for embedding, RMS normalization, causal attention, LoRA linear, and demo-model assembly.
• Candle-based autodiff and Adam update in supported fast paths.
• CPU/CUDA fp32 parity tests on real GPU hardware.
• CUDA LoRA fine-tune and CUDA tic-tac-toe fine-tune demos.
• mlpl-serve --features cuda reports cuda through /v1/devices, allowing the connected web UI to enable CUDA demos and disable unavailable MLX demos.

Critical limitation

General arbitrary-model GPU training is not yet provided by a device-aware autograd tape. The evaluator recognizes specific backend training forms, notably the head-only LoRA path and the two-linear LoRA MLP. Unsupported training graphs can fall through to CPU tape execution. Ordinary supported forward operations may still dispatch to CUDA, but that does not make the whole training loop GPU-resident.

There is not yet a separate mlpl-cuda-serve peer mirroring every aspect of the MLX peer service. The shipped vertical slice uses a CUDA-enabled mlpl-serve on the GPU host. Multi-GPU, NCCL, distributed training, and LAN auto-discovery remain future work.

Best current uses

• Run and study the CUDA LoRA and tic-tac-toe demonstrations.
• Validate CPU/CUDA numerical parity.
• Connect the full browser UI to a Linux/NVIDIA compute host.
• Develop the persistent device-tensor and general GPU-autograd architecture.

Source references: CUDA foundation status, CUDA demo parity, GPU demo roadmap, general GPU training design, and components/cuda-rt.