NVIDIA CUDA Support via Tinygrad - Working Fork Available

[Scottcjn]

Summary

We have successfully restored and verified NVIDIA CUDA support for exo using the tinygrad inference engine. The tinygrad backend was removed in the v1 prep but we have a working fork with CUDA inference confirmed on Tesla V100 and M40 GPUs.

Working Fork

https://github.com/Scottcjn/exo-cuda

Verified Hardware

• Dell PowerEdge C4130 with Tesla V100-SXM2-16GB
• Dell PowerEdge C4130 with Tesla M40 24GB
• Multi-node cluster over ethernet

How to Use

git clone https://github.com/Scottcjn/exo-cuda.git
cd exo-cuda
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

# Requires nvidia-cuda-toolkit
apt install nvidia-cuda-toolkit

# Start with tinygrad CUDA backend
exo --inference-engine tinygrad --chatgpt-api-port 8001 --disable-tui

Test Results

Successful inference with Llama 3.2 1B on V100:

{"model": "llama-3.2-1b", "choices": [{"message": {"content": "Hello, its nice to meet you."}}]}

Request

Would the exo team consider:

1. Re-adding tinygrad support to the main repo for NVIDIA GPU users?
2. Or maintaining this as an official CUDA fork?

Many users have NVIDIA hardware and would benefit from distributed inference support.

---

Tested by Elyan Labs - scott@elyanlabs.ai

[AlexCheema]

We'd be open to supporting tinygrad again in exo 1.0.

exo 1.0 was a complete rewrite so I'm not sure how transferrable the work on the legacy exo would be. But the new design does make it easier to implement different inference engines. Would you like to give this a go in exo 1.0?

[Evanev7]

Closing this as slop after looking at the PR. You're too kind to the bots alex

[Scottcjn]

Understood. For clarity and for anyone reviewing this thread later: the fork linked above is a working implementation, validated on real hardware, restoring CUDA/Tinygrad inference on systems that are no longer part of the current focus.

By background, I’m an Industrial Electronic Technician and Telecom / IT Systems Architect, with 26 years of hands-on systems integration experience, including large-scale compute, networking, and heterogeneous hardware environments. I also have ~10 years of blockchain development experience and ongoing work in applied AI systems.

The intent here wasn’t to diverge from the project’s direction, but to demonstrate that the original goal of broad hardware support is still technically achievable when approached from the hardware, data-movement, and SIMD layers upward. I understand exo 1.0 is a rewrite with different priorities, so this work may be out of scope going forward.

I’ll keep the fork available with benchmarks and documentation for anyone exploring alternative inference paths or non-mainstream architectures. Respectfully(not a software developer) Scott

[Scottcjn]

Closing this as slop after looking at the PR. You're too kind to the bots alex

Slop is a strong word for work you haven’t read.
I’d invite anyone serious about inference architecture to try replicating 85- 147 tok/s on POWER8 using scalar-only paths before deciding what qualifies.

[Evanev7]

I'd love a discussion, and have read your code. Please read it yourself before submitting an issue or opening a pr.