How about bringing GPU to TinyTorch

[aadityansha06]

I'm Aadityansha verma, c and inference developer, mostly self-taught and still learning

My work:

• Wrote a single-layer neural network from scratch in C, including custom SGD
• Optimized 1024×1024 matrix multiplication with multi-level cache tiling, benchmarked against OpenBLAS
• Built a TCP-based P2P file transfer tool and a raw-socket web chat backend
• Wrote a Linux kernel module to detect USB mass storage devices at the hardware level
• Successfully merged matrix transpose-support extension for GEADD in OpenBLAS (PR #5913)

I recently started learning CUDA, and while working through basic kernels (vector/matrix addition, etc.) I ran into a gap I think a lot of students hit: it's very hard to build real intuition about GPU execution without a GPU, and even with one, the "why is this fast/slow" reasoning (memory coalescing, warp divergence, launch overhead, occupancy) isn't something you get from just running code and reading a number.

What I'm proposing

An educational tool with two connected pieces:

1. A CPU-based GPU simulator-lets students write CUDA/Triton-style kernel code (global, thread/block indexing, threadIdx/blockIdx/blockDim) and have it execute on CPU via a simulated thread/block/grid model. Not a real compiler/GPU emulator. The goal is syntax correctness + conceptual execution tracing (which thread touched which memory, in what order, how warps would group), not performance emulation. This means:

Anyone can run and debug kernel logic without owning a GPU
The simulator can visualize what real hardware does that's normally invisible: memory access patterns, coalescing vs. scatter, warp-level divergence from branching
Common math/ML kernels (sigmoid, softmax, matmul, reductions) as guided exercises, since that's where most students' interest actually is

2. A bridge to real frameworks-once the fundamentals click, short walkthroughs connecting what the student just simulated to what actually happens when PyTorch dispatches a tensor op to GPU (kernel launch, memory transfer, why batching/fusion matters). So the mental model built on CPU transfers directly to understanding real framework internals, not just toy examples.

I'd like feedback before I sink real time into this:

[Shashank-Tripathi-07]

Hi @aadityansha06 !

I really think this can be a natural extension to the tinytorch project or we can build a separate project like a GPU-TinyTorch to make this possible. GPU Engineering is the core of Infra, inference and ML Systems and really needs students to know how the code works.

However, TinyTorch was built as a learning methodology of the under-the-hood components of PyTorch, Hence, this can be separate entity than that.

• Also a core thing I don't want to skip in this discussion is we can also use the under-the-hood for the nowadays common GPU-based-ML-Libraries like CuBLAS, CuPy [CUDA version of NumPy]. This will be a really fun thing to build and contribute to the project and I'll be interested to build this alongside. Happy to connect to discuss more on this.

@profvjreddi sir, happy to hear your advice on this, and I'll be happy to work on this too.

[Shashank-Tripathi-07]

@profvjreddi also a side update, I built a 400M parameter LLM on the side, end-to-end, totally free by using Kaggle's TPUs and GPUs.
I ended up also running that model after quantization on my laptop and experimented a lot in my holidays 😄 .

[aadityansha06]

Hey @Shashank-Tripathi-07, thanks a lot man, glad you're into this too. Would love to collaborate on it.

One more thing I want to bake into the simulator: modeling the CPU to GPU data transfer cost itself, not just kernel execution. Like when moving data actually helps vs when the transfer overhead eats the gain, and roughly how much bandwidth is available/used at that point. A lot of real bottlenecks in ML systems come from this exact decision (when NOT to move data), and its usually invisible to students since they just see final runtime numbers.

So the tool could show something like: this kernel is fast, but you're paying more in transfer time than compute time, here's why. That ties in well with your CuBLAS/CuPy idea too since those libraries make these exact tradeoffs under the hood (same reason NumPy style libs offload to GPU only when it's worth it).

@profvjreddi would love your input on this before starting working with the prototype.