Repository navigation
llama2-cu.rs: A Rust Port of llama2.c with cuTile GPU Kernels #1736
SangHyeok-Yoon
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
After hearing about NVIDIA's CUDA support for Rust, I wanted to try it out and started looking for a small learning project. I ended up building llama2-cu.rs, a Rust port of llama2.c with a cuTile GPU backend. I implemented it myself without AI coding agents as a learning exercise, using AI only to help find documentation.
I really enjoyed how cuTile Rust brings Rust's type system and syntax into GPU programming. For someone familiar with Rust, CUDA C++, and NumPy, much of the API should feel familiar. Building this project was a useful way to learn both LLM inference and tile-based GPU programming.
A few things slowed me down along the way:
Tile dimensions: I initially missed the requirement that each tile dimension must be a power of two, which led to substantial rework. The requirement is documented in the NVIDIA Tile IR specification, but making it more prominent in the Rust getting-started guide and
TileAPI documentation would have helped.Control flow: The familiar CUDA pattern of returning early from an
ifstatement for bounds checking was unsupported. The DSL reference explains the restriction, but a more visible guide to supported control flow, with examples of how to adapt common CUDA patterns, would have saved me time.Mapped partition traversal: I used
MappedPartitionMutfor targeted KV-cache updates. The API documentation explains index ranges and disjointness, but I found it harder to picture the traversal itself. A worked example showing which logical tile indices each physical tile block visits, and howmap_shapeandnum_tile_blocksaffect that traversal, would make this much easier to understand.JIT warm-up: I also had to learn how integer scalar arguments affect specialization: their divisibility can produce five buckets per argument, so multiple arguments can produce multiple combinations. Although
.compile()and.compile_on()already support precompiling individual specializations, a higher-level way to enumerate and prewarm all expected specializations for a workload would make first-use latency easier to manage.Overall, I had a positive experience, and I hope the CUDA Rust ecosystem continues to grow.
All reactions