mini-gpu — a from-scratch GPU in Verilog

A complete GPU built in Verilog and verified end-to-end: a fixed-function 3D graphics pipeline, an IEEE-754 float datapath, a programmable SIMT compute core that runs graphics and classical ML and transformer/LLM math, a Turing-era feature set (SFU, Tensor Core, ray-tracing cores), a KV260 FPGA bitstream, and a full Linux stack (fbdev + DRM/KMS driver + QEMU device + Weston desktop) that renders through the RTL.

---

Folder layout

GPU/
  rtl/       All synthesizable Verilog (121 files): SM, warps, FP units,
             rasterizer, texture units, pipeline, tensor core, RT cores,
             SFU, ML/LLM cores, display controller, AXI/MMIO wrappers.
             NOTE: cpu*.v / fcpu*.v here are GPU compute cores (SIMT integer,
             float, and control-flow cores) — NOT a CPU SoC.
  tb/        Testbenches (133 files), one per module / feature.
  model/     Python golden models + assemblers. Every RTL block has a
             bit-exact Python reference here; *_check.py compare RTL vs golden.
             (model/tinyllm/ = small llama reference used by the LLM core work.)
  hex/       Preloadable memories ($readmemh): programs, consts, meshes,
             textures, RT scenes, ML datasets.
  data/      Vector/golden text files used by TBs and *_check.py scripts.
  docs/      Architecture notes + annotated ISA program listings.
  build/     File lists (rtl_files.f, tb_files.f) + xsim helper .bat scripts.
  fpga/      KV260 GPU-on-FPGA flow (GPU-only, self-contained):
             gpu_axi.v (AXI4-Lite wrapper), kv260_top.v, tb_gpu_axi.v,
             kv260.xdc, gpu_rtl.f, build_gpu_kv260.tcl / build_gpu_pipe_kv260.tcl
             + trial TCLs. All paths resolve relative to the script (use ../rtl).
  linux/     Linux / QEMU co-sim software:
             kernel/        ourfb.c (/dev/fb0), ourdrm.c (DRM/KMS), build scripts
             qemu-devices/  PCI devices (our-gpu, our-fb-ram, our-mlacc, ...)
             guest/         guest userspace demos (KMS, GL, fbdev, spin, ML)
  cosim/     Verilator bridges + run scripts + captured screenshots + docs.
             This is where the "render a Linux desktop on our GPU" flow lives.

Key docs

• docs/ARCHITECTURE.md — overall design.
• docs/GPU_ARCH_UPGRADES.md — SFU, Tensor Core, dual-issue, RT cores.
• docs/GPU_README.md — prior narrative README.
• docs/logreg_instructions.txt, docs/xor_instructions.txt — annotated ISA programs.
• cosim/QEMU_BUILD.md, cosim/DRM_KMS_DRIVER.md, cosim/TIER1_WESTON.md, cosim/MESA_OPENGL.md, cosim/KERNEL_FB0_DRIVER.md — the Linux/desktop path.

---

Architecture at a glance

gpu_top  →  NSM × SM (gpu_mp)  →  NWARPS × warps  →  LANES × lanes

• Lane = one 32-bit ALU slot (a thread).
• Warp = LANES threads executing the same instruction (SIMD).
• SM (gpu_mp.v) = warp scheduler + register file + one SIMD ALU + shared memory; interleaves warps for latency hiding.
• gpu_top = NSM SMs (memory shared within an SM, private between SMs).

Compute-core variants: gpu_mp_fma/gpu_top_dn (conformant float+FMA+subnormals), gpu_warp_ml (shared-datapath programmable ML core), gpu_warp_sfu (adds SFU), gpu_warp_rt (adds ray tracing), gpu_warp_llm (adds transformer/LLM prims).

Graphics: gfx_top (fixed-function: command proc → vertex shader → cull → rasterizer → Gouraud + depth → framebuffer). Faster/other variants: gfx_tiled_top (4 parallel rasterizers), gfx_dispatch_top (16-tile dispatch), gfx_tex_top (texture units), gfx_top_pipe (pipelined, closes 100 MHz).

---

How to run

Prereqs differ per flow. On Windows: Icarus Verilog (iverilog/vvp) and/or Vivado xsim. On WSL/Linux: Verilator, and (for the desktop) a QEMU built from source. Python 3 (stdlib only) for the goldens.

NOTE on paths: TBs and RTL load .hex via relative $readmemh paths that expect the hex file in the current working directory. Copy the needed file from hex/ into your working dir first (e.g. copy hex\gwprog.hex .), or run the matching model/export_*.py which regenerates it.

1) Simulate a module (Icarus Verilog, Windows)

cd GPU
if not exist build_out mkdir build_out
iverilog -g2012 -o build_out\sim.out rtl\gpu_mp.v rtl\warp_sched.v rtl\imem.v ^
    rtl\decoder.v rtl\regfile_warps.v rtl\alu_simd.v rtl\alu_lane.v rtl\shmem.v ^
    tb\tb_gpu_mp.v
vvp build_out\sim.out

Look for SUMMARY 0 (0 mismatches) or the module's expected output.

2) Simulate everything (Vivado xsim, Windows)

cd GPU
xvlog -sv -f build\rtl_files.f
xvlog -sv -f build\tb_files.f
build\xsim_all.bat tb_gpu_mp tb_ray_tri tb_tensor_mma   REM ... any top names

build/xsim_all.bat elaborates+runs each top and greps SUMMARY/PASS/mismatch.

3) Verify an algorithmic block against its Python golden

cd GPU\model
python export_ray_tri.py          REM writes vectors/hex
python ray_tri_check.py           REM (or matching *_check.py) -> PASS / mismatch count

The Python golden uses the SAME float ops in the SAME order as the RTL, so comparisons are bit-exact.

4) Run an ML model on the compute core (golden VM — fastest to try)

cd GPU\model
python gw_logreg.py     REM logistic regression training  -> W, accuracy, cycles
python gw_xor.py        REM XOR 2-2-1 threshold MLP        -> 4/4

Other models (all bit-exact vs golden): gw_linreg, gw_softreg, gw_svm, gw_mlp_bp (backprop), gw_kmeans_full, gw_knn, gw_gnb, gw_pca, gw_dtree, gw_softmax_stable. To run the SAME program on the actual RTL, use the matching export_*.py to write the *_prog.hex + *_const.hex, then simulate tb/tb_gw_*.v.

5) Run an ML model on the RTL, fast, with Verilator (WSL)

cd GPU
bash cosim/run_ml_train.sh        # builds Vgpu_warp_ml, trains logreg bit-exact

Verilator is ~5000× faster than iverilog on this core.

6) The graphics pipeline → PNG

cd GPU\model
python gfx_check.py               REM cube through gfx_top golden -> 0 mismatches, PNG

RTL: tb/tb_gfx.v / tb/tb_gfx_mmio.v render the cube; compares vs gfx_model.py.

7) FPGA — KV260 bitstream (Vivado 2025.2, Windows)

cd GPU\fpga
"C:\AMDDesignTools\2025.2\Vivado\bin\vivado.bat" -mode batch -source build_gpu_pipe_kv260.tcl

build_gpu_kv260.tcl builds the base GPU (bitstream generated but misses timing — combinational FP). build_gpu_pipe_kv260.tcl builds the pipelined core (gfx_top_pipe), which closes timing at 100 MHz (WNS +0.1 ns). KV260 has no bare PL pins — the PS drives the GPU over AXI4-Lite, so no hand-written pin .xdc is needed for the compute path. Output lands in fpga/gpu_pipe_build/.

8) Linux desktop on the GPU (WSL + QEMU-from-source)

cd GPU/cosim
sed -i 's/\r$//' *.sh              # strip CRLF from Windows-written scripts first
bash build_verilator.sh           # builds the RTL-hosting bridges
bash run_desktop.sh vnc           # boots Linux + our DRM driver + Weston; VNC localhost:5905

Chain: Weston/Mesa(llvmpipe) → DRM/KMS ioctls → linux/kernel/ourdrm.c → PCI BAR (QEMU linux/qemu-devices/our-fb-ram.c) → Verilator bridge → real RTL framebuffer + display_ctrl.v scan-out → QEMU window. See cosim/TIER1_WESTON.md. Other entry points: run_live.sh (animated desktop), run_gl.sh (Mesa OpenGL), run_gui.sh (KMS compositor). Headless runs capture PNGs (e.g. weston_desktop.png).

---

Honest scope

• Sim/FPGA verified; the desktop path is software-rendered by llvmpipe — our GPU acts as the display/framebuffer/scan-out device (a HW Gallium driver feeding the rasterizer RTL is the big un-done piece). Desktop is 320×240 (Verilator scans ~1 px/tick; on a real FPGA it's video-rate).
• FP is FP32 (subnormal-aware _dn units are the conformant ones); SIGMOID/EXP/ GELU are polynomial/range-reduced approximations, but the golden uses the identical approximation so bit-exactness is real.
• yosys area/Fmax numbers are unmeasured here; model/area_count.py is a labeled proxy, not real synthesis area.

---

Reproducibility notes (gotchas baked in from experience)

• Windows-written files get CRLF; sed -i 's/\r$//' cosim/*.sh before running bash scripts in WSL.
• Prog/const hex are program-specific — re-run the matching export_*.py before simulating a given testbench (a stale hex gives all-zero results, not an error).
• Compile ALL of a module's RTL dependencies together in one command.
• On this GPU core the per-lane transcendentals recompute combinationally every cycle, so iverilog is slow (~tens of cyc/s). Use Verilator for long runs.