worldmodel.c

_{Interactive world generation at 30 FPS — click the preview to watch the full video.}

An experimental standalone C/CUDA runtime for Waypoint 1.5.

The runtime uses CUTLASS for GEMM, attention, and convolution kernels. It links only the CUDA runtime, without cuBLAS or cuDNN. CUDA kernels are tested against PyTorch references. The former CUDA/Vulkan implementation is preserved on the legacy-cuda-vulkan branch.

Architecture

The CUDA backend deliberately follows a small, direct execution model instead of a generic graph or tensor framework:

• src/world_cuda.cu owns model state, weights, KV caches, scheduling, and the explicit Transformer forward pass. It contains no CUDA kernels or launch syntax.
• src/world_cuda_ops.cuh and src/world_cuda_ops.cu provide typed, backend-private Transformer operators over raw device pointers.
• src/world_cuda_vae.cuh and src/world_cuda_vae.cu keep VAE state and the explicit encoder/decoder execution order behind a small opaque interface.
• src/world_cuda_vae_ops.cuh and src/world_cuda_vae_ops.cu contain the typed VAE layout, convolution, streaming, and pixel-shuffle operators.
• src/world_weights.h and src/world_weights.c contain the host-side safetensors/VAE loading shared by the production app and diagnostic tools.
• tools/tests/ links its parity extensions directly against the production Transformer and VAE operator translation units; kernels that are not production operators are isolated in the experimental test extension.

There is no generic tensor object, operation enum, graph executor, or backend vtable. This keeps the network flow readable while leaving each CUDA operator easy to profile and replace independently.

Performance

Measured on an NVIDIA GeForce RTX 4090 D (48 GiB), NVIDIA driver 610.62, using Windows Release builds. Every run uses all 24 layers, 4 denoising steps, and --cache-window 8. A single resident runtime generates 12 consecutive chunks; RGB FPS is the average of the final four full-cache chunks, with four RGB frames emitted per chunk. Peak VRAM is the maximum nvidia-smi memory.used increase over the pre-launch baseline, sampled every 100 ms.

Model	Output	Backend	RGB FPS	Peak VRAM
Waypoint-1.5-1B	1024x512	CUDA	32.3	14.0 GiB
Waypoint-1.5-1B-360P	512x256	CUDA	102.4	11.2 GiB

These figures measure backend throughput rather than the interactive window's display refresh cap. Results vary with drivers, clocks, and background GPU usage.

Requirements

• CMake 3.24+
• CUDA Toolkit
• Python 3 plus PyTorch, pytest, NumPy, safetensors, and PyYAML for parity tests
• Visual Studio 2022 Build Tools on Windows

Setup

git clone --recurse-submodules https://github.com/Wimacs/worldmodel.c.git
cd worldmodel.c
python -m pip install -U huggingface_hub
hf download Overworld/Waypoint-1.5-1B --local-dir Waypoint-1.5-1B
hf download Overworld/Waypoint-1.5-1B-360P --local-dir Waypoint-1.5-1B-360P

Build

Linux CUDA:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Windows CUDA, from an x64 Visual Studio developer shell:

cmake -S . -B build-win -G "Visual Studio 17 2022" -A x64 ^
  -DCMAKE_CUDA_ARCHITECTURES=89
cmake --build build-win --config Release --parallel

Run

Interactive CUDA runtime:

./build/worldmodel_raylib ./Waypoint-1.5-1B/model.safetensors ./image.png

Drop another image onto the window to reset the world from that frame.

Headless smoke test:

./build/worldmodel_raylib --model-dir ./Waypoint-1.5-1B \
  --steps 4 --cache-window 8 --headless-smoke

Use WASD and the mouse in the interactive window. Run an executable without arguments to see its available options.

Experimental CUDA W8A8

The resident CUDA runtime has an opt-in row/channel-wise W8A8 path for WorldDiT QKV, attention output, controller, and MLP linears:

$env:WORLD_W8A8 = '1'
.\build-win\Release\worldmodel_raylib.exe --model-dir .\Waypoint-1.5-1B `
  --steps 4 --cache-window 8 --headless-smoke

This is an experimental post-training quantization path, not a TurboDiffusion-equivalent kernel, and long autoregressive rollouts can diverge from the FP32/FP16 trajectory. The implementation, failed first attempt, benchmarks, quality checks, memory accounting, controls, and exact TurboDiffusion comparison are recorded in docs/cuda_w8a8_optimization.md.

Tests

ctest --test-dir build --output-on-failure
python -m pytest tools/tests

CUDA profilers and standalone diagnostics are built with WORLD_BUILD_TOOLS=ON, which is the default:

./build/tools/worldmodel_cuda_gemm_probe --tensorop
./build/tools/worldmodel_w8a8_probe --case quick

The tools/ directory contains tests, profilers, and reference generators. Implementation and optimization notes are in docs/cuda_cutlass_kernels.md and docs/cuda_w8a8_optimization.md.