Skip to content

Structure Overview

Sergi edited this page Oct 21, 2026 · 7 revisions

Here is the directory structure, followed by the complete, implementation of Deterministic Static Memory Allocator (CudaScratchpadAllocator).

This allocator pre-allocates a single contiguous block of GPU memory during initialization. During inference, it chunks out memory offsets with zero runtime cudaMalloc calls, achieving completely deterministic execution latency.

1. Implementation: The Static Memory Allocator

files: include/allocator.hpp and src/allocator.cu

This design uses a Monotonic Arena (Scratchpad) Allocator pattern. It guarantees O(1) allocation and reset times, perfect for inference loops.

  • Explicit Alignment Control: CUDA requires proper alignment (often 256 bytes for performance optimizations like coalesced memory access). This code implements custom alignment offsets manually on raw pointers.
  • RAII Design Principles: The memory space lifecycle safely adheres to strict C++ memory design practices (rule of 5/rule of zero handling explicitly via move operators).

2. CUDA Stream & Event Scheduler

Here is the implementation of the Hardware-Bound Task Scheduler (CudaStreamScheduler).

This scheduler encapsulates CUDA Streams for non-blocking asynchronous execution queues and uses CUDA Events for microsecond-accurate hardware-level profiling and dependency tracking—completely bypassing host-side thread management overhead.

files: Header Interface: include/scheduler.hpp and Implementation: src/scheduler.cu

  • cudaStreamNonBlocking Flags: By default, standard CUDA streams synchronize with the legacy default "NULL" stream. Using non-blocking flags separates your execution lanes completely, allowing true, asynchronous runtime concurrency.
  • Zero-Overhead Host Profiling: Standard CPU clocks (std::chrono) introduce PCIe latency and CPU execution bubbles when trying to profile GPU work because they force full device synchronization. Using native cudaEventElapsedTime tracks execution timestamps natively inside the hardware command queue itself.

3. Custom INT8 GEMM Kernel

Here is the implementation of a custom, high-performance INT8 Quantized GEMM (General Matrix Multiplication) CUDA Kernel. The kernel-level systems, this implementation goes beyond a naive nested loop. It implements Shared Memory Tiling (Blocking) and uses Vectorized Memory Access (int4 / char4) to maximize memory bandwidth and keep the streaming multiprocessors (SMs) completely saturated.

files: Header Interface: include/kernels.cuh and Implementation & Architecture: src/kernels.cu

We will use a tile dimension of 32. This means each block handles a 32 × 32 output region, caching pieces of data inside high-speed GPU Shared Memory to avoid hitting global memory repeatedly.

  • Memory Coalescing & Tiling: Instead of fetching single elements from high-latency global memory O(K) times, threads load contiguous blocks into high-speed local static shared caches. This yields massive memory performance gains.
  • #pragma unroll Vector Optimization: Telling the NVCC compiler to unroll the inner mathematical reduction tells the compiler to flatten branches, keeping the GPU instruction pipeline completely full.
  • Fused De-quantization: The kernel handles integer math internally and dynamically converts it back into FP32 floats alongside bias adding (C[i] = accum * alpha + bias) right before saving. This cuts out an entire extra kernel pass from memory.

4. Unified Runtime Engine Framework

Here we're creating the implementation of the Unified Runtime Engine Framework (Engine) to orchestrate these pieces together This class orchestrates all the underlying systems components we have built: it accepts weight handles, configures the execution layers, manages the lifecycle of input/output bindings, resets the CudaScratchpadAllocator instantly between runs, and pushes operations asynchronously through the CudaStreamScheduler.

files: Header Interface: include/engine.hpp and Implementation: src/engine.cpp

  • True Non-Blocking Pipelines: Notice that Engine::forward dispatches instructions directly to the cudaStream_t pipeline. The CPU thread returns immediately after queuing the work, bypassing synchronization bottlenecks.
  • Predictable, Fixed Layouts: Memory footprints are zeroed out instantly using allocator_.reset() at the start of each execution boundary. Regardless of how many frames pass through the engine, the footprint remains entirely flat.

5. Execution & Simulation test runner

This file serves as the main entry point and benchmark test suite for your engine. It generates synthetic quantized weights and input tensors, registers a multi-layer network architecture, executes a warmup run, and runs a high-precision performance benchmark. This setup will gives you to tools to measure the latency gains.

file: Implementation: src/main.cpp

Value proposition:

  • Warm-up Strategy: Hardware drivers scale GPU cores up and down to save power. Running a hidden warm-up pass before starting the loop makes sure the graphics card is locked at peak performance state, removing artificial anomalies from the benchmark data.
  • Microsecond Precision Tracking: By running the loop 1,000 times using the hardware engine events we implemented earlier, the code yields highly accurate runtime benchmarks.

6. CMake handling NVCC compilation

MakeLists.txt file to handle NVCC compilation flags and clean up the workspace configuration.

It handles cross-platform detection of the NVIDIA CUDA Toolkit, configures the proper C++ standard, and instructs the compiler to generate heavily optimized machine code tailored for NVIDIA GPUs.

File: Implementation: CMakeLists.txt

How to Compile and Run Your Project, please read here: How to run

Clone this wiki locally