cgpu

Reusable CPU/GPU compute execution layer for Rust.

cgpu provides a unified interface for running compute jobs that can execute on the GPU via wgpu or fall back to CPU execution. It handles batch planning, memory management, shader compilation, and telemetry automatically, letting you focus on defining your domain logic.

Features

• Unified Execution Model: Define work once, run on GPU or CPU
• Automatic Batch Planning: Byte-based batching optimized for VRAM constraints
• GPU Fallback: Graceful degradation to CPU when GPU execution fails
• Memory Management: Automatic VRAM probing and budget-aware scheduling
• Telemetry: Built-in performance tracking and reporting
• Configurable: JSON configuration or programmatic setup
• Feature Flags: Optional GPU support for CPU-only deployments

Quick Start

Add to your Cargo.toml:

[dependencies]
cgpu = "version"

CPU-Only Example

use cgpu::*;

#[derive(Clone)]
struct Item(u32);

impl WorkItem for Item {
    fn bytes_in(&self) -> usize { 4 }
    fn bytes_out(&self) -> usize { 4 }
}

struct DoubleJob {
    items: Vec<Item>,
}

impl GpuJob for DoubleJob {
    type Item = Item;
    type Output = u32;

    fn label(&self) -> &'static str { "double" }
    fn items(&self) -> &[Self::Item] { &self.items }

    fn encode_batch(&self, _span: &BatchSpan) -> Result<EncodedBatch, JobError> {
        // Return error to force CPU fallback
        Err(JobError::EncodingFailed("Using CPU path".into()))
    }

    fn decode_batch(&self, _bytes: &[u8], _span: &BatchSpan) -> Result<Vec<u32>, JobError> {
        Ok(Vec::new())
    }

    fn cpu_fallback(&self, span: &BatchSpan) -> Result<Vec<u32>, JobError> {
        Ok(self.items[span.range()].iter().map(|item| item.0 * 2).collect())
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut config = ExecutorConfig::default();
    config.execution_mode = ExecutionMode::PreferCpu;

    let mut executor = GpuExecutor::with_config(config)?;
    let job = DoubleJob {
        items: vec![Item(1), Item(2), Item(3)],
    };
    
    let report = executor.execute(&job)?;
    assert_eq!(report.outputs, vec![2, 4, 6]);
    
    println!("Execution path: {:?}", report.execution_path);
    println!("Total time: {:.2}ms", report.total_ms);
    
    Ok(())
}

GPU-Enabled Example

use cgpu::*;

struct GpuDoubleJob {
    items: Vec<u32>,
}

impl WorkItem for u32 {
    fn bytes_in(&self) -> usize { 4 }
    fn bytes_out(&self) -> usize { 4 }
}

impl GpuJob for GpuDoubleJob {
    type Item = u32;
    type Output = u32;

    fn label(&self) -> &'static str { "gpu-double" }
    fn items(&self) -> &[Self::Item] { &self.items }

    fn encode_batch(&self, span: &BatchSpan) -> Result<EncodedBatch, JobError> {
        let items = &self.items[span.range()];
        let input_bytes: Vec<u8> = items.iter()
            .flat_map(|x| x.to_ne_bytes())
            .collect();
        let output_size = items.len() * 4;

        let wgsl = r#"
@group(0) @binding(0)
var<storage, read> input: array<u32>;

@group(0) @binding(1)
var<storage, read_write> output: array<u32>;

@compute @workgroup_size(64, 1, 1)
fn main(@builtin(global_invocation_id) global_id: vec3<u32>) {
    if (global_id.x < arrayLength(&input)) {
        output[global_id.x] = input[global_id.x] * 2u;
    }
}
"#;

        let dispatch_x = ((items.len() as u32 + 63) / 64).max(1);

        Ok(EncodedBatch::new()
            .with_label("double-batch")
            .with_wgsl("double-shader", wgsl)
            .add_buffer(EncodedBuffer::storage_read_only(0, input_bytes))
            .add_buffer(
                EncodedBuffer::storage_read_write(1, BufferRole::Output, vec![0u8; output_size])
                    .readback(true)
            )
            .with_dispatch(dispatch_x, 1, 1))
    }

    fn decode_batch(&self, bytes: &[u8], span: &BatchSpan) -> Result<Vec<u32>, JobError> {
        let count = span.len();
        let mut results = Vec::with_capacity(count);
        
        for i in 0..count {
            let offset = i * 4;
            if offset + 4 <= bytes.len() {
                let mut buf = [0u8; 4];
                buf.copy_from_slice(&bytes[offset..offset + 4]);
                results.push(u32::from_ne_bytes(buf));
            }
        }
        
        Ok(results)
    }

    fn cpu_fallback(&self, span: &BatchSpan) -> Result<Vec<u32>, JobError> {
        Ok(self.items[span.range()].iter().map(|x| x * 2).collect())
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut executor = GpuExecutor::new()?;
    
    let job = GpuDoubleJob {
        items: (0..1000).collect(),
    };
    
    let report = executor.execute(&job)?;
    println!("Processed {} items via {:?}", 
             report.outputs.len(), 
             report.execution_path);
    
    Ok(())
}

Configuration

JSON Configuration

Create a cgpu.config.json file in your project root:

{
  "vram_override": null,
  "memory_fill_ratio": 0.9,
  "min_batch_bytes": 4096,
  "max_batch_bytes": 268435456,
  "execution_mode": "auto",
  "cpu_threads": null,
  "parallel_fallback": true,
  "shader_cache": true,
  "shader_optimization": "performance",
  "enable_telemetry": true,
  "telemetry_sink": "log"
}

Config Fields

Field	Default	Meaning
vram_override	null	Optional memory budget override in bytes. Use this when you want deterministic planning or when automatic VRAM probing is not trustworthy on the target machine. null leaves the executor on its normal budget path.
memory_fill_ratio	0.9	Fraction of the available memory budget that batching is allowed to use. Keep this below 1.0 so command buffers, staging buffers, desktop compositors, and other GPU users still have headroom. The practical GPU budget path expects the safe range around 0.90 to 0.95.
min_batch_bytes	4096	Minimum estimated batch size before ExecutionMode::Auto considers GPU execution worthwhile. Smaller batches usually stay on CPU because upload, dispatch, and readback overhead can dominate the work.
max_batch_bytes	268435456	Hard upper bound for one planned batch. This protects the executor from creating oversized buffers even if the reported or overridden memory budget is large.
execution_mode	"auto"	Selects the execution policy. "auto" chooses GPU only when available and the batch is large enough, "prefer_gpu" tries GPU first and falls back to CPU, "prefer_cpu" always uses CPU fallback, and "gpu_only" returns an error instead of falling back.
cpu_threads	null	Optional CPU worker count for CPU execution and fallback policy. null means use the platform/default parallelism. The current generic executor keeps this value in config so fallback scheduling can be tuned without changing the public API.
parallel_fallback	true	Allows CPU fallback work to be parallelized by the fallback policy. Keep it enabled for throughput-oriented jobs; disable it when a caller needs strictly serial fallback behavior.
shader_cache	true	Enables the shader/resource cache policy. The resource cache is active today; shader and pipeline reuse are represented by this knob so generated shader workflows can opt into caching as that layer grows.
shader_optimization	"performance"	Declares the shader generation preference. "debug" favors fast iteration and readable generated WGSL, "performance" is the default throughput profile, and "size" is for smaller generated shader bodies or binary pressure.
enable_telemetry	true	Turns phase timing and byte accounting on or off. Disable it when the caller wants the smallest possible bookkeeping overhead.
telemetry_sink	"log"	Chooses where telemetry is sent. JSON config currently uses "log"; programmatic config can also use callback or channel buffer sinks.

The executor searches for this file in:

1. Current working directory
2. Crate directory
3. Workspace parent directory

Programmatic Configuration

let mut config = ExecutorConfig::default();
config.memory_fill_ratio = 0.85;
config.execution_mode = ExecutionMode::PreferGpu;
config.enable_telemetry = false;

let executor = GpuExecutor::with_config(config)?;

Environment Variables

• REV_GPU_AVAILABLE_BYTES: Override detected VRAM size

Architecture

Core Concepts

WorkItem: Describes a unit of work with byte size estimates

pub trait WorkItem: Send + Sync {
    fn bytes_in(&self) -> usize;
    fn bytes_out(&self) -> usize;
}

GpuJob: Defines how to encode/decode work for GPU execution

pub trait GpuJob: Send + Sync {
    type Item: WorkItem;
    type Output: Send;
    
    fn encode_batch(&self, span: &BatchSpan) -> Result<EncodedBatch, JobError>;
    fn decode_batch(&self, bytes: &[u8], span: &BatchSpan) -> Result<Vec<Self::Output>, JobError>;
    fn cpu_fallback(&self, span: &BatchSpan) -> Result<Vec<Self::Output>, JobError>;
}

GpuExecutor: Manages execution, batching, and resource allocation

Execution Flow

1. Batch Planning: Items are grouped into batches based on byte size and VRAM budget
2. Encoding: Each batch is encoded into WGSL shaders and storage buffers
3. Execution: Batches run on GPU (with optional CPU fallback) or CPU
4. Decoding: GPU output bytes are converted back to domain types
5. Reporting: Structured JobReport with timing and execution path info

Execution Paths

• GpuOnly: All batches executed on GPU
• GpuWithFallback: Some batches fell back to CPU after GPU failure
• CpuOnly: All batches executed on CPU
• Mixed: Combination of GPU and CPU execution

Feature Flags

Feature	Description	Default
gpu	Enable GPU support via wgpu	✓
telemetry	Enable performance telemetry	✓
shader	Enable shader helpers	✓
shader-gen	Enable shader generation utilities	(via shader)
async	Enable async support with tokio
vulkan	Vulkan backend support
dx12	DirectX 12 backend support
metal	Metal backend support (macOS/iOS)
webgl	WebGL backend support

CPU-Only Build

cargo build --no-default-features

Full GPU Build

cargo build

Testing

# Format check
cargo fmt

# Type check
cargo check

# CPU-only tests
cargo test -p cgpu --no-default-features

# Full tests (requires GPU)
cargo test -p cgpu

Performance Considerations

• Batch Size: Larger batches reduce overhead but increase memory usage
• VRAM Budget: Configure memory_fill_ratio to leave headroom for other GPU resources
• Shader Caching: Keep shader_cache enabled for resource reuse and generated shader workflows
• Telemetry: Disable in production for minimal overhead

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

Contributing

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.