Summary
Extend VideoFrame / VideoData to optionally carry a wgpu::Texture instead of CPU pixel data, enabling zero-copy handoff between GPU pipeline stages (compositor → encoder, decoder → compositor, web renderer → compositor, etc.).
This is a foundational change that benefits the entire pipeline, not just the web renderer. The Vulkan Video encoder already notes GPU texture input as a future goal.
Parent issue: #167 (web renderer context), but this is an independent core improvement
Enables: #349 (DMA-BUF zero-copy), future HW encoder GPU texture input
Architecture
// In crates/core/src/types.rs (or similar)
pub enum VideoData {
/// CPU-side pixel buffer (current behavior).
Cpu(PooledVideoData),
/// GPU-resident texture — zero-copy between GPU stages.
Gpu(GpuVideoData),
}
pub struct GpuVideoData {
pub texture: wgpu::Texture,
pub view: wgpu::TextureView,
pub width: u32,
pub height: u32,
pub format: wgpu::TextureFormat,
// Device reference for readback if needed
pub device: Arc<wgpu::Device>,
pub queue: Arc<wgpu::Queue>,
}
impl VideoData {
/// Convert GPU data to CPU data (triggers readback).
/// This is the backward-compatibility escape hatch.
pub async fn to_cpu(&self) -> PooledVideoData { ... }
/// Returns true if this is GPU-resident data.
pub fn is_gpu(&self) -> bool { ... }
}
Implementation Steps
- Define
VideoData enum in crates/core/ with Cpu and Gpu variants
- Update
VideoFrame to use VideoData instead of raw PooledVideoData
- Add
to_cpu() readback — nodes that only handle CPU data call this (backward compatible)
- Update compositor output — when the downstream node accepts GPU data, skip
readback_rgba() and output VideoData::Gpu directly
- Update all existing nodes — most just call
.to_cpu() if they receive GPU data (no behavior change)
- Benchmark — measure compositor → encoder pipeline with GPU passthrough vs. current CPU readback
Acceptance Criteria
Complexity
Effort: L (1–2 weeks)
Touches core types used throughout the pipeline. Requires careful attention to wgpu device/queue lifetime management and backward compatibility.
Design Considerations
- Device sharing: The compositor's
wgpu::Device must be accessible to downstream nodes that want to use the GPU texture. This may require passing a device reference through the pipeline context or storing it in GpuVideoData.
- Texture lifetime: wgpu textures are bound to their device. The texture must remain valid until the downstream node has consumed it. Consider using
Arc<wgpu::Texture> or a pool.
- Format negotiation: Nodes need a way to advertise whether they accept GPU data. This could be a new pin capability or an upstream hint.
- Pool integration:
PooledVideoData has an existing pool mechanism. GpuVideoData may need a similar texture pool to avoid per-frame allocation.
Notes
- This issue is independent of the Servo plugin but was identified during the web renderer roadmap planning.
- The Vulkan Video encoder (
vulkan_video.rs) is a natural first consumer — it already operates on the same Vulkan device as wgpu.
- Future HW encoder GPU texture input (NVENC via
cuImportExternalMemory, VAAPI via DMA-BUF) would also consume VideoData::Gpu.
Summary
Extend
VideoFrame/VideoDatato optionally carry awgpu::Textureinstead of CPU pixel data, enabling zero-copy handoff between GPU pipeline stages (compositor → encoder, decoder → compositor, web renderer → compositor, etc.).This is a foundational change that benefits the entire pipeline, not just the web renderer. The Vulkan Video encoder already notes GPU texture input as a future goal.
Parent issue: #167 (web renderer context), but this is an independent core improvement
Enables: #349 (DMA-BUF zero-copy), future HW encoder GPU texture input
Architecture
Implementation Steps
VideoDataenum incrates/core/withCpuandGpuvariantsVideoFrameto useVideoDatainstead of rawPooledVideoDatato_cpu()readback — nodes that only handle CPU data call this (backward compatible)readback_rgba()and outputVideoData::Gpudirectly.to_cpu()if they receive GPU data (no behavior change)Acceptance Criteria
VideoDataenum withCpuandGpuvariants incrates/core/GpuVideoDatacarries:wgpu::Texture,TextureView, width, height, format, device/queue refsVideoFrameusesVideoDatainstead of rawPooledVideoDataVideoData::Gpu(skipreadback_rgba()) when downstream accepts it.to_cpu()method for backward compatibility (triggers GPU readback on demand)VideoDataenum,to_cpu()conversionComplexity
Effort: L (1–2 weeks)
Touches core types used throughout the pipeline. Requires careful attention to wgpu device/queue lifetime management and backward compatibility.
Design Considerations
wgpu::Devicemust be accessible to downstream nodes that want to use the GPU texture. This may require passing a device reference through the pipeline context or storing it inGpuVideoData.Arc<wgpu::Texture>or a pool.PooledVideoDatahas an existing pool mechanism.GpuVideoDatamay need a similar texture pool to avoid per-frame allocation.Notes
vulkan_video.rs) is a natural first consumer — it already operates on the same Vulkan device as wgpu.cuImportExternalMemory, VAAPI via DMA-BUF) would also consumeVideoData::Gpu.