Editor windows playback performance

[richiemcilroy]

Significantly improve editor playback performance on Windows by enabling native MediaFoundation hardware decoding with zero-copy D3D11 textures, optimizing data transfer with NV12 output, and ensuring high-performance GPU selection.

---

 

Greptile Summary

This PR significantly improves editor playback performance on Windows through three coordinated optimizations: (1) enables DirectX 12 backend preference across all GPU contexts for better hardware acceleration, (2) switches Windows video decoding to prefer native MediaFoundation with hardware acceleration and automatic FFmpeg fallback, and (3) implements D3D11 zero-copy texture sharing for NV12 frames to eliminate expensive memory transfers.

Key changes:

• All GPU initialization now prefers DX12 backend on Windows (wgpu::Backends::DX12 | wgpu::Backends::VULKAN)
• Frontend WebGPU requests high-performance GPU adapter
• Windows decoder switched from FFmpeg-first to MediaFoundation-first with fallback logic
• Editor renderer switched to NV12 output pipeline with RGBA fallback
• Display layer implements D3D11 zero-copy path using shared texture handles for NV12 frames
• Pipeline always flushes to prevent frame loss in the editor

Confidence Score: 4/5

• This PR is safe to merge with careful testing on Windows systems
• The changes are well-structured with proper platform-specific gating and comprehensive fallback paths. The decoder fallback logic ensures robustness when MediaFoundation fails. However, the performance impact should be verified on different Windows GPU configurations, and the .clone() of segment_frames in the editor loop may have performance implications worth investigating.
• Pay close attention to crates/editor/src/editor.rs for the frame cloning overhead and crates/rendering/src/decoder/mod.rs for the decoder fallback logic

Important Files Changed

Filename	Overview
apps/desktop/src/utils/webgpu-renderer.ts	Requests high-performance GPU adapter for WebGPU in editor frontend, improving rendering performance
crates/editor/src/editor.rs	Switches to NV12 rendering pipeline with RGBA fallback; always flushes NV12 pipeline after render to avoid frame loss
crates/rendering/src/decoder/mod.rs	Switches Windows to prefer MediaFoundation hardware decoder with automatic FFmpeg fallback, improving performance and reliability
crates/rendering/src/layers/display.rs	Adds D3D11 zero-copy NV12 texture path for Windows, falling back to memory copy paths when unavailable

Flowchart

flowchart TD
    A[Editor Render Frame] --> B{Try NV12 Pipeline}
    B -->|Success| C{Has D3D11 Handles?}
    C -->|Yes| D[Zero-Copy D3D11 Texture]
    C -->|No| E[CPU Memory Copy]
    D --> F[Output NV12 Frame]
    E --> F
    B -->|Failure| G[Fallback to RGBA]
    G --> H[Output RGBA Frame]
    
    I[Windows Decoder Init] --> J{Try MediaFoundation}
    J -->|Success| K[HW Accelerated MF]
    J -->|Fail| L[Fallback to FFmpeg]
    L --> M{FFmpeg Init}
    M -->|Success| N[FFmpeg Decoder]
    M -->|Fail| O[Error: Both Failed]
    
    P[GPU Context Init] --> Q{Platform?}
    Q -->|Windows| R[DX12 + Vulkan]
    Q -->|Other| S[Default Backends]
    R --> T[High-Performance Adapter]
    S --> T

Loading

_{Last reviewed commit: 62381d0}

[cursor]

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
_{Learn more about Cursor Agents}

[greptile-apps]

_{12 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

.clone() on segment_frames may be expensive if DecodedFrame contains large buffers. Verify whether this clone is necessary or if the data can be moved/borrowed instead.

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/editor/src/editor.rs
Line: 152:152

Comment:
`.clone()` on `segment_frames` may be expensive if `DecodedFrame` contains large buffers. Verify whether this clone is necessary or if the data can be moved/borrowed instead.

How can I resolve this? If you propose a fix, please make it concise.

[tembo]

flush_pipeline_nv12() can also yield Some(Err(_)); logging that case helps avoid silently dropping frames.

Suggested change

	if let Some(Ok(current_frame)) = frame_renderer.flush_pipeline_nv12().await {
	match frame_renderer.flush_pipeline_nv12().await {
	Some(Ok(current_frame)) => {
	(self.frame_cb)(EditorFrameOutput::Nv12(current_frame));
	}
	Some(Err(e)) => {
	tracing::warn!(error = %e, "Failed to flush NV12 pipeline frame");
	}
	None => {}
	}

[tembo]

These MediaFoundation error strings drop useful context (name/path) that the FFmpeg errors include; adding them makes debugging much easier.

Suggested change

	Ok(Ok(Err(e))) => Err(format!("MediaFoundation initialization failed: {e}")),
	Ok(Ok(Err(e))) => Err(format!(
	"'{name}' MediaFoundation initialization failed: {e} ({path_display})"
	)),
	Ok(Err(e)) => Err(format!(
	"'{name}' MediaFoundation channel closed: {e} ({path_display})"
	)),

[tembo]

Since this function currently hard-todo!()s, the Windows-specific InstanceDescriptor block below is dead code and adds noise/duplication.

Suggested change

	#[cfg(target_os = "windows")]
	let instance = wgpu::Instance::new(&wgpu::InstanceDescriptor::default());

[tembo]

Same idea here: try high-performance, but fall back to default adapter request so initialization still works if the preference is unsupported.

Suggested change

	const adapter = await navigator.gpu.requestAdapter({
	powerPreference: "high-performance",
	});
	if (!adapter) {
	throw new Error("No WebGPU adapter available");
	}
	let adapter = await navigator.gpu
	.requestAdapter({
	powerPreference: "high-performance",
	})
	.catch(() => null);
	adapter ??= await navigator.gpu.requestAdapter();
	if (!adapter) {
	throw new Error("No WebGPU adapter available");
	}

[tembo]

Right now conversion failures get silently collapsed to false, which makes it hard to debug why we’re falling back (especially on Windows where multiple paths exist). Consider logging the error at debug to keep it low-noise.

Suggested change

	match convert_result {
	Ok(_) => {
	if self.yuv_converter.output_texture().is_some() {
	self.pending_copy = Some(PendingTextureCopy {
	width: actual_width,
	height: actual_height,
	dst_texture_index: next_texture,
	});
	true
	} else {
	false
	}
	}
	Err(_) => false,
	}
	match convert_result {
	Ok(_) => {
	if self.yuv_converter.output_texture().is_some() {
	self.pending_copy = Some(PendingTextureCopy {
	width: actual_width,
	height: actual_height,
	dst_texture_index: next_texture,
	});
	true
	} else {
	false
	}
	}
	Err(e) => {
	tracing::debug!(error = %e, "NV12 GPU conversion failed");
	false
	}
	}

[tembo]

This option can throw or be ignored on some WebGPU implementations; falling back to a plain requestAdapter() keeps the capability check from returning false for supported devices.

Suggested change

	const adapter = await navigator.gpu.requestAdapter({
	powerPreference: "high-performance",
	});
	const adapter = await navigator.gpu
	.requestAdapter({
	powerPreference: "high-performance",
	})
	.catch(() => navigator.gpu.requestAdapter());