Prefer NVENC for WebRTC H264 encoding

[balthazur]

Summary

• Prefer NVIDIA h264_nvenc for aiortc H.264 WebRTC encoding when available.
• Fall back to the existing libx264 encoder when NVENC is unavailable or fails.
• Add scoped [WEBRTC_NVENC] logs for patch activation, encoder selection, fallback, and resolution-triggered encoder recreation.

Why

Modal GPU workers run on NVIDIA GPUs, so H.264 hardware encoding can reduce server-side encode latency without changing the negotiated browser codec. H.264 stays the WebRTC-compatible path; this does not introduce H.265/HEVC or bitrate/congestion-control changes.

Testing

• python3 -m py_compile inference/core/interfaces/webrtc_worker/h264_nvenc.py inference/core/interfaces/webrtc_worker/webrtc.py inference/core/interfaces/stream_manager/manager_app/webrtc.py
• Local smoke test with repo venv and aiortc 1.14.0: confirmed h264_nvenc fallback to libx264 on a non-NVIDIA Mac still produces H.264 packets.
• git diff --check

Staging benchmark

Encoder	Hardware	1080p encode time	Relative speed	Notes
libx264	CPU	~14-17 ms/frame steady-state	baseline	Works everywhere, but uses CPU and can become expensive at higher resolution/bitrate.
h264_nvenc	NVIDIA GPU	avg ~2.6 ms/frame, min ~2.1 ms, max ~3.1 ms	~5-6x faster	Uses the dedicated NVIDIA video encoder on T4/L4/L40S. Frees CPU and gives much more headroom for 1080p WebRTC output.

In staging, software libx264 encoding took roughly 14-17 ms per 1080p frame. With h264_nvenc, sampled encode time dropped to about 2.6 ms per frame, roughly a 5-6x improvement. This does not by itself solve all WebRTC congestion/ramp-up behavior, but it removes server-side H.264 encoding as a major bottleneck on NVIDIA GPU Modal workers.

---

Note

Medium Risk
Medium risk because it monkey-patches aiortc.codecs.h264.H264Encoder._encode_frame, which can affect all WebRTC H.264 sessions and may behave differently across aiortc/PyAV/FFmpeg builds and GPU availability.

Overview
WebRTC H.264 encoding now prefers NVIDIA hardware encode when available. A new prefer_h264_nvenc_encoder() patch overrides aiortc’s H264Encoder._encode_frame to try h264_nvenc first (with several option presets) and transparently fall back to the original libx264 path if NVENC can’t be opened or errors during encode.

The patch also handles runtime changes by recreating/resetting the encoder on resolution changes, attempting bitrate updates on significant target bitrate shifts, and emitting scoped [WEBRTC_NVENC] logs for selection and fallback. Both the stream manager and worker WebRTC entrypoints now call prefer_h264_nvenc_encoder() at import time so the behavior applies to all H.264 WebRTC outputs.

^{Reviewed by Cursor Bugbot for commit 2ed2809. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6a9096f. Configure here.}