Support for Full NVIDIA Hardware Acceleration (NVENC/CUDA)

[ravirajsinh45]

Moved from issue #37 by @deezid — converting to a discussion to gauge demand before scheduling. If you need NVENC/CUDA transcoding, please 👍 this post or leave a comment describing your workflow and GPU.

---

The ask

Currently, processing high-resolution masters (4K/6K) puts heavy load on the CPU. While the system may be using some hardware encoding, it appears to be a "half-CPU" implementation. For professional studio workflows using NVIDIA cards (like my 1080 Ti), we need full hardware offloading.

Steps to reproduce
Upload a video clip. Wait forever for it to finish since decoding and encoding runs on CPU.

Expected behavior
To fully utilize NVIDIA GPUs, the FFmpeg command needs to include the hardware accelerator flag before the input and the NVENC codec for the output:

```
ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input_master_6K.mov \
-vf "scale_cuda=1920:1080" \
-c:v h264_nvenc -preset p4 -tune hq -b:v 8M \
-c:a aac -b:a 192k \
output_preview_1080p.mp4
```

Requirements

• Ability to use NVIDIA hardware decoder, splitter and scaler.
• Ability to toggle NVIDIA Hardware Acceleration in the settings/env.
• Mapping of `/dev/nvidia*` devices into the worker container.
• Support for 10-bit HEVC decoding via CUDA to prevent CPU bottlenecks on modern cinema camera footage.

Reporter's workaround

Because FreeFrame does not natively support NVENC or CUDA, the reporter is currently using a Python wrapper to intercept the app's FFmpeg calls. The wrapper replaces `libx264` with `h264_nvenc` and translates CRF values to bitrates. Full wrapper script is in the original issue #37.

Limitation: Even with the wrapper, `-threads 0` is required to scale on CPU because the internal logic doesn't support the CUDA hardware context needed for `scale_cuda`. This still results in heavy CPU usage during 6K transcodes — but the whole process is significantly faster than without.

Why this is parked (for now)

This is a sizable feature — it touches the transcoder package, Docker image, compose file, and deployment docs. Rough scope:

• A new CUDA-based worker image variant (`nvidia/cuda:12.x-runtime` base)
• A separate `docker-compose.gpu.yml` override with NVIDIA device passthrough
• Backend abstraction in `packages/transcoder` so CPU and NVENC are swappable (and VAAPI can slot in later for AMD/Intel users)
• NVENC command builder with CRF→bitrate translation and `scale_cuda` filter graph
• Runtime probe + graceful fallback to CPU when CUDA is unavailable
• Handling the NVENC session limit on consumer cards (1080 Ti = 3 concurrent encodes)
• Deployment docs covering driver version, nvidia-container-toolkit, and verification steps

Before committing engineering time, we want to see how many users actually need it. If this affects your workflow, please upvote this discussion or reply with your GPU and typical source format (4K ProRes, 6K R3D, 10-bit HEVC, etc).

[deezid]

For me it's dealing with 6K Opengate footage, which without NVENC takes hours to ingest while with NVENC it's reduced to a few minutes

But even regular 4K renders get ingested many times faster with NVENC.

[deezid]

Handling the NVENC session limit on consumer cards (1080 Ti = 3 concurrent encodes)

I use nvidia patch to overcome this:
https://github.com/keylase/nvidia-patch

[deezid]

Hey, I figured out the bottleneck for 6K Open Gate footage. The hardcoded CPU filter_complex forces uncompressed frames across the PCIe bus, tanking performance. Projects like Unmanic (which is also Python-based) and Tdarr have open-source routing engines that dynamically swap to scale_cuda to keep the whole pipeline in VRAM. Could we look into adapting a similar hardware-path generator for Freeframe?

Right now with the ffmpeg wrapper script (which already increases speed like 10x with 6k footage)

• GPU decodes a 6K frame
• That uncompressed 24-megapixel frame is forced into the PCIe bus traffic jam to get to the system RAM.
• The CPU sweats to perform a split=3 and mathematically resize 24 million pixels down to three smaller sizes.
• Three separate, uncompressed frames are shoved back through the PCIe bus traffic jam to the GPU
• The GPU encodes them

WIth a proper pipeline skipping slow software scaling and moving things around on the bus, things could be even another 2-3x faster.

The wrapper-script I'm using right now:

#!/bin/bash
# --- 55Media NVIDIA GPU Wrapper (v39 - with Logging) ---

# This creates a highly detailed log file inside the worker container
export FFREPORT="file=/tmp/ffreport.log:level=48"

# We will test the 'nvdec' auto-download flag one more time
final_args=("-hwaccel" "nvdec")

for arg in "$@"; do
    if [[ "$arg" == "libx264" ]]; then
        final_args+=("h264_nvenc" "-preset" "p4" "-pix_fmt" "yuv420p")
    elif [[ "$arg" == "-crf" ]]; then
        final_args+=("-rc" "vbr" "-cq")
    else
        final_args+=("$arg")
    fi
done

exec /usr/bin/ffmpeg.original "${final_args[@]}"

[deezid]

This version uses Intel decoding and NVIDIA encoding to create less of a bottleneck.
Achieving 2x encoding speed with this using 4K H265 Main 10 footage input, output in H265 Main10 as well with visibly better video quality.

#!/bin/bash
# ---  INTEL - NVIDIA GPU Wrapper (v42 - High-Motion HEVC Fix) ---

final_args=("-hwaccel" "vaapi" "-hwaccel_device" "/dev/dri/renderD128" "-hwaccel_output_format" "yuv420p")

stream_index=0
skip_next=false

for arg in "$@"; do
    if [ "$skip_next" = true ]; then
        skip_next=false
        continue
    fi

    if [[ "$arg" == "libx264" ]]; then
        if [ "$stream_index" -eq 0 ]; then
            BITRATE="11M"
            MAXRATE="15M"
        elif [ "$stream_index" -eq 1 ]; then
            BITRATE="5M"
            MAXRATE="8M"
        else
            BITRATE="800k"
            MAXRATE="1200k"
        fi

        # The "P6 High Motion" block
        # Added lookahead, spatial-aq, and multipass for stability
        final_args+=("hevc_nvenc" "-preset" "p6" "-pix_fmt" "p010le" "-profile:v" "main10" "-b:v" "$BITRATE" "-maxrate" "$MAXRATE" "-bufsize" "20M" "-rc-lookahead" "32" "-spatial-aq" "1" "-multipass" "fullres")
        ((stream_index++))

    elif [[ "$arg" == "-crf" ]]; then
        skip_next=true
        continue
    elif [[ "$arg" == "-preset" ]]; then
        skip_next=true
        continue
    else
        final_args+=("$arg")
    fi
done

exec /usr/bin/ffmpeg.original "${final_args[@]}" 2> /tmp/ffmpeg_live_stats.log

[deezid]

Found a full hardware accelerated solution boosting speeds incredibly:
#47

[deezid]

@ravirajsinh45 — following up on the original #37 / Discussion #40. I've now implemented a proper portable transcoder backend in packages/transcoder/ffmpeg_transcoder.py (no more wrapper-script hacks intercepting ffmpeg). It directly addresses the "half-CPU" problem and the PCIe-bottleneck we discussed: the whole decode→scale→encode path now stays in VRAM via -hwaccel cuda -hwaccel_output_format cuda + scale_cuda in the filter_complex, so frames are never forced across the bus to the CPU.

What it does:

• detect_backend() auto-probes and prefers the dedicated GPU: nvenc → VAAPI → cpu, with graceful fallback when CUDA/NVENC is unavailable.
• NVENC path builds the full HLS 3-ladder with hevc_nvenc (Main10), scale_cuda, and spatial-AQ. Encoding 6K/4K footage drops from hours to minutes — matching what the wrapper gave, but cleanly.
• Intel/VAAPI path (hevc_vaapi, Main10) added for iGPU/AMD users, since true QSV (oneVPL/MFX) can't init on older Gen9 iGPUs.
• HDR ingest (HLG/PQ 10-bit) tone-maps to Rec.709 SDR while keeping the output Main10 10-bit; thumbnails fixed to be HDR-safe.
• Quality + bug fixes: scale_cuda set to lanczos (interp_algo=4); short-clip thumbnail extraction fixed (was emitting zero frames <10s, leaving phantom thumbnails).

One upstream-relevant note: -temporal-aq breaks Pascal GPUs (GTX 1080 Ti / NVENC 4 — "Provided device doesn't support required NVENC features"), so I gate it behind a compute-capability check (≥7.5 / Turing+). Default-off on inconclusive detection so unknown GPUs are never broken. Happy to drop it entirely if you'd prefer.

Configuration

All via env (wired through the worker environment in compose; defaults shown):

TRANSCODER_PIPELINE — which path. Default Auto.

Value	Resolves to
Auto (default)	detects, prefers the dGPU: nvenc → VAAPI → cpu (skips QSV — MFX won't init on Gen9)
NVIDIA	nvenc
Intel	VAAPI (true QSV can't init on this iGPU, so Intel → VAAPI)
Software	cpu (libx265/libx264)

TRANSCODER_BACKEND is still honored as fallback only if TRANSCODER_PIPELINE is unset.

TRANSCODER_OUTPUT — codec/quality. Default h265_10.

Value	Meaning
h265_10 (default)	HEVC 10-bit, high quality
h264_8	H.264 8-bit, smaller files

• nvenc: hevc_nvenc/h264_nvenc, profile main10/high. h265_10 forces Main10; a 10-bit source → genuinely yuv420p10le, an 8-bit SDR source stays 8-bit pixels under Main10 (no upscale). 10-bit = tuned bitrate ladder; 8-bit = lower NVENC ladder.
• vaapi: hevc_vaapi/h264_vaapi; 10-bit → scale_vaapi:format=p010 + profile main10 + pix_fmt p010 (verified Main10); 8-bit → nv12.
• cpu: libx265/libx264 with explicit yuv420p10le/yuv420p + adjusted CRF.

TRANSCODER_HDR — HDR handling. Default convert.

Value	Meaning
convert (default)	tone-map HDR → Rec.709 SDR
preserve	keep HDR10/HLG passthrough (10-bit, original bt2020 tags, no tone-map); thumbnail still SDR

Detects PQ (smpte2084), HLG (arib-std-b67), DV (smpte2094), bt2020 primaries.

Example: TRANSCODER_PIPELINE=Intel TRANSCODER_OUTPUT=h265_10 TRANSCODER_HDR=convert docker compose … up -d worker

---

I kept the PR scoped to code only — ffmpeg_transcoder.py + the apps/api/Dockerfile.prod non-free driver enablement. The CUDA image variant, docker-compose.gpu.yml, and device passthrough are deployment-specific, so I left them out (per your original scoping) — easy to add as a follow-up if you want them in-tree.

PR is up: #127 — would love a review, or guidance on what to change before it lands.