[SOLVED] 6K Hardware-Native Transcoding Wrapper (3x-7x Speed Boost)

[deezid]

Got 6K OpenGate transcoding at 2.1x speed and 4K 10bit H265 footage at 7.5x with ~15% CPU usage on a 1080 Ti.

Created a zero-copy ffmpeg-wrapper that bypasses any kind of software decoding, filtering and encoding and fully utilizes an NVIDIA GPU for which speeds up ingest tasks up to 30x faster over software ingest.

This script is compatible with all NVIDIA GPUs starting from early Pascal (GTX 10xx).
It will create H265 main10 files in 360p, 720p and 1080p with up to 10-15mbps.

Installing the nvidia patch is mandatory to make the 3 split encoding possible:
https://github.com/keylase/nvidia-patch

Also this needs to be in docker-compose.prod.yaml in worker:

    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all

    volumes:
      - /opt/freeframe/scripts/ffmpeg_wrapper:/usr/bin/ffmpeg:ro
      - /usr/bin/ffmpeg:/usr/bin/ffmpeg.original:ro
      - /opt/freeframe/tmp:/tmp
      - /usr/local/nvidia:/usr/local/nvidia:ro

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu, video, utility]

Implementation Checklist for Users

1. Driver Patch: You must apply the NVIDIA Patch to your host system to bypass the simultaneous stream limit.
2. Wrapper Setup: Save the script to /opt/freeframe/scripts/ffmpeg_wrapper and run chmod +x.
3. Docker Config: Update your docker-compose.prod.yaml with the volume and environment overrides provided above. This "tricks" the worker into using the wrapper instead of the standard binary.

#!/bin/bash
# --- NVIDIA GPU Wrapper (v46.3 - LANCZOS EDITION) ---

# --- MODE DETECTION ---
IS_VIDEO=true
for arg in "$@"; do
    if [[ "$arg" == *"mjpeg"* || "$arg" == *"image2"* || "$arg" == "-vframes" || "$arg" == *".jpg"* ]]; then
        IS_VIDEO=false
        break
    fi
done

if [ "$IS_VIDEO" = true ]; then
    # VIDEO MODE: Pure VRAM path + Lanczos (interp_algo 4)
    # Note: sws_flags here is a fallback, real work happens in filter_complex
    final_args=("-hwaccel" "cuda" "-hwaccel_output_format" "cuda" "-sws_flags" "lanczos")
else
    # THUMBNAIL MODE: HW Decode -> CPU Bridge -> software Lanczos
    final_args=("-hwaccel" "cuda" "-sws_flags" "lanczos")
fi

stream_index=0
while [[ $# -gt 0 ]]; do
    case "$1" in
        -filter_complex)
            raw_filter="$2"
            if [ "$IS_VIDEO" = true ]; then
                # Inject interp_algo=4 for Lanczos hardware scaling
                mod_filter=$(echo "$raw_filter" | sed -E \
                    -e 's/scale=1920:[0-9]+:[^,]+,pad=[^[]+/scale_cuda=1920:-2:interp_algo=4/g' \
                    -e 's/scale=1280:[0-9]+:[^,]+,pad=[^[]+/scale_cuda=1280:-2:interp_algo=4/g' \
                    -e 's/scale=640:[0-9]+:[^,]+,pad=[^[]+/scale_cuda=640:-2:interp_algo=4/g')
                final_args+=("-filter_complex" "$mod_filter")
            else
                final_args+=("-filter_complex" "$raw_filter")
            fi
            shift 2
            ;;
        libx264)
            # High-Fidelity 15Mbps HEVC settings
            if [ "$stream_index" -eq 0 ]; then
                BITRATE="20M"; MAXRATE="25M"; QMIN="1"; QMAX="15"
            elif [ "$stream_index" -eq 1 ]; then
                BITRATE="8M"; MAXRATE="12M"; QMIN="10"; QMAX="22"
            else
                BITRATE="2M"; MAXRATE="4M"; QMIN="15"; QMAX="28"
            fi
            final_args+=("hevc_nvenc" "-preset" "p6" "-profile:v" "main10" "-rc" "vbr" "-b:v" "$BITRATE" "-maxrate" "$MAXRATE" "-qmin" "$QMIN" "-qmax" "$QMAX" "-rc-lookahead" "32" "-weighted_pred" "1" "-spatial-aq" "1" "-multipass" "0")
            stream_index=$((stream_index + 1))
            shift
            ;;
        -crf|-preset|-hwaccel|-hwaccel_device|-hwaccel_output_format|-pix_fmt)
            shift 2
            ;;
        *)
            final_args+=("$1")
            shift
            ;;
    esac
done

exec /usr/bin/ffmpeg.original "${final_args[@]}" 2> /tmp/ffmpeg_live_stats.log

[ravirajsinh45]

This is seriously impressive work. A 3x–7x speed boost on 6K footage by wrapping the ffmpeg binary itself is a clever end-run around the pluggable transcoder layer — you didn't have to touch a single line of Python, and the benchmark numbers speak for themselves.

I spent some time reading through the wrapper and thinking about whether this could land upstream, and honestly I think the best home for it is a fork / overlay you maintain rather than core. A few reasons:

1. Hardware prerequisites — It requires an NVIDIA GPU and the keylase/nvidia-patch to lift the concurrent NVENC session limit. That patch is outside NVIDIA's EULA and we can't ship it as a default recommendation, and the majority of self-hosters won't have a GPU at all.
2. Codec switch — The wrapper changes the output from H.264 (libx264) to HEVC main10 (hevc_nvenc). HEVC playback in browsers is still uneven (Safari is fine, Chrome/Firefox depend on hardware support), so making it the default path would regress compatibility for some viewers.
3. Regex-based filter rewriting — The sed rewrites on filter_complex are tightly coupled to the current command shape in packages/transcoder. Any change on our side (different scale rungs, new filters, audio handling) could silently break it and we wouldn't catch it in CI.

What I'd suggest instead: keep this as a freeframe-nvenc fork (or a small companion repo with the wrapper, the compose overrides, and your README), and I'm very happy to link to it prominently from the main README and pin it here in Discussions as a "Community: hardware-accelerated transcoding" resource. That way users with NVIDIA hardware can opt in without us having to bake NVIDIA assumptions into the default install, and you keep full control as ffmpeg versions move.

If you ever want to take a bigger swing at upstreaming it, the cleanest path would be a new engine inside packages/transcoder alongside the existing ffmpeg one, gated behind TRANSCODER_ENGINE=nvenc, so nothing is monkey-patched at the binary level. That's a bigger lift but it'd be reviewable. Zero pressure either way — the wrapper as-is is a great drop-in for people who know what they're doing.

Thanks a lot for sharing this and for the detailed writeup. It's exactly the kind of thing Show and tell is for.

[deezid]

Coming from pure software decoding/encoding it's a 30x speed increase here actually.

I don't really have any experience hosting anything on github.
Freeframe itself doesn't need any modification, this wrapper can be more seen as an addon?
I may be able to provide a quicksync/vaapi-based version as well but can't prepare and test anything for AMD.