Slow NVEnc H.264 and H.265 encoding on RTX 5070

[daviducz]

Problem Description

Dear HandBrake Support Team,
I am having a performance issue when encoding with the NVEnc encoder (both H.264 and H.265) on my NVIDIA RTX 5070.
Problem description:

Source video: AV1 file exported from DaVinci Resolve
Preset / settings: I tried several NVEnc presets (Quality, High Quality, Fast, etc.)
Result: Encoding is significantly slower than expected and GPU utilization stays at only ~6 % during the whole process.

For comparison, I took exactly the same source file and encoded it to H.264 and H.265 using Shutter Encoder (also with NVEnc). In Shutter Encoder the GPU usage goes to 95–99 % and the encoding is much faster (as expected from a modern RTX 5070).
This makes me believe the issue is specific to HandBrake’s implementation of NVEnc rather than a general GPU/driver problem.

My system:
GPU: NVIDIA GeForce RTX 5070 (latest Studio Driver installed)
OS: [Windows 11 Home]
HandBrake version: 1.11.1 (2026032200)

Could you please advise if this is a known limitation or if there is a setting I am missing that would allow HandBrake to properly utilize the full power of the RTX 5070 with NVEnc?
Thank you very much for your help!
David

Activity Log, Crash Log or any other details

HandBrake 1.11.1 (2026032200)
OS: Microsoft Windows NT 10.0.26200.0
CPU: AMD Ryzen 7 9800X3D 8-Core Processor           
Ram: 48517 MB, 
GPU Information:
  NVIDIA GeForce RTX 5070 - 32.0.15.9579
Screen: 2560x1440
Temp Dir: C:\Users\urbi3\AppData\Local\Temp\
Install Dir: C:\Program Files\HandBrake\
Data Dir: C:\Users\urbi3\AppData\Roaming\HandBrake

-------------------------------------------

 # Starting Encode ...

[23:03:32] base preset: MOBILE
[23:03:32] Remote Process started with Process ID: 15732 using port: 8037. Max Allowed Instances: 1
[23:03:32] Worker: Starting HandBrake Engine ...
[23:03:32] Worker: Parent Process Id 44564
[23:03:32] Worker: Starting Web Server on port 8037 ...
[23:03:32] Worker: Starting Listener: 1
[23:03:32] Worker: Server Started
[23:03:33] Worker: Disconnected worker monitoring enabled!
[23:03:33] Compile-time hardening features are enabled
[23:03:33] nvenc: version 13.0 is available
[23:03:33] nvdec: is available
[23:03:33] CUDA Version: 12.0
[23:03:33] vcn: not available on this system
[23:03:33] qsv: not available on this system
[23:03:33] hb_init: starting libhb thread

[23:03:33] Starting work at: Sun Mar 29 23:03:33 2026
[23:03:33] 1 job(s) to process
[23:03:33] json job:
{
  "Audio": {
    "AudioList": [
      {
        "Bitrate": 160,
        "DRC": 0,
        "Encoder": "av_aac",
        "Gain": 0,
        "Mixdown": 4,
        "NormalizeMixLevel": false,
        "Samplerate": 0,
        "Name": "Stereo",
        "Track": 0,
        "DitherMethod": 0
      }
    ],
    "CopyMask": [
      "copy:aac"
    ],
    "FallbackEncoder": "av_aac"
  },
  "Destination": {
    "ChapterList": [
      {
        "Name": "Chapter 1"
      }
    ],
    "ChapterMarkers": true,
    "AlignAVStart": true,
    "File": "D:\\! Prace videa\\Functions\\Test MOBILE.mp4",
    "Options": {
      "IpodAtom": false,
      "Optimize": true
    },
    "Mux": "av_mp4"
  },
  "Filters": {
    "FilterList": [
      {
        "ID": 20,
        "Settings": {
          "crop-bottom": "0",
          "crop-left": "0",
          "crop-right": "0",
          "crop-top": "0",
          "height": "480",
          "width": "854"
        }
      },
      {
        "ID": 11,
        "Settings": {
          "mode": "1"
        }
      }
    ]
  },
  "PAR": {
    "Num": 1,
    "Den": 1
  },
  "Metadata": {
    "CreationTime": "2026-03-29T17:43:12.000000Z"
  },
  "CoverArts": [],
  "SequenceID": 0,
  "Source": {
    "Angle": 1,
    "Range": {
      "Type": "chapter",
      "Start": 1,
      "End": 1
    },
    "Title": 1,
    "Path": "D:\\! Prace videa\\Functions\\test.mp4",
    "HWDecode": 4,
    "KeepDuplicateTitles": false
  },
  "Subtitle": {
    "Search": {
      "Burn": true,
      "Default": false,
      "Enable": true,
      "Forced": true
    },
    "SubtitleList": []
  },
  "Video": {
    "Encoder": "nvenc_h264",
    "Level": "auto",
    "Bitrate": 2000,
    "MultiPass": false,
    "Turbo": false,
    "ColorMatrixCode": 0,
    "Options": "",
    "Preset": "slower",
    "Profile": "auto",
    "HardwareDecode": 4
  }
}
[23:03:33] OS: Windows 10.0 (26200)
[23:03:33] CPU: Unknown
[23:03:33]  - logical processor count: 16
[23:03:33] Intel Quick Sync Video support: no
[23:03:33] hb_scan: path=D:\! Prace videa\Functions\test.mp4, title_index=1
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'D:\! Prace videa\Functions\test.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomav01iso2mp41
    creation_time   : 2026-03-29T17:43:12.000000Z
    encoder         : Blackmagic Design DaVinci Resolve
  Duration: 00:04:33.86, start: 0.000000, bitrate: 51064 kb/s
  Stream #0:0[0x1](und): Video: av1 (libdav1d) (Main) (av01 / 0x31307661), yuv420p(tv, bt709/bt709/unknown, progressive), 3840x2160, 50744 kb/s, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 12800 tbn (default)
    Metadata:
      creation_time   : 2026-03-29T17:43:12.000000Z
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : AV1 8-bit - NVIDIA
      timecode        : 01:00:00:00
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 320 kb/s (default)
    Metadata:
      creation_time   : 2026-03-29T17:43:12.000000Z
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
  Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74) (default)
    Metadata:
      creation_time   : 2026-03-29T17:43:12.000000Z
      handler_name    : TimeCodeHandler
      timecode        : 01:00:00:00
[23:03:33] scan: decoding previews for title 1
[23:03:33] scan: audio 0x1: aac, rate=48000Hz, bitrate=320000 Unknown (AAC LC, 2.0 ch, 320 kbps)
[23:03:33] using container PAR 1:1
[23:03:33] scan: 10 previews, 3840x2160, 25.000 fps, autocrop = 0/0/0/0, aspect 16:9, PAR 1:1, color profile: 1-1-1, chroma location: unspecified
[23:03:33] scan: supported video decoders: avcodec hwaccel
[23:03:33] libhb: scan thread found 1 valid title(s)
[23:03:33] Skipping subtitle scan.  No suitable subtitle tracks.
[23:03:33] Starting Task: Encoding Pass
[23:03:33] work: track 1, dithering not supported by codec
[23:03:33] work: only 1 chapter, disabling chapter markers
[23:03:33] job configuration:
[23:03:33]  * source
[23:03:33]    + D:\! Prace videa\Functions\test.mp4
[23:03:33]    + title 1, chapter(s) 1 to 1
[23:03:33]    + container: mov,mp4,m4a,3gp,3g2,mj2
[23:03:33]    + data rate: 51064 kbps
[23:03:33]  * destination
[23:03:33]    + D:\! Prace videa\Functions\Test MOBILE.mp4
[23:03:33]    + container: MPEG-4 (libavformat)
[23:03:33]      + optimized for HTTP streaming (fast start)
[23:03:33]      + align initial A/V stream timestamps
[23:03:33]  * video track
[23:03:33]    + decoder: av1 8-bit (yuv420p)
[23:03:33]      + bitrate 50744 kbps
[23:03:33]    + filters
[23:03:33]      + Framerate Shaper (mode=1)
[23:03:33]        + frame rate: 25.000 fps -> constant 25.000 fps
[23:03:33]      + Crop and Scale (width=854:height=480:crop-top=0:crop-bottom=0:crop-left=0:crop-right=0)
[23:03:33]        + source: 3840 * 2160, crop (0/0/0/0): 3840 * 2160, scale: 854 * 480
[23:03:33]    + Output geometry
[23:03:33]      + storage dimensions: 854 x 480
[23:03:33]      + pixel aspect ratio: 1 : 1
[23:03:33]      + display dimensions: 854 x 480
[23:03:33]    + encoder: H.264 (NVEnc)
[23:03:33]      + preset:  slower
[23:03:33]      + profile: auto
[23:03:33]      + level:   auto
[23:03:33]      + bitrate: 2000 kbps, pass: 0
[23:03:33]      + color profile: 1-1-1
[23:03:33]      + color range: tv
[23:03:33]      + chroma location: unspecified
[23:03:33]  * audio track 1
[23:03:33]    + name: Stereo
[23:03:33]    + decoder: Unknown (AAC LC, 2.0 ch, 320 kbps) (track 1, id 0x1)
[23:03:33]      + bitrate: 320 kbps, samplerate: 48000 Hz
[23:03:33]    + mixdown: Stereo
[23:03:33]    + encoder: AAC (libavcodec)
[23:03:33]      + bitrate: 160 kbps, samplerate: 48000 Hz
[23:03:33] sync: expecting 6846 video frames
[23:03:33] encavcodecInit: H.264 (Nvidia NVENC)
[23:03:33] encavcodec: encoding at rc=vbr, Bitrate 2000
[23:03:33] encavcodec: encoding with stored aspect 1/1
[23:03:33] sync: first pts video is 0
[23:03:33] sync: "Chapter 1" (1) at frame 1 time 0
[23:03:33] sync: first pts audio 0x1 is 0
[23:07:03] reader: done. 1 scr changes
[23:07:03] work: average encoding speed for job is 32.378944 fps
[23:07:03] vfr: 6846 frames output, 0 dropped and 0 duped for CFR/PFR
[23:07:03] vfr: lost time: 0 (0 frames)
[23:07:03] vfr: gained time: 0 (0 frames) (0 not accounted for)
[23:07:03] aac-decoder done: 12837 frames, 0 decoder errors
[23:07:03] libdav1d-decoder done: 6846 frames, 0 decoder errors
[23:07:03] sync: got 6846 frames, 6846 expected
[23:07:03] sync: framerate min 25.000 fps, max 25.000 fps, avg 25.000 fps
[23:07:03] mux: track 0, 6846 frames, 69403943 bytes, 2027.28 kbps, fifo 2048
[23:07:03] mux: track 1, 12838 frames, 5487027 bytes, 160.28 kbps, fifo 2048
[23:07:03] Finished work at: Sun Mar 29 23:07:03 2026
[23:07:03] libhb: work result = 0

 # Job Completed!

What Operating System are you running?

Windows 11

What version of HandBrake are you running?

1.11.1 (2026032200)

Where did you download HandBrake from?

handbrake.fr

[galad87]

The NVIDIA hardware scaler was disabled because it was buggy, maybe now it's better, but I don't have any NVIDIA GPU to test, so someone else will have to check and re-enable it if it works.

[galad87]

In the meantime, you can avoid exporting to AV1 from Davinci, all those compression pass won't help the final quality.

[daviducz]

Thank you for response. Hope it will be re-enabled again, love to use HB.

[daviducz]

Today I discovered that when I export from DaVinci Resolve using Avid DNxHR and then encode to H.265 with NVEnc in HandBrake, my RTX 5070 runs at 100% utilization and I get approximately 140 fps in 4K — which is fantastic.

When encoding to H.264 in HandBrake using NVEnc, the process relies mainly on the CPU, while the GPU utilization ranges from about 10 % to 70 %, depending on the target resolution. Despite the lower GPU usage, the encoding speeds are still excellent.

It looks like the codec of the source file is very important. This setup only works well with the DNxHR intermediate file. If I export directly to AV1 using NVEnc from DaVinci, the high GPU usage and speed are not achieved.