Feature Request: NUMA enabled windows build

[SystemOfSilence]

Create a nightly build or provide documentation for a user to build an NUMA-enabled binary for windows (primary concern is x265 numa support). Not looking for a stable and supported release even though that would be nice. A DIY faq/guide/hints for those who are not familiar with handbrake code is sufficient given the probably low demand.

The current behavior on systems with many cores (48 in my case) when wpp,pmode,pme are set for x265 will create a large thread pool that can span nodes tanking performance. x265 provides native numa support but it is disabled in the source probably to maintain Win Vista support. Limiting the number of threads with "pools={some sane number}:frame-threads={some sane number" and running many instances of HB is a work around but native NUMA is preferred.

What are the steps to reproduce this problem: Supply "wpp:pmode:pme" in x265 advanced options.

Please provide the full activity log for the encode or scan attempt. You may attach the log as a file, post a pastebin URL to the log, or place the log inline below:

...(trimmed lines)...
HandBrake 1.0.1 (2016122900) - 64bit
OS: Microsoft Windows NT 10.0.14393.0 - 64bit
CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
Ram: 65312 MB, 
GPU Information:
  Microsoft Basic Display Adapter - 10.0.14393.0
...(trimmed lines)...
[16:15:48] job configuration:
[16:15:48]  * source
[16:15:48]    + E:\Media_Queue\1_RAW\BD\AVATAR
[16:15:48]    + title 184, chapter(s) 1 to 36
[16:15:48]  * destination
[16:15:48]    + E:\Media_Queue\2_Transcoded\junk.mkv
[16:15:48]    + container: Matroska (libavformat)
[16:15:48]      + chapter markers
[16:15:48]  * video track
[16:15:48]    + decoder: h264
[16:15:48]      + bitrate 200 kbps
[16:15:48]    + filters
[16:15:48]      + Detelecine (pullup) ()
[16:15:48]      + Comb Detect (mode=3:spatial-metric=2:motion-thresh=1:spatial-thresh=1:filter-mode=2:block-thresh=40:block-width=16:block-height=16)
[16:15:48]      + Decomb (mode=39)
[16:15:48]      + Framerate Shaper (mode=0)
[16:15:48]        + frame rate: same as source (around 23.976 fps)
[16:15:48]      + Crop and Scale (width=1920:height=1080:crop-top=2:crop-bottom=0:crop-left=2:crop-right=2)
[16:15:48]        + source: 1920 * 1080, crop (2/0/2/2): 1916 * 1078, scale: 1920 * 1080
[16:15:48]    + Output geometry
[16:15:48]      + storage dimensions: 1920 x 1080
[16:15:48]      + pixel aspect ratio: 1 : 1
[16:15:48]      + display dimensions: 1920 x 1080
[16:15:48]    + encoder: H.265 (libx265)
[16:15:48]      + preset:  fast
[16:15:48]      + options: wpp:pmode:pme
[16:15:48]      + profile: main
[16:15:48]      + quality: 20.00 (RF)
[16:15:48]  * audio track 1
[16:15:48]    + decoder: English (DTS) (5.1 ch) (track 1, id 0x711100)
[16:15:48]      + bitrate: 1536 kbps, samplerate: 48000 Hz
[16:15:48]    + mixdown: Dolby Pro Logic II
[16:15:48]    + encoder: AAC (libavcodec)
[16:15:48]      + bitrate: 256 kbps, samplerate: 48000 Hz
[16:15:48] sync: expecting 232608 video frames
[16:15:48] thread 1ef47905920 started ("Audio Synchronization")
x265 [info]: HEVC encoder version 2.1
x265 [info]: build info [Windows][GCC 5.4.0][64 bit] 8bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
x265 [warning]: Limit reference options 2 and 3 are not supported with pmode. Disabling limit reference
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 48 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 6 / wpp(17 rows)+pmode+pme
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut       : 24 / 240 / 40
x265 [info]: Lookahead / bframes / badapt        : 15 / 4 / 0
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 1 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : CRF-20.0 / 0.60
x265 [info]: tools: rd=2 psy-rd=2.00 rskip signhide tmvp fast-intra
x265 [info]: tools: strong-intra-smoothing lslices=6 deblock sao
[16:15:48] Writing Metadata to output file...
[16:15:48] thread 1ef496c3620 started ("Muxer")
[16:15:48] thread 1ef496c37e0 started ("Reader")
[16:15:48] thread 1ef4edbab20 started ("Audio decoder (libavcodec)")
[16:15:48] thread 1ef4edbace0 started ("Video decoder (libavcodec)")
[16:15:48] thread 1ef4edc07e0 started ("Video Synchronization")
[16:15:48] thread 1ef4e10a9f0 started ("AVCodec Audio encoder (libavcodec)")
[16:15:48] thread 1ef4e10a4b0 started ("H.265/HEVC encoder (libx265)")
[16:15:48] thread 1ef4e10a670 started ("Muxer")
[16:15:48] thread 1ef4e10a830 started ("Detelecine (pullup)")
[16:15:48] thread 1ef493ef3b0 started ("Comb Detect")
[16:15:48] thread 1ef493ef030 started ("Decomb")
[16:15:48] thread 1ef493eee70 started ("Framerate Shaper")
[16:15:48] thread 1ef493ee930 started ("Crop and Scale")
[h264 @ 000001ef48416fc0] Application has requested 25 threads. Using a thread count greater than 16 is not recommended.
...(trimmed lines)...

[kawan2]

Wonder if this is better solved by deleting -DWINXP_SUPPORT=ON in the x265 module.defs? According to the x265 doc, it will use the newer Windows api to automatically discover and exploit the NUMA characteristics.

Of course supporting the wpp (default on), pmode and pme options would also give additional flexibility.

On a separate note, was there a reason for not specifying -DCMAKE_BUILD_TYPE=Release ?

[maximd33]

just thinking loud: as per x265 and its numa support - what are performance differences can be expected for different modes?

[bradleysepos]

Looks like @Rodeo314 added -DWINXP_SUPPORT=ON when we updated to x265 1.0 about 3 years ago, prior to HandBrake 0.10.0. I'm not sure it's relevant now, so with some basic testing we should be able to remove it.

[sr55]

Yeh, we don't need XP support since we don't support it. As for Vista, it dropped below my support threashold some time ago (in terms of user count). If that's at risk also, I'm OK with that.

[bradleysepos]

Seems fine, I'll remove it shortly.

[bradleysepos]

If there are other issues preventing NUMA from working, feel free to comment further and we can reopen this.