nvenc RGB pixel upload mode

[totaam]

Issue migrated from trac ticket # 1347

component: encodings | priority: major | resolution: fixed

2016-10-28 09:44:50: antoine created the issue

---

Split from #1260, see also #1308 (10-bit).
Saves us doing the CSC conversion using CUDA - though we may still want to keep this code around for older cards / drivers.

[totaam]

2017-02-20 11:43:30: antoine changed status from new to assigned

[totaam]

2017-02-20 11:43:30: antoine commented

---

Makes sense to deal with this one at the same time as #1317 and #1308.

[totaam]

2017-05-02 17:52:39: antoine commented

---

Done:

• preparatory work in r15770: splitting code into more re-usable methods
• RGB upload supported added in r15778: the CUDA kernel step is now optional, RGB mode uploads straight to the encoder's input buffer (confusingly called CUDA output buffer.. since this is normally the output of the CUDA kernel)

We also gained:

• more fine-grained attributes: the "YUV444" and "lossless" flags are now per-codec. (ie: we don't disable them globally for "H264" if "HEVC" doesn't support it - and vice versa)
• ability to turn off some input formats using env vars: XPRA_NVENC_NATIVE_RGB=0, XPRA_NVENC_YUV420P=0 and XPRA_NVENC_YUV444P=0.

Tested on MS Windows only with NVFBC (#1317) because the NVENC encoder refuses to run on my Linux system: with the GTX 1070 it crashes, with other cards it errors out (and the exact same card and code work on MS Windows!)

I am keeping this ticket open because I would like to test RGB upload on Linux. (maybe try a different card, or diff the long list of API call arguments with a working example: ffmpeg or the samples provided)

[totaam]

2017-07-23 14:24:13: antoine changed status from assigned to closed

[totaam]

2017-07-23 14:24:13: antoine set resolution to fixed

[totaam]

2017-07-23 14:24:13: antoine commented

---

Not sure how I got it to work on MS Windows in the past, but since it just refuses to work with the latest code, r16455 disables it on win32.
See #1552#comment:4 for details. No big loss since we often get the buffers from NVFBC anyway (#1317) so doing the CSC via CUDA is cheap.