Repository for research into "data-moshing" bug in Unity Render Streaming. Visibly from missing or malformed keyframes or headers (IDR, PPS, SPS) causing predictive-frames to be glitchy. A comparison is made with Unreal4 PixelStreaming (requires signup to Unreal github) which also uses NVIDIA SDK, but does not have this bug.
- autr/com.unity.webrtc (fork with NVIDIA settings exposed)
- autr/UnityRenderStreaming-glitches (test project implementing plugin fork)
- Setting up Unity and Unreal4 development environment on PopOS (Linux)
- Setting up a development environment for com.unity.webrtc (Windows)
- Comparison between Unreal and Unity NVIDIA API settings
Codecs:
- H264 = AVC / MPEG4 Pt.10 (advanced video coding)
- HEVC = H265 / MPEG-H Pt.2 (high efficiency video coding)
- VP8 = alt codec, H264 equivalency (google)
- VP9 = alt codec, H265 equivalency (google)
Properties:
- GOP = length of GOP (group of pictures)
- RCM = rate control mode (VBR, CBR etc)
- QP = quantization parameter (compression amount)
- PPS = picture parameter set
- SPS = sequence parameter set
- I-FRAME = intra frame
- IDR-FRAME = instantaneous decoder refresh frame
- P-FRAME = predictive frame
- B-FRAME = bi-directional predictive frame
- NALU = network abstraction layer units
Error Resilience Mechanisms:
- NACK = negative acknowledgement
- FIR = full intra request
- PLI = picture loss indication (see
webrtc::RTCRtpStreamStats.pliCount, see also) - SLI = slice loss indication (see
webrtc::RTCRtpStreamStats.sliCount) - LTR = long term reference pictures (mysterious undocumented NVIDIA setting)
Official recommendations:
Low-latency use cases like game-streaming, video conferencing etc.
Ultra-low latency or low latency Tuning Info
Rate control mode = CBR
Multi Pass – Quarter/Full (evaluate and decide)
Very low VBV buffer size (e.g. single frame = bitrate/framerate)
No B Frames
Infinite GOP length
Adaptive quantization (AQ) enabled**
Long term reference pictures***
Intra refresh***
Non-reference P frames***
Force IDR***
*: Recommended for low motion games and natural video.
**: Recommended on second generation Maxwell GPUs and above.
***: These features are useful for error recovery during transmission across noisy mediums.
Elsewhere:
High quality Tuning Info
‣ Rate control mode = CBR
‣ Medium VBV buffer size (1 second)
‣ B Frames*
‣ Look-ahead
- https://docs.nvidia.com/video-technologies/video-codec-sdk/pdf/NVENC_VideoEncoder_API_ProgGuide.pdf
- https://on-demand.gputechconf.com/gtc/2014/presentations/S4654-detailed-overview-nvenc-encoder-api.pdf
- https://docs.nvidia.com/video-technologies/video-codec-sdk
- http://developer.download.nvidia.com/compute/nvenc/v4.0/NVENC_AppNote.pdf
- https://docs.nvidia.com/video-technologies/video-codec-sdk/nvenc-video-encoder-api-prog-guide
- https://github.com/rigaya/NVEnc
Unreal implementation references:
- NvEncoder.cpp (see lines: 407 / 742 / 742)
- VideoEncoder.cpp (see lines: 193 / 269)