Support HW encoding and decoding on HL2

[grbury]

Do we get performance improvements on HL2 by leveraging HW encoding and decoding? Would be great to know whats possible.

[djee-ms]

This is planned for the short term.

Note that there is already some preliminary support for selecting a preferred audio/video codec by manipulating the SDP offer packet:

MixedReality-WebRTC/libs/Microsoft.MixedReality.WebRTC/PeerConnection.cs

Lines 1177 to 1186 in 7c58813

	if ((PreferredAudioCodec.Length > 0) || (PreferredVideoCodec.Length > 0))
	{
	var builder = new StringBuilder(sdp.Length);
	ulong lengthInOut = (ulong)builder.Capacity;
	if (NativeMethods.SdpForceCodecs(sdp, PreferredAudioCodec, PreferredVideoCodec, builder, ref lengthInOut))
	{
	builder.Length = (int)lengthInOut;
	sdp = builder.ToString();
	}
	}

This is also exposed as a property in the LocalVideoSource and LocalAudioSource components of the Unity integration.

However I didn't verify yet if that guarantees hardware encoding/decoding, nor what kind of performance and thermal gain we get from this.

[djee-ms]

Update: Hardware encoding is confirmed to work on HoloLens 2 simply by using PreferredVideoCodec = "H264", without any other change.

[jahnotto]

Update: Hardware encoding is confirmed to work on HoloLens 2 simply by using PreferredVideoCodec = "H264", without any other change.

Decoding as well?

[djee-ms]

I didn't close this issue because I am sure at 99.99% it is, but I need to run that against the Media Foundation team at some point to validate 100% this is indeed the case (I already got that validation for the encoder). But yes, the code is in place and unless there is a bug in the decoder configuration I am not aware of, the decoder is also currently configured to use hardware decoding where available.

[djee-ms]

Update: hardware H.264 encoding is confirmed on UWP as stated above, but hardware decoding was never confirmed, and shall be assumed not present. We have no official visibility/plans on adding it at this time, due to resourcing, so removing any milestone reference for this feature.

[jahnotto]

Any updates on hardware decoding of H.264 and/or H.265 on HoloLens 2? @djee-ms

[djee-ms]

Amazingly, the Media Foundation team has been unable to provide us with a reliable method to determine which decoder is used internally when using their API (hardware or fallback software). It seems their API acts as a complete blackbox, and even with ETW traces we can't tell anything about its functioning. Since I never heard about any decoding performance issue from any user, and never saw it appear in profiling myself, we have reasonable confidence that hardware decoding is indeed used, because we expect software H.264 decoding would show up in profiling. But we can't confirm it 100%, that's why we didn't close this issue.

If you have some performance issue decoding please provide us with info and we can investigate. Otherwise unfortunately there is little chance we are looking into a 100% confidence answer to this anytime soon if this doesn't prove an issue in practice for anyone.