Implement RTC VTB decoders in GPUProcess

https://bugs.webkit.org/show_bug.cgi?id=205607

Reviewed by Eric Carlson.

Source/ThirdParty/libwebrtc:

Expose remote decoder abilities with C like functions.
This allows WebProcess to implement IPC-based decoders.
Expose VTB H264 decoder as C like functions.
This allows GPU process to instantiate wasily H2664 decoders.

* Configurations/libwebrtc.iOS.exp:
* Configurations/libwebrtc.iOSsim.exp:
* Configurations/libwebrtc.mac.exp:
* Source/webrtc/sdk/WebKit/WebKitUtilities.h:
* Source/webrtc/sdk/WebKit/WebKitUtilities.mm:
(webrtc::videoDecoderCallbacks):
(webrtc::setVideoDecoderCallbacks):
(webrtc::RemoteVideoDecoder::RemoteVideoDecoder):
(webrtc::RemoteVideoDecoder::decodeComplete):
(webrtc::RemoteVideoDecoder::InitDecode):
(webrtc::RemoteVideoDecoder::Decode):
(webrtc::RemoteVideoDecoder::RegisterDecodeCompleteCallback):
(webrtc::RemoteVideoDecoder::Release):
(webrtc::RemoteVideoDecoderFactory::RemoteVideoDecoderFactory):
(webrtc::RemoteVideoDecoderFactory::GetSupportedFormats const):
(webrtc::RemoteVideoDecoderFactory::CreateVideoDecoder):
(webrtc::createWebKitDecoderFactory):
(webrtc::createLocalDecoder):
(webrtc::releaseLocalDecoder):
(webrtc::decodeFrame):
* Source/webrtc/sdk/objc/components/video_codec/RTCVideoDecoderH264.h:
* Source/webrtc/sdk/objc/components/video_codec/RTCVideoDecoderH264.mm:
(-[RTCVideoDecoderH264 decode:missingFrames:codecSpecificInfo:renderTimeMs:]):
(-[RTCVideoDecoderH264 decodeData:size:timeStamp:]):

Source/WebCore:

Add routine to create a RemoveVideoSample from a pixel buffer.
Update LibWebRTCProvider to enable/disable decoding in GPU Process and add internals API.

Test: webrtc/video-gpuProcess.html

* platform/graphics/RemoteVideoSample.cpp:
(WebCore::RemoteVideoSample::create):
* platform/graphics/RemoteVideoSample.h:
* platform/graphics/cv/ImageTransferSessionVT.h:
* platform/mediastream/libwebrtc/LibWebRTCProvider.cpp:
(WebCore::LibWebRTCProvider::setUseGPUProcess):
* platform/mediastream/libwebrtc/LibWebRTCProvider.h:
* platform/mediastream/libwebrtc/LibWebRTCProviderCocoa.h:
* testing/Internals.cpp:
(WebCore::Internals::resetToConsistentState):
(WebCore::Internals::setUseGPUProcessForWebRTC):
* testing/Internals.h:
* testing/Internals.idl:

Source/WebKit:

Implement decoder factory callbacks.
Implement WebProcess codecs by sending IPC for creating/releasing/decoding a frame.
WebProcess receives IPC messages from GPU Process whenever a frame is decoded.

* Configurations/WebKit.xcconfig:
* DerivedSources-input.xcfilelist:
* DerivedSources-output.xcfilelist:
* DerivedSources.make:
* GPUProcess/GPUConnectionToWebProcess.cpp:
(WebKit::GPUConnectionToWebProcess::libWebRTCCodecsProxy):
(WebKit::GPUConnectionToWebProcess::didReceiveMessage):
* GPUProcess/GPUConnectionToWebProcess.h:
* GPUProcess/webrtc/LibWebRTCCodecsProxy.h: Added.
(WebKit::LibWebRTCCodecsProxy::didReceiveMessageFromWebProcess):
* GPUProcess/webrtc/LibWebRTCCodecsProxy.messages.in: Added.
* GPUProcess/webrtc/LibWebRTCCodecsProxy.mm: Added.
(WebKit::LibWebRTCCodecsProxy::LibWebRTCCodecsProxy):
(WebKit::LibWebRTCCodecsProxy::~LibWebRTCCodecsProxy):
(WebKit::LibWebRTCCodecsProxy::createDecoder):
(WebKit::LibWebRTCCodecsProxy::releaseDecoder):
(WebKit::LibWebRTCCodecsProxy::decodeFrame):
* Scripts/webkit/messages.py:
* Sources.txt:
* SourcesCocoa.txt:
* WebKit.xcodeproj/project.pbxproj:
* WebProcess/GPU/GPUProcessConnection.cpp:
(WebKit::GPUProcessConnection::didReceiveMessage):
* WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp: Added.
(WebKit::createVideoDecoder):
(WebKit::releaseVideoDecoder):
(WebKit::decodeVideoFrame):
(WebKit::registerDecodeCompleteCallback):
(WebKit::LibWebRTCCodecs::setVideoDecoderCallbacks):
(WebKit::LibWebRTCCodecs::createDecoder):
(WebKit::LibWebRTCCodecs::releaseDecoder):
(WebKit::LibWebRTCCodecs::decodeFrame):
(WebKit::LibWebRTCCodecs::registerDecodeFrameCallback):
(WebKit::LibWebRTCCodecs::failedDecoding):
(WebKit::LibWebRTCCodecs::completedDecoding):
* WebProcess/GPU/webrtc/LibWebRTCCodecs.h: Added.
* WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in: Added.
* WebProcess/GPU/webrtc/RTCDecoderIdentifier.h: Added.
* WebProcess/Network/webrtc/LibWebRTCProvider.cpp:
(WebKit::LibWebRTCProvider::createDecoderFactory):
* WebProcess/Network/webrtc/LibWebRTCProvider.h:
* WebProcess/WebProcess.cpp:
(WebKit::WebProcess::libWebRTCCodecs):
* WebProcess/WebProcess.h:

LayoutTests:

* webrtc/video-gpuProcess-expected.txt: Added.
* webrtc/video-gpuProcess.html: Added.


Canonical link: https://commits.webkit.org/218833@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@253957 268f45cc-cd09-0410-ab3c-d52691b4dbfc

LayoutTests/ChangeLog

@@ -1,3 +1,13 @@
+2019-12-31  youenn fablet  <youenn@apple.com>
+
+        Implement RTC VTB decoders in GPUProcess
+        https://bugs.webkit.org/show_bug.cgi?id=205607
+
+        Reviewed by Eric Carlson.
+
+        * webrtc/video-gpuProcess-expected.txt: Added.
+        * webrtc/video-gpuProcess.html: Added.
+
 2019-12-30  youenn fablet  <youenn@apple.com>
 
         Ignore URL host for schemes that are not using host information

LayoutTests/webrtc/video-gpuProcess-expected.txt

@@ -0,0 +1,6 @@
+
+
+PASS Basic video exchange 
+PASS Call setParameters to disable sending a given encoding 
+PASS Call setParameters to reenable sending a given encoding 
+

LayoutTests/webrtc/video-gpuProcess.html

@@ -0,0 +1,129 @@
+<!doctype html>
+<html>
+    <head>
+        <meta charset="utf-8">
+        <title>Video exchange using GPU process</title>
+        <script src="../resources/testharness.js"></script>
+        <script src="../resources/testharnessreport.js"></script>
+    </head>
+    <body>
+        <video id="video" autoplay=""></video>
+        <canvas id="canvas" width="640" height="480"></canvas>
+        <script src ="routines.js"></script>
+        <script>
+if (window.internals)
+    internals.setUseGPUProcessForWebRTC(true);
+
+video = document.getElementById("video");
+canvas = document.getElementById("canvas");
+
+function grabFrameData(x, y, w, h)
+{
+    canvas.width = video.videoWidth;
+    canvas.height = video.videoHeight;
+
+    canvas.getContext('2d').drawImage(video, x, y, w, h, x, y, w, h);
+    return canvas.getContext('2d').getImageData(x, y, w, h).data;
+}
+
+function testImage()
+{
+    const data = grabFrameData(10, 325, 250, 1);
+
+    var index = 20;
+    assert_true(data[index] < 100);
+    assert_true(data[index + 1] < 100);
+    assert_true(data[index + 2] < 100);
+
+    index = 80;
+    assert_true(data[index] > 200);
+    assert_true(data[index + 1] > 200);
+    assert_true(data[index + 2] > 200);
+
+    index += 80;
+    assert_true(data[index] > 200);
+    assert_true(data[index + 1] > 200);
+    assert_true(data[index + 2] < 100);
+}
+
+var pc1, pc2;
+promise_test(async (test) => {
+    if (window.testRunner)
+        testRunner.setUserMediaPermission(true);
+
+    const localStream = await navigator.mediaDevices.getUserMedia({video: {advanced: [{width:{min:640}}, {height:{min:480} } ]}});
+    if (window.internals)
+        assert_true(internals.pageMediaState().includes('HasActiveVideoCaptureDevice'), "Unexpected HasActiveVideoCaptureDevice");
+    const stream = await new Promise((resolve, reject) => {
+        createConnections((firstConnection) => {
+            pc1 = firstConnection;
+            firstConnection.addTrack(localStream.getVideoTracks()[0], localStream);
+        }, (secondConnection) => {
+            pc2 = secondConnection;
+            secondConnection.ontrack = (trackEvent) => {
+                assert_true(trackEvent.track instanceof MediaStreamTrack);
+                assert_true(trackEvent.receiver instanceof RTCRtpReceiver);
+                assert_true(Array.isArray(trackEvent.streams), "Array.isArray() should return true");
+                assert_true(Object.isFrozen(trackEvent.streams), "Object.isFrozen() should return true");
+                assert_equals(trackEvent.track.id, localStream.getVideoTracks()[0].id);
+                assert_equals(trackEvent.track, trackEvent.streams[0].getVideoTracks()[0]);
+                resolve(trackEvent.streams[0]);
+            };
+        });
+        setTimeout(() => reject("Test timed out"), 5000);
+    });
+
+    video.srcObject = stream;
+    await video.play();
+
+    testImage();
+}, "Basic video exchange");
+
+function getCircleImageData()
+{
+    return grabFrameData(450, 100, 150, 100);
+}
+
+async function checkVideoIsUpdated(shouldBeUpdated, count, referenceData)
+{
+    if (count === undefined)
+        count = 0;
+    else if (count >= 20)
+        return Promise.reject("checkVideoIsUpdated timed out :" + shouldBeUpdated + " " + count);
+
+    if (referenceData === undefined)
+        referenceData = getCircleImageData();
+
+    await waitFor(200);
+    const newData = getCircleImageData();
+
+    if (shouldBeUpdated === (JSON.stringify(referenceData) !== JSON.stringify(newData)))
+        return;
+
+    await checkVideoIsUpdated(shouldBeUpdated, ++count, newData);
+}
+
+promise_test(async (test) => {
+    const sender = pc1.getSenders()[0];
+    let p = sender.getParameters();
+    p.encodings[0].active = false;
+    await sender.setParameters(p);
+
+    assert_false(sender.getParameters().encodings[0].active, "encodings[0].active should be false");
+
+    await checkVideoIsUpdated(false);
+}, "Call setParameters to disable sending a given encoding");
+
+promise_test(async (test) => {
+    const sender = pc1.getSenders()[0];
+    let p = sender.getParameters();
+    p.encodings[0].active = true;
+    await sender.setParameters(p);
+
+    assert_true(sender.getParameters().encodings[0].active, "encodings[0].active should be true");
+
+    await checkVideoIsUpdated(true);
+}, "Call setParameters to reenable sending a given encoding");
+        </script>
+    </body>
+</html>

Source/ThirdParty/libwebrtc/ChangeLog

@@ -1,3 +1,40 @@
+2019-12-31  youenn fablet  <youenn@apple.com>
+
+        Implement RTC VTB decoders in GPUProcess
+        https://bugs.webkit.org/show_bug.cgi?id=205607
+
+        Reviewed by Eric Carlson.
+
+        Expose remote decoder abilities with C like functions.
+        This allows WebProcess to implement IPC-based decoders.
+        Expose VTB H264 decoder as C like functions.
+        This allows GPU process to instantiate wasily H2664 decoders.
+
+        * Configurations/libwebrtc.iOS.exp:
+        * Configurations/libwebrtc.iOSsim.exp:
+        * Configurations/libwebrtc.mac.exp:
+        * Source/webrtc/sdk/WebKit/WebKitUtilities.h:
+        * Source/webrtc/sdk/WebKit/WebKitUtilities.mm:
+        (webrtc::videoDecoderCallbacks):
+        (webrtc::setVideoDecoderCallbacks):
+        (webrtc::RemoteVideoDecoder::RemoteVideoDecoder):
+        (webrtc::RemoteVideoDecoder::decodeComplete):
+        (webrtc::RemoteVideoDecoder::InitDecode):
+        (webrtc::RemoteVideoDecoder::Decode):
+        (webrtc::RemoteVideoDecoder::RegisterDecodeCompleteCallback):
+        (webrtc::RemoteVideoDecoder::Release):
+        (webrtc::RemoteVideoDecoderFactory::RemoteVideoDecoderFactory):
+        (webrtc::RemoteVideoDecoderFactory::GetSupportedFormats const):
+        (webrtc::RemoteVideoDecoderFactory::CreateVideoDecoder):
+        (webrtc::createWebKitDecoderFactory):
+        (webrtc::createLocalDecoder):
+        (webrtc::releaseLocalDecoder):
+        (webrtc::decodeFrame):
+        * Source/webrtc/sdk/objc/components/video_codec/RTCVideoDecoderH264.h:
+        * Source/webrtc/sdk/objc/components/video_codec/RTCVideoDecoderH264.mm:
+        (-[RTCVideoDecoderH264 decode:missingFrames:codecSpecificInfo:renderTimeMs:]):
+        (-[RTCVideoDecoderH264 decodeData:size:timeStamp:]):
+
 2019-12-30  youenn fablet  <youenn@apple.com>
 
         Do not build yasm for iOS and iOS simulator

Source/ThirdParty/libwebrtc/Configurations/libwebrtc.iOS.exp

@@ -262,3 +262,8 @@ __ZNK6webrtc20DataChannelInterface17maxRetransmitsOptEv
 __ZNK6webrtc20RtpReceiverInterface14dtls_transportEv
 __ZNK6webrtc23PeerConnectionInterface16GetSctpTransportEv
 __ZNK6webrtc23RtpTransceiverInterface17codec_preferencesEv
+__ZN6webrtc18createLocalDecoderEU13block_pointerFvP10__CVBufferjjE
+__ZN6webrtc19releaseLocalDecoderEPv
+__ZN6webrtc11decodeFrameEPvjPKhm
+__ZN6webrtc18RemoteVideoDecoder14decodeCompleteEPvjP10__CVBufferj
+__ZN6webrtc24setVideoDecoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_jPKhmEPFiS0_S0_E

Source/ThirdParty/libwebrtc/Configurations/libwebrtc.iOSsim.exp

@@ -262,3 +262,8 @@ __ZNK6webrtc20DataChannelInterface17maxRetransmitsOptEv
 __ZNK6webrtc20RtpReceiverInterface14dtls_transportEv
 __ZNK6webrtc23PeerConnectionInterface16GetSctpTransportEv
 __ZNK6webrtc23RtpTransceiverInterface17codec_preferencesEv
+__ZN6webrtc18createLocalDecoderEU13block_pointerFvP10__CVBufferjjE
+__ZN6webrtc19releaseLocalDecoderEPv
+__ZN6webrtc11decodeFrameEPvjPKhm
+__ZN6webrtc18RemoteVideoDecoder14decodeCompleteEPvjP10__CVBufferj
+__ZN6webrtc24setVideoDecoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_jPKhmEPFiS0_S0_E

Source/ThirdParty/libwebrtc/Configurations/libwebrtc.mac.exp

@@ -262,3 +262,8 @@ __ZNK6webrtc20DataChannelInterface17maxRetransmitsOptEv
 __ZNK6webrtc20RtpReceiverInterface14dtls_transportEv
 __ZNK6webrtc23PeerConnectionInterface16GetSctpTransportEv
 __ZNK6webrtc23RtpTransceiverInterface17codec_preferencesEv
+__ZN6webrtc18createLocalDecoderEU13block_pointerFvP10__CVBufferjjE
+__ZN6webrtc19releaseLocalDecoderEPv
+__ZN6webrtc11decodeFrameEPvjPKhm
+__ZN6webrtc18RemoteVideoDecoder14decodeCompleteEPvjP10__CVBufferj
+__ZN6webrtc24setVideoDecoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_jPKhmEPFiS0_S0_E

Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitUtilities.h

@@ -69,4 +69,47 @@ class VideoEncoderFactoryWithSimulcast final : public VideoEncoderFactory {
     const std::unique_ptr<VideoEncoderFactory> m_internalEncoderFactory;
 };
 
+using WebKitVideoDecoder = void*;
+using VideoDecoderCreateCallback = WebKitVideoDecoder(*)(const SdpVideoFormat& format);
+using VideoDecoderReleaseCallback = int32_t(*)(WebKitVideoDecoder);
+using VideoDecoderDecodeCallback = int32_t(*)(WebKitVideoDecoder, uint32_t timeStamp, const uint8_t*, size_t length);
+using VideoDecoderRegisterDecodeCompleteCallback = int32_t(*)(WebKitVideoDecoder, void* decodedImageCallback);
+
+void setVideoDecoderCallbacks(VideoDecoderCreateCallback, VideoDecoderReleaseCallback, VideoDecoderDecodeCallback, VideoDecoderRegisterDecodeCompleteCallback);
+
+class RemoteVideoDecoderFactory final : public VideoDecoderFactory {
+public:
+    explicit RemoteVideoDecoderFactory(std::unique_ptr<VideoDecoderFactory>&&);
+    ~RemoteVideoDecoderFactory() = default;
+
+private:
+    std::vector<SdpVideoFormat> GetSupportedFormats() const final;
+    std::unique_ptr<VideoDecoder> CreateVideoDecoder(const SdpVideoFormat& format) final;
+
+    std::unique_ptr<VideoDecoderFactory> m_internalFactory;
+};
+
+class RemoteVideoDecoder final : public webrtc::VideoDecoder {
+public:
+    explicit RemoteVideoDecoder(WebKitVideoDecoder);
+    ~RemoteVideoDecoder() = default;
+
+    static void decodeComplete(void* callback, uint32_t timeStamp, CVPixelBufferRef, uint32_t timeStampRTP);
+
+private:
+    int32_t InitDecode(const VideoCodec*, int32_t number_of_cores) final;
+    int32_t Decode(const EncodedImage&, bool missing_frames, int64_t render_time_ms) final;
+    int32_t RegisterDecodeCompleteCallback(DecodedImageCallback*) final;
+    int32_t Release() final;
+    const char* ImplementationName() const final { return "RemoteVideoToolBox"; }
+
+    WebKitVideoDecoder m_internalDecoder;
+};
+
+using LocalDecoder = void*;
+using LocalDecoderCallback = void (^)(CVPixelBufferRef, uint32_t timeStamp, uint32_t timeStampNs);
+void* createLocalDecoder(LocalDecoderCallback);
+void releaseLocalDecoder(LocalDecoder);
+int32_t decodeFrame(LocalDecoder, uint32_t timeStamp, const uint8_t*, size_t);
+
 }