Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GStreamer][WebCodecs] Video encoding and decoding support #8399

Merged
merged 1 commit into from Jan 16, 2023

Conversation

philn
Copy link
Member

@philn philn commented Jan 9, 2023

c6f4630

[GStreamer][WebCodecs] Video encoding and decoding support
https://bugs.webkit.org/show_bug.cgi?id=249972

Reviewed by Xabier Rodriguez-Calvar.

The video encoder makes use of the WebRTC encoder. The video decoder relies on the platform decoder
previously probed through the registry scanner. In order to have integrate the encoding and decoding
at the lowest level possible we make use of a new component called GstElementHarness, so we have
full control on the input and output workflow of the encoder and decoder.

* LayoutTests/platform/glib/TestExpectations:
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/temporal-svc-encoding.https.any.worker_h264-expected.txt: Added.
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/temporal-svc-encoding.https.any.worker_vp8-expected.txt: Added.
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/temporal-svc-encoding.https.any.worker_vp9-expected.txt: Added.
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/videoFrame-createImageBitmap.any-expected.txt: Added.
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/videoFrame-drawImage.any-expected.txt: Added.
* Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoFrame.cpp:
(WebCore::WebCodecsVideoFrame::create):
* Source/WebCore/platform/GStreamer.cmake:
* Source/WebCore/platform/VideoDecoder.cpp:
(WebCore::VideoDecoder::createLocalDecoder):
* Source/WebCore/platform/VideoEncoder.cpp:
(WebCore::VideoEncoder::createLocalEncoder):
* Source/WebCore/platform/VideoFrame.cpp:
(WebCore::VideoFrame::paintInContext):
* Source/WebCore/platform/graphics/gstreamer/VideoDecoderGStreamer.cpp: Added.
(WebCore::gstWorkQueue):
(WebCore::GStreamerInternalVideoDecoder::create):
(WebCore::GStreamerInternalVideoDecoder::postTask):
(WebCore::GStreamerInternalVideoDecoder::close):
(WebCore::GStreamerVideoDecoder::create):
(WebCore::GStreamerVideoDecoder::GStreamerVideoDecoder):
(WebCore::GStreamerVideoDecoder::~GStreamerVideoDecoder):
(WebCore::GStreamerVideoDecoder::decode):
(WebCore::GStreamerVideoDecoder::flush):
(WebCore::GStreamerVideoDecoder::reset):
(WebCore::GStreamerVideoDecoder::close):
(WebCore::GStreamerInternalVideoDecoder::GStreamerInternalVideoDecoder):
(WebCore::GStreamerInternalVideoDecoder::decode):
(WebCore::GStreamerInternalVideoDecoder::flush):
* Source/WebCore/platform/graphics/gstreamer/VideoDecoderGStreamer.h: Added.
* Source/WebCore/platform/graphics/gstreamer/VideoEncoderGStreamer.cpp: Added.
(WebCore::gstWorkQueue):
(WebCore::GStreamerInternalVideoEncoder::create):
(WebCore::GStreamerInternalVideoEncoder::postTask):
(WebCore::GStreamerInternalVideoEncoder::close):
(WebCore::GStreamerInternalVideoEncoder::harness const):
(WebCore::GStreamerInternalVideoEncoder::isClosed const):
(WebCore::GStreamerVideoEncoder::create):
(WebCore::GStreamerVideoEncoder::GStreamerVideoEncoder):
(WebCore::GStreamerVideoEncoder::~GStreamerVideoEncoder):
(WebCore::GStreamerVideoEncoder::initialize):
(WebCore::GStreamerVideoEncoder::encode):
(WebCore::GStreamerVideoEncoder::flush):
(WebCore::GStreamerVideoEncoder::reset):
(WebCore::GStreamerVideoEncoder::close):
(WebCore::GStreamerInternalVideoEncoder::GStreamerInternalVideoEncoder):
(WebCore::GStreamerInternalVideoEncoder::initialize):
(WebCore::GStreamerInternalVideoEncoder::encode):
(WebCore::GStreamerInternalVideoEncoder::flush):
* Source/WebCore/platform/graphics/gstreamer/VideoEncoderGStreamer.h: Added.
* Source/WebCore/platform/graphics/gstreamer/VideoFrameGStreamer.cpp:
(WebCore::GstSampleColorConverter::convertSample):
(WebCore::GstSampleColorConverter::convertSampleToImage):
(WebCore::VideoFrame::fromNativeImage):
(WebCore::copyToGstBufferPlane):
(WebCore::VideoFrame::createNV12):
(WebCore::VideoFrame::createRGBA):
(WebCore::VideoFrame::createBGRA):
(WebCore::VideoFrame::createI420):
(WebCore::VideoFrameGStreamer::create):
(WebCore::VideoFrameGStreamer::createWrappedSample):
(WebCore::VideoFrameGStreamer::VideoFrameGStreamer):
(WebCore::copyPlane):
(WebCore::VideoFrame::copyTo):
(WebCore::VideoFrame::paintInContext):
* Source/WebCore/platform/graphics/gstreamer/VideoFrameGStreamer.h:
* Source/WebCore/platform/gstreamer/GStreamerElementHarness.cpp: Added.
(WebCore::GStreamerElementHarness::GStreamerElementHarness):
(WebCore::GStreamerElementHarness::~GStreamerElementHarness):
(WebCore::GStreamerElementHarness::start):
(WebCore::GStreamerElementHarness::pushStickyEvents):
(WebCore::GStreamerElementHarness::pushSample):
(WebCore::GStreamerElementHarness::pushBuffer):
(WebCore::GStreamerElementHarness::pushEvent):
(WebCore::GStreamerElementHarness::pullBuffer):
(WebCore::GStreamerElementHarness::pullEvent):
(WebCore::GStreamerElementHarness::outputCaps):
(WebCore::GStreamerElementHarness::chainBuffer):
(WebCore::GStreamerElementHarness::sinkQuery):
(WebCore::GStreamerElementHarness::sinkEvent):
(WebCore::GStreamerElementHarness::srcQuery):
(WebCore::GStreamerElementHarness::srcEvent):
(WebCore::GStreamerElementHarness::processOutputBuffers):
(WebCore::GStreamerElementHarness::flush):
* Source/WebCore/platform/gstreamer/GStreamerElementHarness.h: Added.
(WebCore::GStreamerElementHarness::create):
(WebCore::GStreamerElementHarness::element const):
* Source/cmake/OptionsGTK.cmake:
* Source/cmake/OptionsWPE.cmake:
* Tools/TestWebKitAPI/PlatformGTK.cmake:
* Tools/TestWebKitAPI/PlatformWPE.cmake:
* Tools/TestWebKitAPI/Tests/WebCore/gstreamer/GstElementHarness.cpp: Added.
(TestWebKitAPI::TEST_F):

Canonical link: https://commits.webkit.org/258949@main

19f922e

Misc iOS, tvOS & watchOS macOS Linux Windows
βœ… πŸ§ͺ style βœ… πŸ›  ios βœ… πŸ›  mac ❌ πŸ›  wpe πŸ’₯ πŸ›  πŸ§ͺ win
βœ… πŸ§ͺ bindings βœ… πŸ›  ios-sim βœ… πŸ›  mac-AS-debug βœ… πŸ›  gtk βœ… πŸ›  wincairo
βœ… πŸ§ͺ webkitperl βœ… πŸ§ͺ ios-wk2   πŸ§ͺ api-mac   πŸ§ͺ gtk-wk2
βœ… πŸ§ͺ api-ios βœ… πŸ§ͺ mac-wk1 βœ… πŸ§ͺ api-gtk
βœ… πŸ›  πŸ§ͺ jsc βœ… πŸ›  tv βœ… πŸ§ͺ mac-wk2 βœ… πŸ›  jsc-armv7
βœ… πŸ›  πŸ§ͺ jsc-arm64 βœ… πŸ›  tv-sim βœ… πŸ§ͺ mac-AS-debug-wk2 βœ… πŸ§ͺ jsc-armv7-tests
βœ… πŸ›  watch βœ… πŸ§ͺ mac-wk2-stress βœ… πŸ›  jsc-mips
❌ πŸ›  πŸ§ͺ merge βœ… πŸ›  watch-sim βœ… πŸ§ͺ jsc-mips-tests

@philn philn requested a review from a team as a code owner January 9, 2023 16:30
@philn philn self-assigned this Jan 9, 2023
@philn philn added the Platform Portability improvements and other general platform improvements not driven directly by site bugs. label Jan 9, 2023
@philn philn requested a review from calvaris January 9, 2023 16:31
@philn
Copy link
Member Author

philn commented Jan 9, 2023

I wasn't able to make style-check happy. This is too cryptic for me:

ERROR: /var/home/phil/WebKit/LayoutTests/platform/glib/TestExpectations:1152: More specific entry for imported/w3c/web-platform-tests/webcodecs/videoFrame-texImage.any.worker.html on line LayoutTests/platform/glib/TestExpectations:1152 overrides line LayoutTests/platform/glib/TestExpectations:1151. [test/expectations] [5]
Total errors found: 1 in 26 files

@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Jan 9, 2023
@philn philn removed the merging-blocked Applied to prevent a change from being merged label Jan 9, 2023
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Jan 9, 2023
LayoutTests/platform/glib/TestExpectations Outdated Show resolved Hide resolved
return VideoFrameGStreamer::create(WTFMove(sample), presentationSize);
}

RefPtr<VideoFrame> VideoFrame::createBGRA(Span<const uint8_t> span, size_t width, size_t height, const ComputedPlaneLayout& plane, PlatformVideoColorSpace&& colorSpace)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you can rework these two functions to use the same code but a flag variable? Am I missing any other difference? Not big deal anyway.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can refactor this with a macro.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not a big fan of macros, TBH

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too bad ;)

Source/WebCore/platform/VideoDecoder.cpp Show resolved Hide resolved
@philn philn removed the merging-blocked Applied to prevent a change from being merged label Jan 12, 2023
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Jan 12, 2023
@philn
Copy link
Member Author

philn commented Jan 12, 2023

I can reproduce the webrtc/vp9.html crash in release (bt useless) but not in debug, so... 🀷🏻

@philn philn removed the merging-blocked Applied to prevent a change from being merged label Jan 13, 2023
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Jan 13, 2023
@philn
Copy link
Member Author

philn commented Jan 13, 2023

I can reproduce the webrtc/vp9.html crash in release (bt useless) but not in debug, so... 🀷🏻

Actually, this is due to the llvm15 update! The test also crashes in main for builds made with clang15. Gonna revert that one.

@philn philn removed the merging-blocked Applied to prevent a change from being merged label Jan 13, 2023
@philn philn requested a review from calvaris January 13, 2023 12:34
GstCaps* filter = nullptr;
GRefPtr<GstCaps> caps;
if (m_outputCaps)
caps = m_outputCaps.get();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to keep the .get() ? I think you don't

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the way it was before, I think it was leaking.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

GstCaps* filter = nullptr;
GRefPtr<GstCaps> caps;
if (m_inputCaps)
caps = m_inputCaps.get();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto

@philn philn added the merge-queue Applied to send a pull request to merge-queue label Jan 16, 2023
https://bugs.webkit.org/show_bug.cgi?id=249972

Reviewed by Xabier Rodriguez-Calvar.

The video encoder makes use of the WebRTC encoder. The video decoder relies on the platform decoder
previously probed through the registry scanner. In order to have integrate the encoding and decoding
at the lowest level possible we make use of a new component called GstElementHarness, so we have
full control on the input and output workflow of the encoder and decoder.

* LayoutTests/platform/glib/TestExpectations:
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/temporal-svc-encoding.https.any.worker_h264-expected.txt: Added.
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/temporal-svc-encoding.https.any.worker_vp8-expected.txt: Added.
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/temporal-svc-encoding.https.any.worker_vp9-expected.txt: Added.
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/videoFrame-createImageBitmap.any-expected.txt: Added.
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/videoFrame-drawImage.any-expected.txt: Added.
* Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoFrame.cpp:
(WebCore::WebCodecsVideoFrame::create):
* Source/WebCore/platform/GStreamer.cmake:
* Source/WebCore/platform/VideoDecoder.cpp:
(WebCore::VideoDecoder::createLocalDecoder):
* Source/WebCore/platform/VideoEncoder.cpp:
(WebCore::VideoEncoder::createLocalEncoder):
* Source/WebCore/platform/VideoFrame.cpp:
(WebCore::VideoFrame::paintInContext):
* Source/WebCore/platform/graphics/gstreamer/VideoDecoderGStreamer.cpp: Added.
(WebCore::gstWorkQueue):
(WebCore::GStreamerInternalVideoDecoder::create):
(WebCore::GStreamerInternalVideoDecoder::postTask):
(WebCore::GStreamerInternalVideoDecoder::close):
(WebCore::GStreamerVideoDecoder::create):
(WebCore::GStreamerVideoDecoder::GStreamerVideoDecoder):
(WebCore::GStreamerVideoDecoder::~GStreamerVideoDecoder):
(WebCore::GStreamerVideoDecoder::decode):
(WebCore::GStreamerVideoDecoder::flush):
(WebCore::GStreamerVideoDecoder::reset):
(WebCore::GStreamerVideoDecoder::close):
(WebCore::GStreamerInternalVideoDecoder::GStreamerInternalVideoDecoder):
(WebCore::GStreamerInternalVideoDecoder::decode):
(WebCore::GStreamerInternalVideoDecoder::flush):
* Source/WebCore/platform/graphics/gstreamer/VideoDecoderGStreamer.h: Added.
* Source/WebCore/platform/graphics/gstreamer/VideoEncoderGStreamer.cpp: Added.
(WebCore::gstWorkQueue):
(WebCore::GStreamerInternalVideoEncoder::create):
(WebCore::GStreamerInternalVideoEncoder::postTask):
(WebCore::GStreamerInternalVideoEncoder::close):
(WebCore::GStreamerInternalVideoEncoder::harness const):
(WebCore::GStreamerInternalVideoEncoder::isClosed const):
(WebCore::GStreamerVideoEncoder::create):
(WebCore::GStreamerVideoEncoder::GStreamerVideoEncoder):
(WebCore::GStreamerVideoEncoder::~GStreamerVideoEncoder):
(WebCore::GStreamerVideoEncoder::initialize):
(WebCore::GStreamerVideoEncoder::encode):
(WebCore::GStreamerVideoEncoder::flush):
(WebCore::GStreamerVideoEncoder::reset):
(WebCore::GStreamerVideoEncoder::close):
(WebCore::GStreamerInternalVideoEncoder::GStreamerInternalVideoEncoder):
(WebCore::GStreamerInternalVideoEncoder::initialize):
(WebCore::GStreamerInternalVideoEncoder::encode):
(WebCore::GStreamerInternalVideoEncoder::flush):
* Source/WebCore/platform/graphics/gstreamer/VideoEncoderGStreamer.h: Added.
* Source/WebCore/platform/graphics/gstreamer/VideoFrameGStreamer.cpp:
(WebCore::GstSampleColorConverter::convertSample):
(WebCore::GstSampleColorConverter::convertSampleToImage):
(WebCore::VideoFrame::fromNativeImage):
(WebCore::copyToGstBufferPlane):
(WebCore::VideoFrame::createNV12):
(WebCore::VideoFrame::createRGBA):
(WebCore::VideoFrame::createBGRA):
(WebCore::VideoFrame::createI420):
(WebCore::VideoFrameGStreamer::create):
(WebCore::VideoFrameGStreamer::createWrappedSample):
(WebCore::VideoFrameGStreamer::VideoFrameGStreamer):
(WebCore::copyPlane):
(WebCore::VideoFrame::copyTo):
(WebCore::VideoFrame::paintInContext):
* Source/WebCore/platform/graphics/gstreamer/VideoFrameGStreamer.h:
* Source/WebCore/platform/gstreamer/GStreamerElementHarness.cpp: Added.
(WebCore::GStreamerElementHarness::GStreamerElementHarness):
(WebCore::GStreamerElementHarness::~GStreamerElementHarness):
(WebCore::GStreamerElementHarness::start):
(WebCore::GStreamerElementHarness::pushStickyEvents):
(WebCore::GStreamerElementHarness::pushSample):
(WebCore::GStreamerElementHarness::pushBuffer):
(WebCore::GStreamerElementHarness::pushEvent):
(WebCore::GStreamerElementHarness::pullBuffer):
(WebCore::GStreamerElementHarness::pullEvent):
(WebCore::GStreamerElementHarness::outputCaps):
(WebCore::GStreamerElementHarness::chainBuffer):
(WebCore::GStreamerElementHarness::sinkQuery):
(WebCore::GStreamerElementHarness::sinkEvent):
(WebCore::GStreamerElementHarness::srcQuery):
(WebCore::GStreamerElementHarness::srcEvent):
(WebCore::GStreamerElementHarness::processOutputBuffers):
(WebCore::GStreamerElementHarness::flush):
* Source/WebCore/platform/gstreamer/GStreamerElementHarness.h: Added.
(WebCore::GStreamerElementHarness::create):
(WebCore::GStreamerElementHarness::element const):
* Source/cmake/OptionsGTK.cmake:
* Source/cmake/OptionsWPE.cmake:
* Tools/TestWebKitAPI/PlatformGTK.cmake:
* Tools/TestWebKitAPI/PlatformWPE.cmake:
* Tools/TestWebKitAPI/Tests/WebCore/gstreamer/GstElementHarness.cpp: Added.
(TestWebKitAPI::TEST_F):

Canonical link: https://commits.webkit.org/258949@main
@webkit-commit-queue
Copy link
Collaborator

Committed 258949@main (c6f4630): https://commits.webkit.org/258949@main

Reviewed commits have been landed. Closing PR #8399 and removing active labels.

@webkit-commit-queue webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Jan 16, 2023
@webkit-early-warning-system webkit-early-warning-system merged commit c6f4630 into WebKit:main Jan 16, 2023
@philn philn deleted the eng/249972 branch January 16, 2023 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Platform Portability improvements and other general platform improvements not driven directly by site bugs.
Projects
None yet
6 participants