Skip to content
Permalink
Browse files
Add initial implementation to VideoFrame
https://bugs.webkit.org/show_bug.cgi?id=245824
rdar://problem/100557159

Reviewed by Eric Carlson.

Add initial implementation of VideoFrame.
In particular we add constructors and start supporting NV12, RGBA, BGRA.
We also support I420 (using NV12 as the backend on iOS/macOS).

Covered by rebased tests.

* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoDecoder-codec-specific.https.any.worker_vp8-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoDecoder-codec-specific.https.any.worker_vp9-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoDecoder-codec-specific.https.any_vp8-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoDecoder-codec-specific.https.any_vp9-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-construction.any-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-construction.any.worker-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-construction.crossOriginIsolated.https.any-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-construction.crossOriginIsolated.https.any.worker-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-construction.window-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-copyTo.any-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-copyTo.any.worker-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-copyTo.crossOriginIsolated.https.any-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-copyTo.crossOriginIsolated.https.any.worker-expected.txt:
* Source/ThirdParty/libwebrtc/Configurations/libwebrtc.iOS.exp:
* Source/ThirdParty/libwebrtc/Configurations/libwebrtc.iOSsim.exp:
* Source/ThirdParty/libwebrtc/Configurations/libwebrtc.mac.exp:
* Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitUtilities.h:
* Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitUtilities.mm:
* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::configure):
* Source/WebCore/Modules/webcodecs/WebCodecsVideoFrame.cpp:
(WebCore::checkImageUsability):
(WebCore::WebCodecsVideoFrame::create):
(WebCore::WebCodecsVideoFrame::initializeFrameFromOtherFrame):
(WebCore::WebCodecsVideoFrame::initializeFrameWithResourceAndSize):
(WebCore::WebCodecsVideoFrame::allocationSize):
(WebCore::WebCodecsVideoFrame::copyTo):
(WebCore::WebCodecsVideoFrame::setDisplaySize):
(WebCore::WebCodecsVideoFrame::setVisibleRect):
* Source/WebCore/Modules/webcodecs/WebCodecsVideoFrame.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoFrame.idl:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoFrameAlgorithms.cpp: Added.
(WebCore::isValidVideoFrameBufferInit):
(WebCore::verifyRectOffsetAlignment):
(WebCore::verifyRectSizeAlignment):
(WebCore::parseVisibleRect):
(WebCore::videoPixelFormatToPlaneCount):
(WebCore::videoPixelFormatToSampleByteSizePerPlane):
(WebCore::videoPixelFormatToSubSampling):
(WebCore::computeLayoutAndAllocationSize):
(WebCore::parseVideoFrameCopyToOptions):
(WebCore::initializeVisibleRectAndDisplaySize):
(WebCore::videoFramePickColorSpace):
* Source/WebCore/Modules/webcodecs/WebCodecsVideoFrameAlgorithms.h: Added.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/platform/PlaneLayout.h: Added.
* Source/WebCore/platform/VideoFrame.cpp:
(WebCore::VideoFrame::fromNativeImage):
(WebCore::VideoFrame::createNV12):
(WebCore::VideoFrame::createRGBA):
(WebCore::VideoFrame::createBGRA):
(WebCore::VideoFrame::createI420):
(WebCore::VideoFrame::copyTo):
* Source/WebCore/platform/VideoFrame.h:
* Source/WebCore/platform/VideoPixelFormat.cpp: Added.
(WebCore::convertVideoFramePixelFormat):
* Source/WebCore/platform/VideoPixelFormat.h: Added.
(WebCore::isRGBVideoPixelFormat):
* Source/WebCore/platform/graphics/cv/VideoFrameCV.mm:
* Source/WebCore/platform/mediastream/libwebrtc/LibWebRTCDTMFSenderBackend.h:
* Source/WebCore/platform/mediastream/libwebrtc/LibWebRTCProviderCocoa.cpp:

Canonical link: https://commits.webkit.org/255259@main
  • Loading branch information
youennf committed Oct 7, 2022
1 parent 263fc5c commit 5d0f369579801f91293bcae0845cca26c816ea44
Show file tree
Hide file tree
Showing 34 changed files with 1,125 additions and 100 deletions.
@@ -4,7 +4,7 @@ PASS Test isConfigSupported() with 1080p crop
PASS Test that isConfigSupported() returns a parsed configuration
FAIL Test invalid configs assert_unreached: Should have rejected: invalid codedWidth Reached unreachable code
PASS Test configure()
FAIL Decode a key frame assert_equals: duration expected 1 but got 0
PASS Decode a key frame
PASS Decode a non key frame first fails
FAIL Verify reset() suppresses outputs assert_greater_than: expected a number greater than 0 but got 0
PASS Test unconfigured VideoDecoder operations
@@ -13,7 +13,7 @@ FAIL Decode empty frame promise_rejects_dom: function "function () { throw e }"
FAIL Decode corrupt frame promise_rejects_dom: function "function () { throw e }" threw object "EncodingError: VPx decoding failed with error -1" that is not a DOMException AbortError: property "code" is equal to 0, expected 20
PASS Close while decoding corrupt frame
PASS Test decoding after flush
FAIL Test decoding a with negative timestamp assert_equals: timestamp expected -42 but got 0
PASS Test decoding a with negative timestamp
PASS Test reset during flush
PASS Test low-latency decoding
PASS VideoDecoder decodeQueueSize test
@@ -4,7 +4,7 @@ PASS Test isConfigSupported() with 1080p crop
PASS Test that isConfigSupported() returns a parsed configuration
FAIL Test invalid configs assert_unreached: Should have rejected: invalid codedWidth Reached unreachable code
PASS Test configure()
FAIL Decode a key frame assert_equals: duration expected 1 but got 0
PASS Decode a key frame
PASS Decode a non key frame first fails
FAIL Verify reset() suppresses outputs assert_greater_than: expected a number greater than 0 but got 0
PASS Test unconfigured VideoDecoder operations
@@ -13,7 +13,7 @@ FAIL Decode empty frame promise_rejects_dom: function "function () { throw e }"
FAIL Decode corrupt frame promise_rejects_dom: function "function () { throw e }" threw object "EncodingError: VPx decoding failed with error -1" that is not a DOMException AbortError: property "code" is equal to 0, expected 20
PASS Close while decoding corrupt frame
PASS Test decoding after flush
FAIL Test decoding a with negative timestamp assert_equals: timestamp expected -42 but got 0
PASS Test decoding a with negative timestamp
PASS Test reset during flush
PASS Test low-latency decoding
PASS VideoDecoder decodeQueueSize test
@@ -4,7 +4,7 @@ PASS Test isConfigSupported() with 1080p crop
PASS Test that isConfigSupported() returns a parsed configuration
FAIL Test invalid configs assert_unreached: Should have rejected: invalid codedWidth Reached unreachable code
PASS Test configure()
FAIL Decode a key frame assert_equals: duration expected 1 but got 0
PASS Decode a key frame
PASS Decode a non key frame first fails
FAIL Verify reset() suppresses outputs assert_greater_than: expected a number greater than 0 but got 0
PASS Test unconfigured VideoDecoder operations
@@ -13,7 +13,7 @@ FAIL Decode empty frame promise_rejects_dom: function "function () { throw e }"
FAIL Decode corrupt frame promise_rejects_dom: function "function () { throw e }" threw object "EncodingError: VPx decoding failed with error -1" that is not a DOMException AbortError: property "code" is equal to 0, expected 20
PASS Close while decoding corrupt frame
PASS Test decoding after flush
FAIL Test decoding a with negative timestamp assert_equals: timestamp expected -42 but got 0
PASS Test decoding a with negative timestamp
PASS Test reset during flush
PASS Test low-latency decoding
PASS VideoDecoder decodeQueueSize test
@@ -4,7 +4,7 @@ PASS Test isConfigSupported() with 1080p crop
PASS Test that isConfigSupported() returns a parsed configuration
FAIL Test invalid configs assert_unreached: Should have rejected: invalid codedWidth Reached unreachable code
PASS Test configure()
FAIL Decode a key frame assert_equals: duration expected 1 but got 0
PASS Decode a key frame
PASS Decode a non key frame first fails
FAIL Verify reset() suppresses outputs assert_greater_than: expected a number greater than 0 but got 0
PASS Test unconfigured VideoDecoder operations
@@ -13,7 +13,7 @@ FAIL Decode empty frame promise_rejects_dom: function "function () { throw e }"
FAIL Decode corrupt frame promise_rejects_dom: function "function () { throw e }" threw object "EncodingError: VPx decoding failed with error -1" that is not a DOMException AbortError: property "code" is equal to 0, expected 20
PASS Close while decoding corrupt frame
PASS Test decoding after flush
FAIL Test decoding a with negative timestamp assert_equals: timestamp expected -42 but got 0
PASS Test decoding a with negative timestamp
PASS Test reset during flush
PASS Test low-latency decoding
PASS VideoDecoder decodeQueueSize test
@@ -4,35 +4,29 @@ FAIL Test closed VideoFrame. Can't find variable: OffscreenCanvas
FAIL Test we can construct a VideoFrame with a negative timestamp. Can't find variable: OffscreenCanvas
FAIL Test that timestamp is required when constructing VideoFrame from ImageBitmap promise_test: Unhandled rejection with value: object "ReferenceError: Can't find variable: OffscreenCanvas"
FAIL Test that timestamp is required when constructing VideoFrame from OffscreenCanvas promise_test: Unhandled rejection with value: object "ReferenceError: Can't find variable: OffscreenCanvas"
FAIL Test that timestamp is NOT required when constructing VideoFrame from another VideoFrame promise_test: Unhandled rejection with value: object "TypeError: Type error"
PASS Test that timestamp is NOT required when constructing VideoFrame from another VideoFrame
FAIL Test we can construct an odd-sized VideoFrame. Can't find variable: OffscreenCanvas
FAIL Test constructing w/ unusable image argument throws: HAVE_NOTHING <video>. assert_throws_dom: function "() => {
let frame = new VideoFrame(video, {timestamp: 10});
}" did not throw
PASS Test constructing w/ unusable image argument throws: HAVE_NOTHING <video>.
FAIL Test constructing w/ unusable image argument throws: emtpy Canvas. Can't find variable: OffscreenCanvas
FAIL Test constructing w/ unusable image argument throws: closed ImageBitmap. Can't find variable: OffscreenCanvas
FAIL Test constructing w/ unusable image argument throws: closed VideoFrame. Can't find variable: OffscreenCanvas
FAIL Test invalid CanvasImageSource constructed VideoFrames Can't find variable: OffscreenCanvas
FAIL Test visibleRect metadata override where source display size = visible size Type error
FAIL Test visibleRect metadata override where source display width = 2 * visible width (anamorphic) Type error
FAIL Test visibleRect metadata override where source display size = 2 * visible size for both width and height Type error
PASS Test visibleRect metadata override where source display size = visible size
FAIL Test visibleRect metadata override where source display width = 2 * visible width (anamorphic) assert_equals: cropRightFrame.displayWidth expected 4 but got 2
FAIL Test visibleRect metadata override where source display size = 2 * visible size for both width and height assert_equals: cropRightFrame.displayWidth expected 4 but got 2
FAIL Test visibleRect + display size metadata override Can't find variable: OffscreenCanvas
FAIL Test display size metadata override Can't find variable: OffscreenCanvas
FAIL Test invalid buffer constructed VideoFrames assert_throws_js: invalid coded size function "() => constructFrame({
timestamp: 1234,
codedWidth: Math.pow(2, 32) - 1,
codedHeight: Math.pow(2, 32) - 1,
})" did not throw
FAIL Test Uint8Array(ArrayBuffer) constructed I420 VideoFrame assert_equals: plane format expected (string) "I420" but got (object) null
FAIL Test ArrayBuffer constructed I420 VideoFrame assert_equals: plane format expected (string) "I420" but got (object) null
FAIL Test planar constructed I420 VideoFrame with colorSpace null is not an object (evaluating 'frame.colorSpace.primaries')
FAIL Test buffer constructed I420+Alpha VideoFrame assert_equals: plane format expected (string) "I420A" but got (object) null
FAIL Test buffer constructed NV12 VideoFrame assert_equals: plane format expected (string) "NV12" but got (object) null
FAIL Test buffer constructed RGB VideoFrames assert_equals: plane format expected (string) "RGBA" but got (object) null
FAIL Test invalid buffer constructed VideoFrames assert_throws_js: odd coded height function "() => constructFrame({timestamp: 1234, codedWidth: 4, codedHeight: 1})" did not throw
PASS Test Uint8Array(ArrayBuffer) constructed I420 VideoFrame
PASS Test ArrayBuffer constructed I420 VideoFrame
PASS Test planar constructed I420 VideoFrame with colorSpace
FAIL Test buffer constructed I420+Alpha VideoFrame VideoPixelFormat is not supported
PASS Test buffer constructed NV12 VideoFrame
FAIL Test buffer constructed RGB VideoFrames VideoPixelFormat is not supported
FAIL Test VideoFrame constructed VideoFrame Can't find variable: OffscreenCanvas
FAIL Test we can construct a VideoFrame from an offscreen canvas. Can't find variable: OffscreenCanvas
FAIL Test I420 VideoFrame with odd visible size assert_equals: format expected (string) "I420" but got (object) null
FAIL Test I420A VideoFrame and alpha={keep,discard} assert_equals: plane format expected (string) "I420A" but got (object) null
FAIL Test RGBA, BGRA VideoFrames with alpha={keep,discard} assert_equals: plane format expected (string) "RGBA" but got (object) null
PASS Test I420 VideoFrame with odd visible size
FAIL Test I420A VideoFrame and alpha={keep,discard} VideoPixelFormat is not supported
FAIL Test RGBA, BGRA VideoFrames with alpha={keep,discard} assert_equals: plane format expected "RGBX" but got "RGBA"
FAIL Test a VideoFrame constructed from canvas can drop the alpha channel. Can't find variable: OffscreenCanvas

@@ -4,33 +4,29 @@ FAIL Test closed VideoFrame. Can't find variable: OffscreenCanvas
FAIL Test we can construct a VideoFrame with a negative timestamp. Can't find variable: OffscreenCanvas
FAIL Test that timestamp is required when constructing VideoFrame from ImageBitmap promise_test: Unhandled rejection with value: object "ReferenceError: Can't find variable: OffscreenCanvas"
FAIL Test that timestamp is required when constructing VideoFrame from OffscreenCanvas promise_test: Unhandled rejection with value: object "ReferenceError: Can't find variable: OffscreenCanvas"
FAIL Test that timestamp is NOT required when constructing VideoFrame from another VideoFrame promise_test: Unhandled rejection with value: object "TypeError: Type error"
PASS Test that timestamp is NOT required when constructing VideoFrame from another VideoFrame
FAIL Test we can construct an odd-sized VideoFrame. Can't find variable: OffscreenCanvas
PASS Test constructing w/ unusable image argument throws: HAVE_NOTHING <video>.
FAIL Test constructing w/ unusable image argument throws: emtpy Canvas. Can't find variable: OffscreenCanvas
FAIL Test constructing w/ unusable image argument throws: closed ImageBitmap. Can't find variable: OffscreenCanvas
FAIL Test constructing w/ unusable image argument throws: closed VideoFrame. Can't find variable: OffscreenCanvas
FAIL Test invalid CanvasImageSource constructed VideoFrames Can't find variable: OffscreenCanvas
FAIL Test visibleRect metadata override where source display size = visible size Type error
FAIL Test visibleRect metadata override where source display width = 2 * visible width (anamorphic) Type error
FAIL Test visibleRect metadata override where source display size = 2 * visible size for both width and height Type error
PASS Test visibleRect metadata override where source display size = visible size
FAIL Test visibleRect metadata override where source display width = 2 * visible width (anamorphic) assert_equals: cropRightFrame.displayWidth expected 4 but got 2
FAIL Test visibleRect metadata override where source display size = 2 * visible size for both width and height assert_equals: cropRightFrame.displayWidth expected 4 but got 2
FAIL Test visibleRect + display size metadata override Can't find variable: OffscreenCanvas
FAIL Test display size metadata override Can't find variable: OffscreenCanvas
FAIL Test invalid buffer constructed VideoFrames assert_throws_js: invalid coded size function "() => constructFrame({
timestamp: 1234,
codedWidth: Math.pow(2, 32) - 1,
codedHeight: Math.pow(2, 32) - 1,
})" did not throw
FAIL Test Uint8Array(ArrayBuffer) constructed I420 VideoFrame assert_equals: plane format expected (string) "I420" but got (object) null
FAIL Test ArrayBuffer constructed I420 VideoFrame assert_equals: plane format expected (string) "I420" but got (object) null
FAIL Test planar constructed I420 VideoFrame with colorSpace null is not an object (evaluating 'frame.colorSpace.primaries')
FAIL Test buffer constructed I420+Alpha VideoFrame assert_equals: plane format expected (string) "I420A" but got (object) null
FAIL Test buffer constructed NV12 VideoFrame assert_equals: plane format expected (string) "NV12" but got (object) null
FAIL Test buffer constructed RGB VideoFrames assert_equals: plane format expected (string) "RGBA" but got (object) null
FAIL Test invalid buffer constructed VideoFrames assert_throws_js: odd coded height function "() => constructFrame({timestamp: 1234, codedWidth: 4, codedHeight: 1})" did not throw
PASS Test Uint8Array(ArrayBuffer) constructed I420 VideoFrame
PASS Test ArrayBuffer constructed I420 VideoFrame
PASS Test planar constructed I420 VideoFrame with colorSpace
FAIL Test buffer constructed I420+Alpha VideoFrame VideoPixelFormat is not supported
PASS Test buffer constructed NV12 VideoFrame
FAIL Test buffer constructed RGB VideoFrames VideoPixelFormat is not supported
FAIL Test VideoFrame constructed VideoFrame Can't find variable: OffscreenCanvas
FAIL Test we can construct a VideoFrame from an offscreen canvas. Can't find variable: OffscreenCanvas
FAIL Test I420 VideoFrame with odd visible size assert_equals: format expected (string) "I420" but got (object) null
FAIL Test I420A VideoFrame and alpha={keep,discard} assert_equals: plane format expected (string) "I420A" but got (object) null
FAIL Test RGBA, BGRA VideoFrames with alpha={keep,discard} assert_equals: plane format expected (string) "RGBA" but got (object) null
PASS Test I420 VideoFrame with odd visible size
FAIL Test I420A VideoFrame and alpha={keep,discard} VideoPixelFormat is not supported
FAIL Test RGBA, BGRA VideoFrames with alpha={keep,discard} assert_equals: plane format expected "RGBX" but got "RGBA"
FAIL Test a VideoFrame constructed from canvas can drop the alpha channel. Can't find variable: OffscreenCanvas

@@ -1,4 +1,4 @@

FAIL Test SharedArrayBuffer constructed I420 VideoFrame assert_equals: plane format expected (string) "I420" but got (object) null
FAIL Test Uint8Array(SharedArrayBuffer) constructed I420 VideoFrame assert_equals: plane format expected (string) "I420" but got (object) null
PASS Test SharedArrayBuffer constructed I420 VideoFrame
PASS Test Uint8Array(SharedArrayBuffer) constructed I420 VideoFrame

@@ -1,4 +1,4 @@

FAIL Test SharedArrayBuffer constructed I420 VideoFrame assert_equals: plane format expected (string) "I420" but got (object) null
FAIL Test Uint8Array(SharedArrayBuffer) constructed I420 VideoFrame assert_equals: plane format expected (string) "I420" but got (object) null
PASS Test SharedArrayBuffer constructed I420 VideoFrame
PASS Test Uint8Array(SharedArrayBuffer) constructed I420 VideoFrame

@@ -1,7 +1,7 @@

Harness Error (TIMEOUT), message = null

FAIL Test that timestamp is required when constructing VideoFrame from HTMLImageElement assert_throws_js: timestamp required to construct VideoFrame from this source function "() => new VideoFrame(imageSource)" did not throw
PASS Test that timestamp is required when constructing VideoFrame from HTMLImageElement
TIMEOUT Test that timestamp is required when constructing VideoFrame from SVGImageElement Test timed out
NOTRUN Test that timeamp is required when constructing VideoFrame from HTMLCanvasElement

@@ -1,14 +1,14 @@

FAIL Test closed frame. assert_throws_dom: allocationSize() function "() => frame.allocationSize()" threw object "TypeError: Member VideoFrameCopyToOptions.rect is required and must be an instance of DOMRectInit" that is not a DOMException InvalidStateError: property "code" is equal to undefined, expected 11
FAIL Test copying I420 frame to a non-shared ArrayBuffer promise_test: Unhandled rejection with value: object "TypeError: Member VideoFrameCopyToOptions.rect is required and must be an instance of DOMRectInit"
FAIL Test copying I420 frame to a non-shared ArrayBufferView promise_test: Unhandled rejection with value: object "TypeError: Member VideoFrameCopyToOptions.rect is required and must be an instance of DOMRectInit"
FAIL Test RGBA frame. promise_test: Unhandled rejection with value: object "TypeError: Member VideoFrameCopyToOptions.rect is required and must be an instance of DOMRectInit"
PASS Test closed frame.
FAIL Test copying I420 frame to a non-shared ArrayBuffer assert_equals: allocationSize() expected 12 but got 0
FAIL Test copying I420 frame to a non-shared ArrayBufferView assert_equals: allocationSize() expected 12 but got 0
FAIL Test RGBA frame. assert_equals: allocationSize() expected 16 but got 0
PASS Test undersized buffer.
PASS Test incorrect plane count.
FAIL Test stride and offset work. promise_test: Unhandled rejection with value: object "TypeError: Member VideoFrameCopyToOptions.rect is required and must be an instance of DOMRectInit"
FAIL Test stride and offset with padding. promise_test: Unhandled rejection with value: object "TypeError: Member VideoFrameCopyToOptions.rect is required and must be an instance of DOMRectInit"
PASS Test invalid stride.
PASS Test address overflow.
FAIL Test incorrect plane count. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test stride and offset work. assert_equals: allocationSize() expected 12 but got 0
FAIL Test stride and offset with padding. assert_equals: allocationSize() expected 19 but got 0
FAIL Test invalid stride. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test address overflow. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test codedRect. assert_equals: allocationSize() expected 12 but got 0
FAIL Test empty rect. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test unaligned rect. assert_throws_js: function "() => frame.allocationSize(options)" did not throw

0 comments on commit 5d0f369

Please sign in to comment.