Skip to content
Permalink
Browse files
Implement WebCodecs VideoFrame allocationSize
https://bugs.webkit.org/show_bug.cgi?id=246209
rdar://problem/100893685

Reviewed by Eric Carlson.

Make CopyToOptions optionals so that they work with current algorithms.
Implement allocationSize by calling existing parseVideoFrameCopyToOptions function.
Update the allocation size to account for RGB formats which have 4 samples per pixel.

* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-copyTo.any-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-copyTo.any.worker-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-copyTo.crossOriginIsolated.https.any-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/videoFrame-copyTo.crossOriginIsolated.https.any.worker-expected.txt:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoFrame.cpp:
(WebCore::WebCodecsVideoFrame::allocationSize):
* Source/WebCore/Modules/webcodecs/WebCodecsVideoFrame.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoFrameAlgorithms.cpp:
(WebCore::parseVisibleRect):
(WebCore::sampleCountPerPixel):
(WebCore::computeLayoutAndAllocationSize):
(WebCore::parseVideoFrameCopyToOptions):
* Source/WebCore/platform/graphics/cv/VideoFrameCV.mm:
(WebCore::VideoFrame::copyTo):

Canonical link: https://commits.webkit.org/255313@main
  • Loading branch information
youennf committed Oct 8, 2022
1 parent 3cf19dc commit 60db1be1b9161f6c34aa7eb03c2198f5d19f9c49
Show file tree
Hide file tree
Showing 8 changed files with 65 additions and 40 deletions.
@@ -1,17 +1,17 @@

PASS Test closed frame.
FAIL Test copying I420 frame to a non-shared ArrayBuffer assert_equals: allocationSize() expected 12 but got 0
FAIL Test copying I420 frame to a non-shared ArrayBufferView assert_equals: allocationSize() expected 12 but got 0
FAIL Test RGBA frame. assert_equals: allocationSize() expected 16 but got 0
FAIL Test copying I420 frame to a non-shared ArrayBuffer promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
FAIL Test copying I420 frame to a non-shared ArrayBufferView promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
FAIL Test RGBA frame. assert_object_equals: plane 0 layout property "stride" expected 8 got 0
PASS Test undersized buffer.
FAIL Test incorrect plane count. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test stride and offset work. assert_equals: allocationSize() expected 12 but got 0
FAIL Test stride and offset with padding. assert_equals: allocationSize() expected 19 but got 0
FAIL Test invalid stride. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
PASS Test incorrect plane count.
FAIL Test stride and offset work. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
FAIL Test stride and offset with padding. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
PASS Test invalid stride.
FAIL Test address overflow. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test codedRect. assert_equals: allocationSize() expected 12 but got 0
FAIL Test empty rect. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test unaligned rect. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test left crop. assert_equals: allocationSize() expected 6 but got 0
FAIL Test invalid rect. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test codedRect. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
PASS Test empty rect.
PASS Test unaligned rect.
FAIL Test left crop. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
PASS Test invalid rect.

@@ -1,17 +1,17 @@

PASS Test closed frame.
FAIL Test copying I420 frame to a non-shared ArrayBuffer assert_equals: allocationSize() expected 12 but got 0
FAIL Test copying I420 frame to a non-shared ArrayBufferView assert_equals: allocationSize() expected 12 but got 0
FAIL Test RGBA frame. assert_equals: allocationSize() expected 16 but got 0
FAIL Test copying I420 frame to a non-shared ArrayBuffer promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
FAIL Test copying I420 frame to a non-shared ArrayBufferView promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
FAIL Test RGBA frame. assert_object_equals: plane 0 layout property "stride" expected 8 got 0
PASS Test undersized buffer.
FAIL Test incorrect plane count. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test stride and offset work. assert_equals: allocationSize() expected 12 but got 0
FAIL Test stride and offset with padding. assert_equals: allocationSize() expected 19 but got 0
FAIL Test invalid stride. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
PASS Test incorrect plane count.
FAIL Test stride and offset work. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
FAIL Test stride and offset with padding. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
PASS Test invalid stride.
FAIL Test address overflow. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test codedRect. assert_equals: allocationSize() expected 12 but got 0
FAIL Test empty rect. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test unaligned rect. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test left crop. assert_equals: allocationSize() expected 6 but got 0
FAIL Test invalid rect. assert_throws_js: function "() => frame.allocationSize(options)" did not throw
FAIL Test codedRect. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
PASS Test empty rect.
PASS Test unaligned rect.
FAIL Test left crop. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
PASS Test invalid rect.

@@ -1,4 +1,4 @@

FAIL Test copying I420 frame to SharedArrayBuffer. assert_equals: allocationSize() expected 12 but got 0
FAIL Test copying I420 frame to shared ArrayBufferView. assert_equals: allocationSize() expected 12 but got 0
FAIL Test copying I420 frame to SharedArrayBuffer. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
FAIL Test copying I420 frame to shared ArrayBufferView. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"

@@ -1,4 +1,4 @@

FAIL Test copying I420 frame to SharedArrayBuffer. assert_equals: allocationSize() expected 12 but got 0
FAIL Test copying I420 frame to shared ArrayBufferView. assert_equals: allocationSize() expected 12 but got 0
FAIL Test copying I420 frame to SharedArrayBuffer. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"
FAIL Test copying I420 frame to shared ArrayBufferView. promise_test: Unhandled rejection with value: object "TypeError: Unable to copy data"

@@ -293,16 +293,19 @@ ExceptionOr<Ref<WebCodecsVideoFrame>> WebCodecsVideoFrame::initializeFrameWithRe
}


ExceptionOr<size_t> WebCodecsVideoFrame::allocationSize(CopyToOptions&&)
ExceptionOr<size_t> WebCodecsVideoFrame::allocationSize(const CopyToOptions& options)
{
if (isDetached())
return Exception { InvalidStateError, "VideoFrame is detached"_s };

if (!m_format)
return Exception { NotSupportedError, "VideoFrame has no format"_s };

// FIXME: Implement.
return 0;
auto layoutOrException = parseVideoFrameCopyToOptions(*this, options);
if (layoutOrException.hasException())
return layoutOrException.releaseException();

return layoutOrException.returnValue().allocationSize;
}

void WebCodecsVideoFrame::copyTo(BufferSource&& source, CopyToOptions&& options, CopyToPromise&& promise)
@@ -106,10 +106,10 @@ class WebCodecsVideoFrame : public RefCounted<WebCodecsVideoFrame> {
VideoColorSpace* colorSpace() const { return m_colorSpace.get(); }

struct CopyToOptions {
DOMRectInit rect;
Vector<PlaneLayout> layout;
std::optional<DOMRectInit> rect;
std::optional<Vector<PlaneLayout>> layout;
};
ExceptionOr<size_t> allocationSize(CopyToOptions&&);
ExceptionOr<size_t> allocationSize(const CopyToOptions&);

using CopyToPromise = DOMPromiseDeferred<IDLSequence<IDLDictionary<PlaneLayout>>>;
void copyTo(BufferSource&&, CopyToOptions&&, CopyToPromise&&);
@@ -112,10 +112,10 @@ ExceptionOr<DOMRectInit> parseVisibleRect(const DOMRectInit& defaultRect, const
if (overrideRect) {
if (!overrideRect->width || !overrideRect->height)
return Exception { TypeError, "overrideRect is not valid"_s };
if (overrideRect->y + overrideRect->height > codedHeight)
return Exception { TypeError, "overrideRect is not valid"_s };
if (overrideRect->x + overrideRect->width > codedWidth)
return Exception { TypeError, "overrideRect is not valid"_s };
if (overrideRect->y + overrideRect->height > codedHeight)
return Exception { TypeError, "overrideRect is not valid"_s };
sourceRect = *overrideRect;
}
if (!verifyRectOffsetAlignment(format, sourceRect))
@@ -147,6 +147,24 @@ size_t videoPixelFormatToSampleByteSizePerPlane()
return 1;
}

static inline size_t sampleCountPerPixel(VideoPixelFormat format)
{
switch (format) {
case VideoPixelFormat::I420:
case VideoPixelFormat::I420A:
case VideoPixelFormat::I444:
case VideoPixelFormat::I422:
case VideoPixelFormat::NV12:
return 1;
case VideoPixelFormat::RGBA:
case VideoPixelFormat::RGBX:
case VideoPixelFormat::BGRA:
case VideoPixelFormat::BGRX:
return 4;
}
return 1;
}

size_t videoPixelFormatToSubSampling(VideoPixelFormat format, size_t planeNumber)
{
switch (format) {
@@ -172,6 +190,7 @@ ExceptionOr<CombinedPlaneLayout> computeLayoutAndAllocationSize(const DOMRectIni
if (layout && layout->size() != planeCount)
return Exception { TypeError, "layout size is invalid"_s };

size_t pixelSampleCount = sampleCountPerPixel(format);
size_t minAllocationSize = 0;
Vector<ComputedPlaneLayout> computedLayouts;
computedLayouts.reserveInitialCapacity(planeCount);
@@ -184,8 +203,8 @@ ExceptionOr<CombinedPlaneLayout> computeLayoutAndAllocationSize(const DOMRectIni
ComputedPlaneLayout computedLayout;
computedLayout.sourceTop = parsedRect.y / sampleHeight;
computedLayout.sourceHeight = parsedRect.height / sampleHeight;
computedLayout.sourceLeftBytes = parsedRect.x / sampleWidthBytes;
computedLayout.sourceWidthBytes = parsedRect.width / sampleWidthBytes;
computedLayout.sourceLeftBytes = pixelSampleCount * parsedRect.x / sampleWidthBytes;
computedLayout.sourceWidthBytes = pixelSampleCount * parsedRect.width / sampleWidthBytes;

if (layout) {
if (layout.value()[i].stride < computedLayout.sourceWidthBytes)
@@ -210,14 +229,15 @@ ExceptionOr<CombinedPlaneLayout> computeLayoutAndAllocationSize(const DOMRectIni
return CombinedPlaneLayout { minAllocationSize, WTFMove(computedLayouts) };
}

// https://w3c.github.io/webcodecs/#videoframe-parse-videoframecopytooptions
ExceptionOr<CombinedPlaneLayout> parseVideoFrameCopyToOptions(const WebCodecsVideoFrame& frame, const WebCodecsVideoFrame::CopyToOptions& options)
{
ASSERT(!frame.isDetached());
ASSERT(frame.format());

if (!verifyRectSizeAlignment(*frame.format(), options.rect))
if (options.rect && !verifyRectSizeAlignment(*frame.format(), *options.rect))
return Exception { TypeError, "rect size alignment is invalid"_s };

// Are we sure frame.visibleRect() is not null?
auto& visibleRect = *frame.visibleRect();
auto parsedRect = parseVisibleRect({ visibleRect.x(), visibleRect.y(), visibleRect.width(), visibleRect.height() }, options.rect, frame.codedWidth(), frame.codedHeight(), *frame.format());

@@ -218,6 +218,7 @@
}

Vector<PlaneLayout> planeLayouts;
// FIXME: Fill PlaneLayout
planeLayouts.append(PlaneLayout { });
callback(WTFMove(planeLayouts));
return;
@@ -250,6 +251,7 @@
std::memcpy(span.data(), planeA, planeASize);

Vector<PlaneLayout> planeLayouts;
// FIXME: Fill PlaneLayout
planeLayouts.append(PlaneLayout { });
callback(WTFMove(planeLayouts));
return;

0 comments on commit 60db1be

Please sign in to comment.