VP8 WebCoreDecompressionSession should attribute its IOSurfaces to the media player resource owner #26032

youennf · 2024-03-18T09:31:37Z

`fa9c6d7`

VP8 WebCoreDecompressionSession should attribute its IOSurfaces to the media player resource owner
https://bugs.webkit.org/show_bug.cgi?id=271144
rdar://123795173

Reviewed by Jean-Yves Avenard.

When WebCoreDecompressionSession uses a VideoDecoder to do its decoding, it uses an IOSurface pixel buffer pool.
We then need to attribute these buffers to the corresponding resource owner.
To do so, we are setting a resourceOwner in WebCoreDecompressionSession from VideoMediaSampleRenderer, which gets it from MediaPlayerPrivateWebM.
The WebCoreDecompressionSession is creating a VideoDecoder that is given the resourceOwner so that,
everytime we have a pixel buffer coming from the buffer pool, we then do the attribution.

We also do this for MediaPlayerPrivateMediaSourceAVFObjC's session.
This means piping the resource owner to MediaSourcePrivate -> SourceBufferPrivate.

* Source/WebCore/platform/VideoDecoder.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::load):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::ensureDecompressionSession):
* Source/WebCore/platform/graphics/avfoundation/objc/MediaSourcePrivateAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaSourcePrivateAVFObjC.mm:
(WebCore::MediaSourcePrivateAVFObjC::addSourceBuffer):
* Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.mm:
(WebCore::SourceBufferPrivateAVFObjC::setVideoRenderer):
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm:
(WebCore::MediaPlayerPrivateWebM::ensureLayer):
(WebCore::MediaPlayerPrivateWebM::ensureDecompressionSession):
* Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.h:
(WebCore::VideoMediaSampleRenderer::setResourceOwner):
* Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.mm:
(WebCore::VideoMediaSampleRenderer::initializeDecompressionSession):
* Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.h:
* Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm:
(WebCore::WebCoreDecompressionSession::initializeVideoDecoder):
* Source/WebCore/platform/libwebrtc/LibWebRTCVPXVideoDecoder.cpp:
(WebCore::LibWebRTCVPXInternalVideoDecoder::LibWebRTCVPXInternalVideoDecoder):
(WebCore::LibWebRTCVPXInternalVideoDecoder::Decoded):

Canonical link: https://commits.webkit.org/276270@main

0984310

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
❌ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 wincairo
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-skia
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 tv	✅ 🧪 mac-AS-debug-wk2	❌ 🧪 gtk-wk2
	✅ 🛠 tv-sim		~~🧪 api-gtk~~
✅ 🛠 🧪 merge	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2024-03-18T09:31:43Z

EWS run on previous version of this PR (hash 3bafd19)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
❌ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	~~🛠 wincairo~~
✅ 🧪 bindings	✅ 🛠 ios-sim	❌ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	~~🧪 api-mac~~	~~🧪 api-wpe~~
	⏳ 🧪 ios-wk2-wpt	~~🧪 mac-wk1~~	✅ 🛠 wpe-skia
	✅ 🧪 api-ios	~~🧪 mac-wk2~~	✅ 🛠 gtk
	✅ 🛠 tv	❌ 🧪 mac-AS-debug-wk2	~~🧪 gtk-wk2~~
	~~🛠 tv-sim~~		✅ 🧪 api-gtk
	~~🛠 watch~~
	✅ 🛠 watch-sim

jyavenard

this WebCoreDecompressionSession is only used if we are painting on a canvas ; and the buffer allocation is already handled in updateLastPixelBuffer()

The WebCoreDecompressionSession that needs change is the one managed by VideoMediaSampleRenderer

webkit-early-warning-system · 2024-03-18T10:02:34Z

EWS run on previous version of this PR (hash f6f9b6a)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
❌ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	~~🛠 wpe~~	~~🛠 wincairo~~
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~
✅ 🧪 webkitperl	~~🧪 ios-wk2~~	~~🧪 api-mac~~	~~🧪 api-wpe~~
	⏳ 🧪 ios-wk2-wpt	~~🧪 mac-wk1~~	✅ 🛠 wpe-skia
	⏳ 🧪 api-ios	~~🧪 mac-wk2~~	✅ 🛠 gtk
	✅ 🛠 tv	✅ 🧪 mac-AS-debug-wk2	~~🧪 gtk-wk2~~
	~~🛠 tv-sim~~		~~🧪 api-gtk~~
	~~🛠 watch~~
	✅ 🛠 watch-sim

webkit-early-warning-system · 2024-03-18T10:18:57Z

EWS run on previous version of this PR (hash 0b966ec)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
❌ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 wincairo
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	~~🧪 ios-wk2-wpt~~	✅ 🧪 mac-wk1	✅ 🛠 wpe-skia
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 tv	✅ 🧪 mac-AS-debug-wk2	~~🧪 gtk-wk2~~
	✅ 🛠 tv-sim		~~🧪 api-gtk~~
	✅ 🛠 watch
	✅ 🛠 watch-sim

Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.h

webkit-early-warning-system · 2024-03-18T11:59:46Z

EWS run on current version of this PR (hash 0984310)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
❌ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 wincairo
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-skia
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 tv	✅ 🧪 mac-AS-debug-wk2	❌ 🧪 gtk-wk2
	✅ 🛠 tv-sim		~~🧪 api-gtk~~
✅ 🛠 🧪 merge	✅ 🛠 watch
	✅ 🛠 watch-sim

youennf · 2024-03-18T14:30:10Z

gtk test failure unrelated.

…e media player resource owner https://bugs.webkit.org/show_bug.cgi?id=271144 rdar://123795173 Reviewed by Jean-Yves Avenard. When WebCoreDecompressionSession uses a VideoDecoder to do its decoding, it uses an IOSurface pixel buffer pool. We then need to attribute these buffers to the corresponding resource owner. To do so, we are setting a resourceOwner in WebCoreDecompressionSession from VideoMediaSampleRenderer, which gets it from MediaPlayerPrivateWebM. The WebCoreDecompressionSession is creating a VideoDecoder that is given the resourceOwner so that, everytime we have a pixel buffer coming from the buffer pool, we then do the attribution. We also do this for MediaPlayerPrivateMediaSourceAVFObjC's session. This means piping the resource owner to MediaSourcePrivate -> SourceBufferPrivate. * Source/WebCore/platform/VideoDecoder.h: * Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm: (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::load): (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::ensureDecompressionSession): * Source/WebCore/platform/graphics/avfoundation/objc/MediaSourcePrivateAVFObjC.h: * Source/WebCore/platform/graphics/avfoundation/objc/MediaSourcePrivateAVFObjC.mm: (WebCore::MediaSourcePrivateAVFObjC::addSourceBuffer): * Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.h: * Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.mm: (WebCore::SourceBufferPrivateAVFObjC::setVideoRenderer): * Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm: (WebCore::MediaPlayerPrivateWebM::ensureLayer): (WebCore::MediaPlayerPrivateWebM::ensureDecompressionSession): * Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.h: (WebCore::VideoMediaSampleRenderer::setResourceOwner): * Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.mm: (WebCore::VideoMediaSampleRenderer::initializeDecompressionSession): * Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.h: * Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm: (WebCore::WebCoreDecompressionSession::initializeVideoDecoder): * Source/WebCore/platform/libwebrtc/LibWebRTCVPXVideoDecoder.cpp: (WebCore::LibWebRTCVPXInternalVideoDecoder::LibWebRTCVPXInternalVideoDecoder): (WebCore::LibWebRTCVPXInternalVideoDecoder::Decoded): Canonical link: https://commits.webkit.org/276270@main

webkit-commit-queue · 2024-03-18T15:12:33Z

Committed 276270@main (fa9c6d7): https://commits.webkit.org/276270@main

Reviewed commits have been landed. Closing PR #26032 and removing active labels.

youennf self-assigned this Mar 18, 2024

youennf added the WebRTC For bugs in WebRTC label Mar 18, 2024

youennf requested a review from jyavenard March 18, 2024 09:34

jyavenard reviewed Mar 18, 2024

View reviewed changes

youennf force-pushed the eng/VP8-WebCoreDecompressionSession-should-attribute-its-IOSurfaces-to-the-media-player-resource-owner branch from 3bafd19 to f6f9b6a Compare March 18, 2024 09:58

youennf force-pushed the eng/VP8-WebCoreDecompressionSession-should-attribute-its-IOSurfaces-to-the-media-player-resource-owner branch from f6f9b6a to 0b966ec Compare March 18, 2024 10:18

youennf marked this pull request as ready for review March 18, 2024 10:19

youennf requested a review from jyavenard March 18, 2024 10:19

jyavenard reviewed Mar 18, 2024

View reviewed changes

Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.h Show resolved Hide resolved

youennf force-pushed the eng/VP8-WebCoreDecompressionSession-should-attribute-its-IOSurfaces-to-the-media-player-resource-owner branch from 0b966ec to 0984310 Compare March 18, 2024 11:59

jyavenard approved these changes Mar 18, 2024

View reviewed changes

youennf added the merge-queue Applied to send a pull request to merge-queue label Mar 18, 2024

webkit-ews-buildbot added merging-blocked Applied to prevent a change from being merged and removed merge-queue Applied to send a pull request to merge-queue labels Mar 18, 2024

youennf removed the merging-blocked Applied to prevent a change from being merged label Mar 18, 2024

youennf added the merge-queue Applied to send a pull request to merge-queue label Mar 18, 2024

webkit-commit-queue force-pushed the eng/VP8-WebCoreDecompressionSession-should-attribute-its-IOSurfaces-to-the-media-player-resource-owner branch from 0984310 to fa9c6d7 Compare March 18, 2024 15:12

webkit-commit-queue merged commit fa9c6d7 into WebKit:main Mar 18, 2024

webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Mar 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VP8 WebCoreDecompressionSession should attribute its IOSurfaces to the media player resource owner #26032

VP8 WebCoreDecompressionSession should attribute its IOSurfaces to the media player resource owner #26032

youennf commented Mar 18, 2024 •

edited by webkit-early-warning-system

Loading

webkit-early-warning-system commented Mar 18, 2024 •

edited

Loading

jyavenard left a comment

webkit-early-warning-system commented Mar 18, 2024 •

edited

Loading

webkit-early-warning-system commented Mar 18, 2024 •

edited

Loading

webkit-early-warning-system commented Mar 18, 2024 •

edited

Loading

youennf commented Mar 18, 2024

webkit-commit-queue commented Mar 18, 2024

VP8 WebCoreDecompressionSession should attribute its IOSurfaces to the media player resource owner #26032

VP8 WebCoreDecompressionSession should attribute its IOSurfaces to the media player resource owner #26032

Conversation

youennf commented Mar 18, 2024 • edited by webkit-early-warning-system Loading

fa9c6d7

webkit-early-warning-system commented Mar 18, 2024 • edited Loading

jyavenard left a comment

Choose a reason for hiding this comment

webkit-early-warning-system commented Mar 18, 2024 • edited Loading

webkit-early-warning-system commented Mar 18, 2024 • edited Loading

webkit-early-warning-system commented Mar 18, 2024 • edited Loading

youennf commented Mar 18, 2024

webkit-commit-queue commented Mar 18, 2024

youennf commented Mar 18, 2024 •

edited by webkit-early-warning-system

Loading

`fa9c6d7`

webkit-early-warning-system commented Mar 18, 2024 •

edited

Loading

webkit-early-warning-system commented Mar 18, 2024 •

edited

Loading

webkit-early-warning-system commented Mar 18, 2024 •

edited

Loading

webkit-early-warning-system commented Mar 18, 2024 •

edited

Loading