Skip to content

Conversation

@jyavenard
Copy link
Member

@jyavenard jyavenard commented Jan 14, 2026

ce294ca

Make HTMLVideoElement::bitmapImageForCurrentTime() asynchronous by having it return a NativePromise
https://bugs.webkit.org/show_bug.cgi?id=305365
rdar://168044436

Reviewed by Jer Noble.

We make HTMLVideoElement::bitmapImageForCurrentTime asynchronous by having
it return a NativePromise.
The retrieval of the current MediaPlayer's videoframe required two sync calls.
one to retrieve the RemoteVideoFrameProxyProperties and then one to create
the NativeImage from it.
The ShareableBitmap was created from a NativeImage, itself built from a VideoFrame.
Such constructions could only work on the main thread and required sync calls to the GPU
process.
As such we add IPC APIs to directly return a ShareableBitmap::Handle from the GPU process
where the ShareableBitmap will be created from the MediaPlayer's NativeImage (or AudioVideoRenderer
if the MediaPlayer is running in the content process).

We add an async version of AudioVideoRenderer:currentNativeImage() : currentNativeImageAsync()
which will return a NativePromise.
The MediaPlayerPrivateWebM and MediaPlayerPrivateMediaSourceAVFObjC are made to use this
new API when running in the content process.

No change in observable behaviour.
* Source/WebCore/html/HTMLVideoElement.cpp:
(WebCore::HTMLVideoElement::bitmapImageForCurrentTimeSync const):
(WebCore::HTMLVideoElement::bitmapImageForCurrentTime const):
* Source/WebCore/html/HTMLVideoElement.h:
* Source/WebCore/platform/graphics/AudioVideoRenderer.h:
* Source/WebCore/platform/graphics/MediaPlayer.cpp:
(WebCore::MediaPlayer::bitmapImageForCurrentTimeSync):
(WebCore::MediaPlayer::bitmapImageForCurrentTime):
* Source/WebCore/platform/graphics/MediaPlayer.h:
* Source/WebCore/platform/graphics/MediaPlayerPrivate.cpp:
(WebCore::MediaPlayerPrivateInterface::bitmapFromImage):
(WebCore::MediaPlayerPrivateInterface::bitmapImageForCurrentTimeSync):
(WebCore::MediaPlayerPrivateInterface::bitmapImageForCurrentTime):
* Source/WebCore/platform/graphics/MediaPlayerPrivate.h:
* Source/WebCore/platform/graphics/avfoundation/AudioVideoRendererAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/AudioVideoRendererAVFObjC.mm:
(WebCore::AudioVideoRendererAVFObjC::currentBitmapImage const):
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::bitmapImageForCurrentTime):
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.h:
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm:
(WebCore::MediaPlayerPrivateWebM::bitmapImageForCurrentTime):
* Source/WebKit/GPUProcess/media/RemoteAudioVideoRendererProxyManager.cpp:
(WebKit::RemoteAudioVideoRendererProxyManager::currentBitmapImage const):
* Source/WebKit/GPUProcess/media/RemoteAudioVideoRendererProxyManager.h:
* Source/WebKit/GPUProcess/media/RemoteAudioVideoRendererProxyManager.messages.in:
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.cpp:
(WebKit::RemoteMediaPlayerProxy::bitmapImageForCurrentTime):
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.h:
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.messages.in:
* Source/WebKit/WebProcess/GPU/media/AudioVideoRendererRemote.cpp:
(WebKit::AudioVideoRendererRemote::currentBitmapImage const):
* Source/WebKit/WebProcess/GPU/media/AudioVideoRendererRemote.h:
* Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp:
(WebKit::MediaPlayerPrivateRemote::bitmapImageForCurrentTime):
* Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.h:
* Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:
(WebKit::createShareableBitmap):
(WebKit::createShareableBitmapAsync):
* Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.h:
(WebKit::createShareableBitmapAsync):
* Source/WebKit/WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::requestTextRecognition):
(WebKit::WebPage::beginTextRecognitionForVideoInElementFullScreen):
(WebKit::WebPage::cancelTextRecognitionForVideoInElementFullScreen):
* Source/WebKit/WebProcess/WebPage/WebPage.h:

Canonical link: https://commits.webkit.org/306386@main

87864fe

Misc iOS, visionOS, tvOS & watchOS macOS Linux Windows Apple Internal
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe 🛠 win ⏳ 🛠 ios-apple
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug 🧪 wpe-wk2 🧪 win-tests ⏳ 🛠 mac-apple
✅ 🧪 webkitperl ✅ 🧪 ios-wk2 🧪 api-mac ✅ 🧪 api-wpe ⏳ 🛠 vision-apple
🧪 ios-wk2-wpt 🧪 api-mac-debug ⏳ wpe-libwebrtc
🧪 api-ios 🧪 mac-wk1 ✅ 🛠 gtk
🛠 vision ⏳ 🧪 mac-wk2 🧪 gtk-wk2
✅ 🛠 vision-sim 🧪 mac-AS-debug-wk2 🧪 api-gtk
✅ 🛠 🧪 merge loading 🧪 vision-wk2 🧪 mac-wk2-stress 🛠 playstation
✅ 🛠 tv ⏳ 🧪 mac-intel-wk2
🛠 tv-sim 🛠 mac-safer-cpp
🛠 watch
✅ 🛠 watch-sim

@jyavenard jyavenard self-assigned this Jan 14, 2026
@jyavenard jyavenard added the Media Bugs related to the HTML 5 Media elements. label Jan 14, 2026
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Jan 14, 2026
@webkit-ews-buildbot
Copy link
Collaborator

Safer C++ Build #74561 (09065c2)

❌ Found 3 failing files with 6 issues. Please address these issues before landing. See WebKit Guidelines for Safer C++ Programming.
(cc @rniwa)

@jyavenard jyavenard removed the merging-blocked Applied to prevent a change from being merged label Jan 27, 2026
@jyavenard jyavenard force-pushed the eng/Make-HTMLVideoElement-bitmapImageForCurrentTime-asynchronous-by-having-it-return-a-NativePromise branch from 09065c2 to dcb468e Compare January 27, 2026 13:44
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Jan 27, 2026
@jyavenard jyavenard removed the merging-blocked Applied to prevent a change from being merged label Jan 27, 2026
@jyavenard jyavenard force-pushed the eng/Make-HTMLVideoElement-bitmapImageForCurrentTime-asynchronous-by-having-it-return-a-NativePromise branch from dcb468e to f3e9869 Compare January 27, 2026 14:20
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Jan 27, 2026
@jyavenard jyavenard force-pushed the eng/Make-HTMLVideoElement-bitmapImageForCurrentTime-asynchronous-by-having-it-return-a-NativePromise branch from f3e9869 to 0695b13 Compare January 27, 2026 22:47
@jyavenard jyavenard requested review from a team and youennf January 27, 2026 23:32
@jyavenard jyavenard added merge-queue Applied to send a pull request to merge-queue and removed merging-blocked Applied to prevent a change from being merged labels Jan 28, 2026
@jyavenard jyavenard force-pushed the eng/Make-HTMLVideoElement-bitmapImageForCurrentTime-asynchronous-by-having-it-return-a-NativePromise branch from 0695b13 to 54d798c Compare January 29, 2026 00:49
@webkit-ews-buildbot webkit-ews-buildbot added merging-blocked Applied to prevent a change from being merged and removed merge-queue Applied to send a pull request to merge-queue labels Jan 29, 2026
@jyavenard jyavenard force-pushed the eng/Make-HTMLVideoElement-bitmapImageForCurrentTime-asynchronous-by-having-it-return-a-NativePromise branch from 54d798c to b834263 Compare January 29, 2026 01:00
@jyavenard jyavenard force-pushed the eng/Make-HTMLVideoElement-bitmapImageForCurrentTime-asynchronous-by-having-it-return-a-NativePromise branch from b834263 to 87864fe Compare January 29, 2026 03:02
@jyavenard jyavenard added merge-queue Applied to send a pull request to merge-queue and removed merging-blocked Applied to prevent a change from being merged labels Jan 29, 2026
…ving it return a NativePromise

https://bugs.webkit.org/show_bug.cgi?id=305365
rdar://168044436

Reviewed by Jer Noble.

We make HTMLVideoElement::bitmapImageForCurrentTime asynchronous by having
it return a NativePromise.
The retrieval of the current MediaPlayer's videoframe required two sync calls.
one to retrieve the RemoteVideoFrameProxyProperties and then one to create
the NativeImage from it.
The ShareableBitmap was created from a NativeImage, itself built from a VideoFrame.
Such constructions could only work on the main thread and required sync calls to the GPU
process.
As such we add IPC APIs to directly return a ShareableBitmap::Handle from the GPU process
where the ShareableBitmap will be created from the MediaPlayer's NativeImage (or AudioVideoRenderer
if the MediaPlayer is running in the content process).

We add an async version of AudioVideoRenderer:currentNativeImage() : currentNativeImageAsync()
which will return a NativePromise.
The MediaPlayerPrivateWebM and MediaPlayerPrivateMediaSourceAVFObjC are made to use this
new API when running in the content process.

No change in observable behaviour.
* Source/WebCore/html/HTMLVideoElement.cpp:
(WebCore::HTMLVideoElement::bitmapImageForCurrentTimeSync const):
(WebCore::HTMLVideoElement::bitmapImageForCurrentTime const):
* Source/WebCore/html/HTMLVideoElement.h:
* Source/WebCore/platform/graphics/AudioVideoRenderer.h:
* Source/WebCore/platform/graphics/MediaPlayer.cpp:
(WebCore::MediaPlayer::bitmapImageForCurrentTimeSync):
(WebCore::MediaPlayer::bitmapImageForCurrentTime):
* Source/WebCore/platform/graphics/MediaPlayer.h:
* Source/WebCore/platform/graphics/MediaPlayerPrivate.cpp:
(WebCore::MediaPlayerPrivateInterface::bitmapFromImage):
(WebCore::MediaPlayerPrivateInterface::bitmapImageForCurrentTimeSync):
(WebCore::MediaPlayerPrivateInterface::bitmapImageForCurrentTime):
* Source/WebCore/platform/graphics/MediaPlayerPrivate.h:
* Source/WebCore/platform/graphics/avfoundation/AudioVideoRendererAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/AudioVideoRendererAVFObjC.mm:
(WebCore::AudioVideoRendererAVFObjC::currentBitmapImage const):
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::bitmapImageForCurrentTime):
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.h:
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm:
(WebCore::MediaPlayerPrivateWebM::bitmapImageForCurrentTime):
* Source/WebKit/GPUProcess/media/RemoteAudioVideoRendererProxyManager.cpp:
(WebKit::RemoteAudioVideoRendererProxyManager::currentBitmapImage const):
* Source/WebKit/GPUProcess/media/RemoteAudioVideoRendererProxyManager.h:
* Source/WebKit/GPUProcess/media/RemoteAudioVideoRendererProxyManager.messages.in:
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.cpp:
(WebKit::RemoteMediaPlayerProxy::bitmapImageForCurrentTime):
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.h:
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.messages.in:
* Source/WebKit/WebProcess/GPU/media/AudioVideoRendererRemote.cpp:
(WebKit::AudioVideoRendererRemote::currentBitmapImage const):
* Source/WebKit/WebProcess/GPU/media/AudioVideoRendererRemote.h:
* Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp:
(WebKit::MediaPlayerPrivateRemote::bitmapImageForCurrentTime):
* Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.h:
* Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:
(WebKit::createShareableBitmap):
(WebKit::createShareableBitmapAsync):
* Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.h:
(WebKit::createShareableBitmapAsync):
* Source/WebKit/WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::requestTextRecognition):
(WebKit::WebPage::beginTextRecognitionForVideoInElementFullScreen):
(WebKit::WebPage::cancelTextRecognitionForVideoInElementFullScreen):
* Source/WebKit/WebProcess/WebPage/WebPage.h:

Canonical link: https://commits.webkit.org/306386@main
@webkit-commit-queue webkit-commit-queue force-pushed the eng/Make-HTMLVideoElement-bitmapImageForCurrentTime-asynchronous-by-having-it-return-a-NativePromise branch from 87864fe to ce294ca Compare January 29, 2026 03:37
@webkit-commit-queue webkit-commit-queue merged commit ce294ca into WebKit:main Jan 29, 2026
@webkit-commit-queue
Copy link
Collaborator

Committed 306386@main (ce294ca): https://commits.webkit.org/306386@main

Reviewed commits have been landed. Closing PR #56556 and removing active labels.

@webkit-commit-queue webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Jan 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Media Bugs related to the HTML 5 Media elements.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants