Skip to content

Conversation

@lauromoura
Copy link
Contributor

@lauromoura lauromoura commented Sep 30, 2025

50c1d97

[WPE][WebDriver] Support taking the screenshots from the UIProcess
https://bugs.webkit.org/show_bug.cgi?id=299732

Reviewed by Carlos Garcia Campos.

Currently, WPE's implentation of taking screenshots rely on asking the
root `RenderLayer` on the WebProcess to paint its content. While it
worked fine for most elements, this approach could lead to differences
from the final composited frame that we send back to the UIProcess,
especially for some kinds of accelerated content, like animated 3D CSS
filters.

On top of that, the recent ANGLE bump in 295416@main included a stricter
check on ReadPixels that affected taking screenshots of WebGL scenes on
WebGL 1.0 environments, as the GraphicsContextGLAngle used it to read
the WebGL scenes contents for this code path.

To address these issues, this commits adds support for taking
screenshots on the UIProcess, by reading the last buffer committed to
the system.

* Source/WebKit/Shared/WebBackForwardListItem.h:
* Source/WebKit/SourcesWPE.txt:
* Source/WebKit/UIProcess/API/wpe/PageClientImpl.cpp:
(WebKit::PageClientImpl::takeViewSnapshot):
* Source/WebKit/UIProcess/API/wpe/PageClientImpl.h:
* Source/WebKit/UIProcess/API/wpe/WPEWebViewPlatform.cpp:
(WKWPE::ViewPlatform::takeViewSnapshot):
* Source/WebKit/UIProcess/API/wpe/WPEWebViewPlatform.h:
* Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp:
(WebKit::WebAutomationSession::takeScreenshot):
* Source/WebKit/UIProcess/Automation/skia/WebAutomationSessionSkia.cpp:
(WebKit::base64EncodedPNGData):
(WebKit::WebAutomationSession::platformGetBase64EncodedPNGData):
* Source/WebKit/UIProcess/PageClient.h:
* Source/WebKit/UIProcess/ViewSnapshotStore.cpp:
* Source/WebKit/UIProcess/ViewSnapshotStore.h:
(WebKit::ViewSnapshot::image const):
* Source/WebKit/UIProcess/WebPageProxy.cpp:
* Source/WebKit/UIProcess/WebPageProxy.h:
* Source/WebKit/UIProcess/skia/ViewSnapshotSkia.cpp: Copied from Source/WebKit/UIProcess/Automation/skia/WebAutomationSessionSkia.cpp.
(WebKit::ViewSnapshot::create):
(WebKit::ViewSnapshot::ViewSnapshot):
(WebKit::ViewSnapshot::hasImage const):
(WebKit::ViewSnapshot::clearImage):
(WebKit::ViewSnapshot::estimatedImageSizeInBytes const):
(WebKit::ViewSnapshot::size const):
* Source/WebKit/UIProcess/wpe/AcceleratedBackingStore.cpp:
(WebKit::getImageInfoFromBuffer):
(WebKit::saveBufferSnapshot):
(WebKit::AcceleratedBackingStore::takeSnapshot):
* Source/WebKit/UIProcess/wpe/AcceleratedBackingStore.h:
* WebDriverTests/TestExpectations.json:

Canonical link: https://commits.webkit.org/301948@main

965e9ff

Misc iOS, visionOS, tvOS & watchOS macOS Linux Windows
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe ✅ 🛠 win
✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug ✅ 🧪 wpe-wk2 ❌ 🧪 win-tests
✅ 🧪 webkitperl ✅ 🧪 ios-wk2 ✅ 🧪 api-mac ✅ 🧪 api-wpe
✅ 🧪 ios-wk2-wpt ✅ 🛠 wpe-cairo
✅ 🧪 api-ios ✅ 🧪 mac-wk2 ✅ 🛠 gtk
✅ 🛠 vision ✅ 🧪 mac-AS-debug-wk2 ✅ 🧪 gtk-wk2
✅ 🛠 vision-sim ✅ 🧪 mac-wk2-stress ✅ 🧪 api-gtk
✅ 🛠 🧪 merge ✅ 🧪 vision-wk2 ✅ 🧪 mac-intel-wk2 ✅ 🛠 playstation
✅ 🛠 tv ✅ 🛠 mac-safer-cpp
✅ 🛠 tv-sim
✅ 🛠 watch
✅ 🛠 watch-sim

@lauromoura lauromoura self-assigned this Sep 30, 2025
@lauromoura lauromoura added the WPE WebKit WebKit WPE component label Sep 30, 2025
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Sep 30, 2025
@lauromoura lauromoura removed the merging-blocked Applied to prevent a change from being merged label Sep 30, 2025
@lauromoura lauromoura force-pushed the eng/WPE-WebDriver-Support-taking-the-screenshots-from-the-UIProcess branch from 92d0a63 to 69dceed Compare September 30, 2025 14:44
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Sep 30, 2025
@lauromoura lauromoura removed the merging-blocked Applied to prevent a change from being merged label Oct 6, 2025
@lauromoura lauromoura force-pushed the eng/WPE-WebDriver-Support-taking-the-screenshots-from-the-UIProcess branch from 69dceed to 7d9bd90 Compare October 6, 2025 03:58
@lauromoura lauromoura marked this pull request as ready for review October 6, 2025 04:01
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Oct 6, 2025
Comment on lines 574 to 576
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be endif, we want to fail the same way when building with the new api but the old one is used.

Comment on lines 673 to 674
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Comment on lines 77 to 79
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use SkData::bytesSpan()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I'll also refactor this funtion to use the base64EncodedPNGData(SkImage&) defined above and use byteSpan there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this async version, the previous call to callAfterNextPresentationUpdate should ensure we have a pending buffer ready, I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, using m_commitedBuffer synchronously seems to be working fine. Thanks for the suggestion.

Comment on lines 42 to 44
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we have to call ViewSnapshotStore::singleton().didAddImageToSnapshot(*this) ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And ViewSnapshotStore::singleton().willRemoveImageFromSnapshot(*this) here?

Comment on lines 66 to 70
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could probably use m_image->imageInfo().computeMinByteSize()

Comment on lines 260 to 263
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should try to not use this async approach and get the snapshot after the call after next presenation update using the pending buffer if there's any or committed buffer otherwise.

Comment on lines 42 to 43
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed it because it relied on taking the screenshots synchronously. I'll test this approach, reading the last m_commitedBuffer, so we can avoid this guard.

@lauromoura lauromoura removed the merging-blocked Applied to prevent a change from being merged label Oct 7, 2025
@lauromoura lauromoura force-pushed the eng/WPE-WebDriver-Support-taking-the-screenshots-from-the-UIProcess branch from 7d9bd90 to 6b86043 Compare October 7, 2025 03:57
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Oct 7, 2025
@lauromoura lauromoura removed the merging-blocked Applied to prevent a change from being merged label Oct 7, 2025
@lauromoura lauromoura force-pushed the eng/WPE-WebDriver-Support-taking-the-screenshots-from-the-UIProcess branch from 6b86043 to 284ef69 Compare October 7, 2025 04:59
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Oct 7, 2025
@lauromoura lauromoura removed the merging-blocked Applied to prevent a change from being merged label Oct 8, 2025
@lauromoura lauromoura force-pushed the eng/WPE-WebDriver-Support-taking-the-screenshots-from-the-UIProcess branch from 284ef69 to 4f47248 Compare October 8, 2025 03:33
@lauromoura lauromoura force-pushed the eng/WPE-WebDriver-Support-taking-the-screenshots-from-the-UIProcess branch from 4f47248 to b2563f6 Compare October 8, 2025 14:00
@lauromoura
Copy link
Contributor Author

@carlosgcampos I'm afraid the #if checks still seem to be a bit confusing.

Maybe ViewSnapshotStoreGtk3.cpp could be renamed to ViewSnapshotCairo.cpp, alongside adding some ENABLE(VIEWSNAPSHOT_SKIA) and ENABLE(VIEWSNAPSHOT_CAIRO) helpers? WDYT?

Comment on lines 2616 to 2628
#elif PLATFORM(WPE) && USE(SKIA)
Function<void(WebPageProxy&, std::optional<WebCore::IntRect>&&, CommandCallback<String>&&)> takeViewSnapshot = [](WebPageProxy& page, std::optional<WebCore::IntRect>&& rect, CommandCallback<String>&& callback) {
// FIXME Now that we're adding a takeViewSnapshotAsync, should we still wait for the next presentation update?
page.callAfterNextPresentationUpdate([page = Ref { page }, rect = WTFMove(rect), callback = WTFMove(callback)] () mutable {
auto snapshot = page->takeViewSnapshotWithErrorDetail(WTFMove(rect));
ASYNC_FAIL_WITH_PREDEFINED_ERROR_AND_DETAILS_IF(!snapshot, InternalError, snapshot.error());

std::optional<String> base64EncodedData = platformGetBase64EncodedPNGData(snapshot.value().get());
ASYNC_FAIL_WITH_PREDEFINED_ERROR_AND_DETAILS_IF(!base64EncodedData, InternalError, "Failed to extract base64 data from screenshot."_s);

callback(base64EncodedData.value());
});
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a leftover from previous patch? or is there any difference with the cocoa/gtk implementation? Can't use use the same?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was related to the takeViewSnapshotWithErrorDetail approach. I've updated the patch to make WPE use the same RefPtr in its PageClient instead of Expected for now, and reuse this same callback.

In a follow-up commit for https://webkit.org/b/300271, I'll switch the return type in the PageClient/WebPageProxy for all ports to forward the error.

Comment on lines 232 to 233
if (!m_committedBuffer) [[unlikely]]
return makeUnexpected("No commited buffer to create snapshot from"_s);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't have a committed buffer, but there's a pending one, we should use the pending.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. But should we do any kind of check regarding the fences to read from the pending buffer?

Comment on lines 398 to 400
#if PLATFORM(WPE) && USE(SKIA)
virtual Expected<Ref<ViewSnapshot>, String> takeViewSnapshotWithErrorDetail(std::optional<WebCore::IntRect>&&) = 0;
#endif
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does wpe need the error details and other ports don't?

Comment on lines 13652 to 13653
// FIXME log error
return nullptr;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we add a specific method for WPE to get the error details and then we don't use them? I think it's better to use the existing method and in a follow up change it to return the error message but for all the ports (and actually use it)

if (!pageClient)
return makeUnexpected("No PageClient available to handle snapshot request"_s);

return pageClient->takeViewSnapshotWithErrorDetail(WTFMove(clipRect));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah! I see it's used in this case. I still don't see why this is wpe specific.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned in the previous reply, this was postponed to a follow-up commit, in https//webkit.org/b/300271

@lauromoura lauromoura force-pushed the eng/WPE-WebDriver-Support-taking-the-screenshots-from-the-UIProcess branch from b2563f6 to 965e9ff Compare October 17, 2025 04:58
@lauromoura lauromoura added the merge-queue Applied to send a pull request to merge-queue label Oct 22, 2025
https://bugs.webkit.org/show_bug.cgi?id=299732

Reviewed by Carlos Garcia Campos.

Currently, WPE's implentation of taking screenshots rely on asking the
root `RenderLayer` on the WebProcess to paint its content. While it
worked fine for most elements, this approach could lead to differences
from the final composited frame that we send back to the UIProcess,
especially for some kinds of accelerated content, like animated 3D CSS
filters.

On top of that, the recent ANGLE bump in 295416@main included a stricter
check on ReadPixels that affected taking screenshots of WebGL scenes on
WebGL 1.0 environments, as the GraphicsContextGLAngle used it to read
the WebGL scenes contents for this code path.

To address these issues, this commits adds support for taking
screenshots on the UIProcess, by reading the last buffer committed to
the system.

* Source/WebKit/Shared/WebBackForwardListItem.h:
* Source/WebKit/SourcesWPE.txt:
* Source/WebKit/UIProcess/API/wpe/PageClientImpl.cpp:
(WebKit::PageClientImpl::takeViewSnapshot):
* Source/WebKit/UIProcess/API/wpe/PageClientImpl.h:
* Source/WebKit/UIProcess/API/wpe/WPEWebViewPlatform.cpp:
(WKWPE::ViewPlatform::takeViewSnapshot):
* Source/WebKit/UIProcess/API/wpe/WPEWebViewPlatform.h:
* Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp:
(WebKit::WebAutomationSession::takeScreenshot):
* Source/WebKit/UIProcess/Automation/skia/WebAutomationSessionSkia.cpp:
(WebKit::base64EncodedPNGData):
(WebKit::WebAutomationSession::platformGetBase64EncodedPNGData):
* Source/WebKit/UIProcess/PageClient.h:
* Source/WebKit/UIProcess/ViewSnapshotStore.cpp:
* Source/WebKit/UIProcess/ViewSnapshotStore.h:
(WebKit::ViewSnapshot::image const):
* Source/WebKit/UIProcess/WebPageProxy.cpp:
* Source/WebKit/UIProcess/WebPageProxy.h:
* Source/WebKit/UIProcess/skia/ViewSnapshotSkia.cpp: Copied from Source/WebKit/UIProcess/Automation/skia/WebAutomationSessionSkia.cpp.
(WebKit::ViewSnapshot::create):
(WebKit::ViewSnapshot::ViewSnapshot):
(WebKit::ViewSnapshot::hasImage const):
(WebKit::ViewSnapshot::clearImage):
(WebKit::ViewSnapshot::estimatedImageSizeInBytes const):
(WebKit::ViewSnapshot::size const):
* Source/WebKit/UIProcess/wpe/AcceleratedBackingStore.cpp:
(WebKit::getImageInfoFromBuffer):
(WebKit::saveBufferSnapshot):
(WebKit::AcceleratedBackingStore::takeSnapshot):
* Source/WebKit/UIProcess/wpe/AcceleratedBackingStore.h:
* WebDriverTests/TestExpectations.json:

Canonical link: https://commits.webkit.org/301948@main
@webkit-commit-queue webkit-commit-queue force-pushed the eng/WPE-WebDriver-Support-taking-the-screenshots-from-the-UIProcess branch from 965e9ff to 50c1d97 Compare October 22, 2025 15:23
@webkit-commit-queue
Copy link
Collaborator

Committed 301948@main (50c1d97): https://commits.webkit.org/301948@main

Reviewed commits have been landed. Closing PR #51535 and removing active labels.

@webkit-commit-queue webkit-commit-queue merged commit 50c1d97 into WebKit:main Oct 22, 2025
@webkit-commit-queue webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Oct 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

WPE WebKit WebKit WPE component

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants