8357584: [XWayland] [OL10] Robot.mousePress() is delivered to wrong place #1834

azvegint · 2025-06-18T15:33:45Z

This changeset introduces an adapted version of the OpenJDK fix for mouse and keyboard interactions with Robot.
More info about the issue itself is available in the PR description

In short, the currently used XTest for keyboard and mouse interactions may not be suitable for automated testing at some point, as it may require user confirmation to control the mouse or keyboard from time to time.

This fix adds support for the Remote Desktop XDG portal.
This allows us to control the keyboard and mouse with Robot on Wayland, even outside the XWayland server (e.g., clicking on window headers and Wayland native apps).

The remote desktop for Robot is enabled by default on GnomeShell 47+
It can be enabled manually by setting the javafx.robot.screenshotMethod system property to dbusRemoteDesktop (e.g. it works on Ubuntu 24.04 with GnomeShell 46)
The key handling might still have bugs.
The main goal is to add this new functionality. If there are small issues that can't be solved right away, I will prefer to fix them in follow up fixes.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed (2 reviews required, with at least 1 Reviewer, 1 Author)

Issue

JDK-8357584: [XWayland] [OL10] Robot.mousePress() is delivered to wrong place (Bug - P3)

Reviewers

Lukasz Kostyra (@lukostyra - Reviewer)
Ambarish Rapte (@arapte - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jfx.git pull/1834/head:pull/1834
$ git checkout pull/1834

Update a local copy of the PR:
$ git checkout pull/1834
$ git pull https://git.openjdk.org/jfx.git pull/1834/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 1834

View PR using the GUI difftool:
$ git pr show -t 1834

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jfx/pull/1834.diff

Using Webrev

Link to Webrev Comment

…lace

bridgekeeper · 2025-06-18T15:34:00Z

👋 Welcome back azvegint! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2025-06-18T15:35:07Z

@azvegint This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8357584: [XWayland] [OL10] Robot.mousePress() is delivered to wrong place

Reviewed-by: lkostyra, arapte

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 8 new commits pushed to the master branch:

0270847: 8351867: No UI changes while iconified
b9dd4de: 8357393: RichTextArea: fails to properly save text attributes
d4cb730: 8356652: Input field ignores custom input source characters
... and 5 more: https://git.openjdk.org/jfx/compare/48282b1067e37092916a9f7f0edafa66a43cfb5a...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

mlbridge · 2025-06-18T15:39:16Z

Webrevs

00: Full (df733e7c)

kevinrushforth · 2025-06-18T17:28:09Z

Reviewers: @kevinrushforth @lukostyra

/reviewers 2

openjdk · 2025-06-18T17:28:41Z

@kevinrushforth
The total number of required reviews for this PR (including the jcheck configuration and the last /reviewers command) is now set to 2 (with at least 1 Reviewer, 1 Author).

kevinrushforth · 2025-06-18T22:12:00Z

@arapte Would you also be able to look into this in case I am not able to?

lukostyra

Code looks good on first pass. I've given this quite a bit of testing both on Ubuntu 24.04 (Gnome 46) and Fedora 21 (Gnome 47) and the results are mostly similar. There's a small collection of tests failing on my machine, but that could be for any reason whatsoever. They also fail on master so I suspect it's not the fault of this change.

I see most checks relate to Gnome and its version - what would be the behavior if we ran the tests on a different desktop env like KDE?

azvegint · 2025-06-25T14:46:09Z

I see most checks relate to Gnome and its version - what would be the behavior if we ran the tests on a different desktop env like KDE?

This question has already been raised on the OpenJDK PR. The short answer is that it's an unsupported configuration (I am unsure about JFX), so I didn't test it. If some tuning is required to support it, I would prefer to address that as a follow-up fix.

Please see quote from openjdk/jdk#25265 (comment) below:

BTW how it is expected to work on KDE + wayland?

It is not a supported configuration.However, when I tested it a while ago, it only worked in a single-screen scenario for the Screencast.
There was an issue with the screen streams; they did not specify their x and y coordinates.
This prevented the correct arrangement of multiple screens.
I don't know if this has been fixed, but the issue is addressed in our code here:

https://github.com/openjdk/jdk/blob/3dd34517000e4ce1a21619922c62c025f98aad44/src/java.desktop/unix/native/libawt_xawt/awt/screencast_portal.c#L132-L155

In this case, the Screencast is simply disabled.

The Remote Desktop uses the same streams from Screencast portal to control the mouse position.

lukostyra · 2025-06-26T14:18:53Z

modules/javafx.graphics/src/main/native-glass/gtk/screencast_pipewire.c

+        }                                       \
+    } while (0)
+
+struct pw_buffer *(*fp_pw_stream_dequeue_buffer)(struct pw_stream *stream);


These PipeWire declarations have been just moved here from the original header. Is this move necessary?

Also (a general thought towards past implementation, not just to your change) I think we should not define those manually at all. There are PipeWire headers defining those that we should include (and I believe we do already include some in screencast_pipewire.h

These PipeWire declarations have been just moved here from the original header. Is this move necessary?

It is about clang 17 warning, it was fixed on the jdk side.
I am trying to keep the code similar between OpenJDK and OpenJFX, so I brought this fix here to make maintenance easier.

Also (a general thought towards past implementation, not just to your change) I think we should not define those manually at all. There are PipeWire headers defining those that we should include (and I believe we do already include some in screencast_pipewire.h

The point is basically the same: ease of maintenance between OpenJDK and OpenJFX.
OpenJDK can also be built on very old Linux distributions that lack the necessary headers.

lukostyra

LGTM

arapte

LGTM,
Verified that no test fail on ubuntu 24.04. and changes are very similar to that of JDK.

azvegint · 2025-07-07T18:43:02Z

/integrate

openjdk · 2025-07-07T18:44:34Z

Going to push as commit 639a595.
Since your change was applied there have been 8 commits pushed to the master branch:

0270847: 8351867: No UI changes while iconified
b9dd4de: 8357393: RichTextArea: fails to properly save text attributes
d4cb730: 8356652: Input field ignores custom input source characters
... and 5 more: https://git.openjdk.org/jfx/compare/48282b1067e37092916a9f7f0edafa66a43cfb5a...master

Your commit was automatically rebased without conflicts.

openjdk · 2025-07-07T18:44:45Z

@azvegint Pushed as commit 639a595.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8357584: [XWayland] [OL10] Robot.mousePress() is delivered to wrong p…

df733e7

…lace

openjdk bot added the rfr Ready for review label Jun 18, 2025

kevinrushforth requested review from kevinrushforth and arapte and removed request for kevinrushforth June 18, 2025 22:09

lukostyra reviewed Jun 25, 2025

View reviewed changes

lukostyra reviewed Jun 26, 2025

View reviewed changes

lukostyra approved these changes Jun 27, 2025

View reviewed changes

arapte approved these changes Jul 4, 2025

View reviewed changes

openjdk bot added the ready Ready to be integrated label Jul 4, 2025

openjdk bot added the integrated Pull request has been integrated label Jul 7, 2025

openjdk bot closed this Jul 7, 2025

openjdk bot removed ready Ready to be integrated rfr Ready for review labels Jul 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

8357584: [XWayland] [OL10] Robot.mousePress() is delivered to wrong place #1834

8357584: [XWayland] [OL10] Robot.mousePress() is delivered to wrong place #1834

Uh oh!

azvegint commented Jun 18, 2025 •

edited by openjdk bot

Loading

Uh oh!

bridgekeeper bot commented Jun 18, 2025

Uh oh!

openjdk bot commented Jun 18, 2025 •

edited

Loading

Uh oh!

mlbridge bot commented Jun 18, 2025

Uh oh!

kevinrushforth commented Jun 18, 2025

Uh oh!

openjdk bot commented Jun 18, 2025

Uh oh!

kevinrushforth commented Jun 18, 2025

Uh oh!

lukostyra left a comment

Uh oh!

azvegint commented Jun 25, 2025

Uh oh!

lukostyra Jun 26, 2025

Uh oh!

azvegint Jun 26, 2025

Uh oh!

lukostyra left a comment

Uh oh!

arapte left a comment •

edited

Loading

Uh oh!

azvegint commented Jul 7, 2025

Uh oh!

openjdk bot commented Jul 7, 2025

Uh oh!

openjdk bot commented Jul 7, 2025

Uh oh!

Uh oh!

8357584: [XWayland] [OL10] Robot.mousePress() is delivered to wrong place #1834

8357584: [XWayland] [OL10] Robot.mousePress() is delivered to wrong place #1834

Uh oh!

Conversation

azvegint commented Jun 18, 2025 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Reviewing

Uh oh!

bridgekeeper bot commented Jun 18, 2025

Uh oh!

openjdk bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mlbridge bot commented Jun 18, 2025

Webrevs

Uh oh!

kevinrushforth commented Jun 18, 2025

Uh oh!

openjdk bot commented Jun 18, 2025

Uh oh!

kevinrushforth commented Jun 18, 2025

Uh oh!

lukostyra left a comment

Choose a reason for hiding this comment

Uh oh!

azvegint commented Jun 25, 2025

Uh oh!

lukostyra Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

azvegint Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

lukostyra left a comment

Choose a reason for hiding this comment

Uh oh!

arapte left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

azvegint commented Jul 7, 2025

Uh oh!

openjdk bot commented Jul 7, 2025

Uh oh!

openjdk bot commented Jul 7, 2025

Uh oh!

Uh oh!

azvegint commented Jun 18, 2025 •

edited by openjdk bot

Loading

openjdk bot commented Jun 18, 2025 •

edited

Loading

arapte left a comment •

edited

Loading