Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8280861: Robot color picker broken on Linux with scaling above 100% #7425

Closed
wants to merge 5 commits into from

Conversation

mkartashev
Copy link
Member

@mkartashev mkartashev commented Feb 10, 2022

The primary API for image capture on modern Linuxes is gdk_pixbuf_get_from_window() that expects both coordinates and the size unscaled (i.e. not multiplied by the current desktop scale). At the same time, gtk3_interface.c:gtk3_get_drawable_data() gets the coordinates from Robot that pre-scales them (multiplying by the scale known to Java, however, not necessarily the current desktop scale). The problem with the size had been partly taken care of in JDK-8225118, but coordinates are still passed to gdk_pixbuf_get_from_window() pre-scaled.

The idea of the fix is to capture a possibly larger area that is guaranteed to contain the one that is interesting to the caller and then only copy the interesting pixels to the output image. As a positive side effect, the size of the captured area cannot be less than 1x1 (provided the correct input, of course). This solves the problem of zero size passed to (*fp_gdk_pixbuf_get_from_drawable) when the desktop scale is 3 and we're asked to capture just one pixel. In that case, the previous formula (width / (float) win_scale + 0.5) would have yielded 0.

A related issue would be that tests written specifically for this general area (java/awt/Robot) didn't catch this problem and only somewhat unrelated tests (javax/swing/...) were affected.

This one is solved by adding pixel-sized areas to the test image in HiDPIRobotScreenCaptureTest.java, which precise colors are then verified to match the expected ones. In addition to that, instead of verifying just the color of the center of a large area, 4 more pixels close to the area's border are checked. This helps to make sure that captured area coordinates aren't way off.

The tests under javax/swing/... that originally helped to identify this problem were modified to provide more screen space for painting their components as extreme desktop scaling (300%) on Ubuntu doesn't leave much room due to a humongous title bar.

The fix was tested by running all the modified tests on Ubuntu with desktop scaling set to 100%, 200%, and 300%. The Robot tests were also executed on Windows (300% scaling) and MacOS.


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issue

  • JDK-8280861: Robot color picker broken on Linux with scaling above 100%

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/7425/head:pull/7425
$ git checkout pull/7425

Update a local copy of the PR:
$ git checkout pull/7425
$ git pull https://git.openjdk.java.net/jdk pull/7425/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 7425

View PR using the GUI difftool:
$ git pr show -t 7425

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/7425.diff

@bridgekeeper
Copy link

bridgekeeper bot commented Feb 10, 2022

👋 Welcome back mkartashev! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk openjdk bot added the rfr label Feb 10, 2022
@openjdk
Copy link

openjdk bot commented Feb 10, 2022

@mkartashev The following label will be automatically applied to this pull request:

  • client

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the client label Feb 10, 2022
@mlbridge
Copy link

mlbridge bot commented Feb 10, 2022

Webrevs

inline static int scale_down_to_minus_inf(int what, int scale) {
return (int)floorf(what / (float)scale);
}

static gboolean gtk3_get_drawable_data(JNIEnv *env, jintArray pixelArray,
Copy link
Member

@mrserb mrserb Feb 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we need to do the same for the gtk2? I suggest to update one of the test below to cover gtk_robot on/off for both gtk2/gtk3.

Copy link
Member Author

@mkartashev mkartashev Feb 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe not as the previous fix was mode for gtk3 only, but I can double-check. Do you know if MATE desktop environment is a good candidate for finding gtk2 or are there easier options on Ubuntu?

Copy link
Contributor

@prrace prrace Feb 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea what MATE is but if you are saying you have a recent Ubuntu and you don't have GTK2, I am at least 95% sure GTK2 is still available from Ubuntu package manager for all current releases.

Copy link
Member Author

@mkartashev mkartashev Feb 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MATE desktop continues the "traditional" Gnome2 desktop development, so I thought it would be a good test for Gtk2 interfaces. FWIW, I ran the tests on such a system with DPI set to 200 (there's no other way to scale the interface under MATE AFAIK) and they all pass. However, the test executes the same code path as before going by if (gtk3_version_3_10) branch.

@prrace

GTK2 is still available from Ubuntu package manager for all current releases.

Right, but as long as gtk3 is also available, that's what we are going to use as AWT backend, see the code in gtk_interface.c.
I hacked around to forbid gtk3 from loading and the tests still passed (this time, without executing my changes).
I think gtk2 has no real notion of scale so the coordinates and size are always in screen pixels. See, for example, the documentation for gdk_pixbuf_get_from_drawable().

Copy link
Member

@mrserb mrserb Feb 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You do not need to hack the code, just add a "jdk.gtk.version" option to enable specific gtk version, this is what I suggested above. to disable the gtk usage in the robot the "awt.robot.gtk" can be used. So you can add that to some test to check that code paths works.

Copy link
Member Author

@mkartashev mkartashev Feb 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the pointers, I'll set up a separate test for that.

inline static int scale_down_to_minus_inf(int what, int scale) {
return (int)floorf(what / (float)scale);
}

static gboolean gtk3_get_drawable_data(JNIEnv *env, jintArray pixelArray,
int x, jint y, jint width, jint height, jint jwidth, int dx, int dy,
jint scale) {
Copy link
Member

@mrserb mrserb Feb 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this scale parameter is simply ignored? If the passed parameters are always in the device space and the array is allocated properly then we should not care about this scale(especially in case of gtk2)?

Copy link
Member Author

@mkartashev mkartashev Feb 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this scale parameter is simply ignored?

Looks like it. I can only find one call site of get_drawable_data() (in Java_sun_awt_X11_XRobotPeer_getRGBPixelsImpl()) and the scale parameter is always 1. I think I can drop it from the interface if you prefer.

Copy link
Member

@mrserb mrserb Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes please, this "scale" just make the code complicated w/o reason.

Copy link
Member Author

@mkartashev mkartashev Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's done; please, take a look at the updated code.

@@ -2861,35 +2863,48 @@ static void transform_detail_string (const gchar *detail,
}
}

inline static int scale_down_to_plus_inf(int what, int scale) {
return (int)ceilf(what / (float)scale);
Copy link
Contributor

@prrace prrace Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find the name of this and the companion function confusing.
what is "inf" short for ? I see that and I think plus_infinity.
scaled_ceiling and scaled_floor might be better names based on what I see.

Copy link
Member Author

@mkartashev mkartashev Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, "inf" is for infinity; this is sort of how the rounding modes are described in the IEEE754 standard. But since full length words make the function name too long and short versions are confusing, I will rename as you suggested.

Copy link
Member Author

@mkartashev mkartashev Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@prrace The change has been made, please, take a look.

mrserb
mrserb approved these changes Feb 18, 2022
@openjdk
Copy link

openjdk bot commented Feb 18, 2022

@mkartashev This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8280861: Robot color picker broken on Linux with scaling above 100%

Reviewed-by: serb

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 96 new commits pushed to the master branch:

  • c928958: 8281936: compiler/arguments/TestCodeEntryAlignment.java fails on AVX512 machines
  • a22f422: 8037573: Typo in DefaultTreeModel docs: askAllowsChildren instead of asksAllowsChildren
  • fdce35f: 8282025: assert(ctrl != __null) failed: control out is assumed to be unique after JDK-8281732
  • f830cbe: 8188073: Add Capstone as backend for hsdis
  • 69fc273: 8282075: ProblemList 3 compiler/whitebox tests on macosx-x64
  • 1292776: 8281317: CompactNumberFormat displays 4-digit values when rounding to a new range
  • cd9a3cf: 8282017: sun/net/www/protocol/https/HttpsURLConnection/B6216082.java fails with "SocketException: Unexpected end of file from server"
  • a6f8a38: 8281000: ClassLoader::registerAsParallelCapable throws NPE if caller is null
  • 4c7f8b4: 8268250: Class.arrayType() for a 255-d array throws undocumented IllegalArgumentException
  • d0e1180: 8282019: Unused static fields DEGREES_TO_RADIANS, RADIANS_TO_DEGREES in StrictMath
  • ... and 86 more: https://git.openjdk.java.net/jdk/compare/039313d65d47dc85cb8c91d3e1d2752d365f70f9...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@mrserb) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

@openjdk openjdk bot added the ready label Feb 18, 2022
@mkartashev
Copy link
Member Author

mkartashev commented Feb 18, 2022

/integrate

@openjdk openjdk bot added the sponsor label Feb 18, 2022
@openjdk
Copy link

openjdk bot commented Feb 18, 2022

@mkartashev
Your change (at version 4a8c20a) is now ready to be sponsored by a Committer.

@forantar
Copy link
Contributor

forantar commented Feb 21, 2022

/sponsor

@openjdk
Copy link

openjdk bot commented Feb 21, 2022

Going to push as commit cc7cf81.
Since your change was applied there have been 118 commits pushed to the master branch:

  • d7a706a: 8253757: Add LLVM-based backend for hsdis
  • bdae1d8: 8282147: [TESTBUG] waitForIdle after creating frame in JSpinnerMouseAndKeyPressTest.java
  • 51f4420: 8282130: (bf) Remove unused ARRAY_BASE_OFFSET, ARRAY_INDEX_SCALE from read-only Heap Buffers
  • 34aae32: 8282166: JDK-8282158 changed ECParameters' package by accident
  • c5d9142: 8282096: G1: Remove redundant checks in G1CardSet::free_mem_object
  • 52a85d8: 8282158: ECParameters InvalidParameterSpecException messages missed ECKeySizeParameterSpec
  • 4e0b81c: 8281544: assert(VM_Version::supports_avx512bw()) failed for Tests jdk/incubator/vector/
  • 8563d86: 8282085: The REGISTER_DEFINITION macro is useless after JDK-8269122
  • d28b048: 8281815: x86: Use short jumps in TIG::generate_slow_signature_handler
  • d7f31d0: 8282077: PKCS11 provider C_sign() impl should handle CKR_BUFFER_TOO_SMALL error
  • ... and 108 more: https://git.openjdk.java.net/jdk/compare/039313d65d47dc85cb8c91d3e1d2752d365f70f9...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated label Feb 21, 2022
@openjdk openjdk bot closed this Feb 21, 2022
@openjdk openjdk bot removed ready rfr sponsor labels Feb 21, 2022
@openjdk
Copy link

openjdk bot commented Feb 21, 2022

@forantar @mkartashev Pushed as commit cc7cf81.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
client integrated
4 participants