Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8087700: [KeyCombination, Mac] KeyCharacterCombinations behave erratically #1209

Closed
wants to merge 1 commit into from

Conversation

beldenfox
Copy link
Contributor

@beldenfox beldenfox commented Aug 14, 2023

A KeyCharacterCombination should match a key if the target character is printed on that key. For example, the user should be able to invoke the Shortcut+'+' combination by holding down the Shortcut key and pressing a key that has '+' printed on it. This should work even if '+' is a shifted symbol but the user doesn't hold down the Shift key.

The Mac implements KeyCharacterCombinations by monitoring keystrokes to discover the relationship between keys and characters. Currently the system only records the character the user typed and no other characters on the same key. This means a shortcut targeting a shifted character may not work until the user types that character using Shift so the system learns the relationship.

This PR keeps the same mechanism in place but always records the shifted and unshifted character for each keystroke.

For the Mac the KeyboardTest app was modified to remove tests for characters accessed using Option. We don't look for these characters because under the hood just about every key has some symbol assigned to the Option modifier that the user probably isn't even aware of. For these character we fall back to the existing logic; once the user types the character it will start working as a shortcut.


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed (2 reviews required, with at least 1 Reviewer, 1 Author)

Issue

  • JDK-8087700: [KeyCombination, Mac] KeyCharacterCombinations behave erratically (Bug - P3)

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jfx.git pull/1209/head:pull/1209
$ git checkout pull/1209

Update a local copy of the PR:
$ git checkout pull/1209
$ git pull https://git.openjdk.org/jfx.git pull/1209/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 1209

View PR using the GUI difftool:
$ git pr show -t 1209

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jfx/pull/1209.diff

Webrev

Link to Webrev Comment

@bridgekeeper
Copy link

bridgekeeper bot commented Aug 14, 2023

👋 Welcome back beldenfox! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk openjdk bot added the rfr Ready for review label Aug 14, 2023
@mlbridge
Copy link

mlbridge bot commented Aug 14, 2023

Webrevs

@kevinrushforth
Copy link
Member

/reviewers 2

@openjdk
Copy link

openjdk bot commented Aug 17, 2023

@kevinrushforth
The total number of required reviews for this PR (including the jcheck configuration and the last /reviewers command) is now set to 2 (with at least 1 Reviewer, 1 Author).

@beldenfox beldenfox changed the title 8087700: [KeyCombination, Mac] One key event sent on Mac versus two on Windows 8087700: [KeyCombination, Mac] KeyCharacterCombinations behave erratically Aug 22, 2023
@beldenfox
Copy link
Contributor Author

beldenfox commented Aug 22, 2023

Updated the JBS and PR title and added a new test case. The underlying problem is that KeyCharacterCombinations don't work until the user types the character directly which leads to confusing behavior e.g. Cmd+'+' doesn't work on the main keyboard until the user types '+' but then stops working if they type '+' on the numeric keypad.

@bridgekeeper
Copy link

bridgekeeper bot commented Sep 19, 2023

@beldenfox This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply add a new comment to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

@beldenfox
Copy link
Contributor Author

Adding a comment to keep the bot away. Still hoping this PR gets some review cycles (it's only a tiny little thin one).

@beldenfox
Copy link
Contributor Author

Adding a comment to keeper the robot away.

@beldenfox
Copy link
Contributor Author

This is a customer facing issue (see defold/defold#5850). Like most shortcut-related problems there are duplicates in the Defold issue list because KeyCharacterCombinations are malfunctioning in different ways on different platforms and the behavior varies based on the keyboard layout.

@kevinrushforth
Copy link
Member

@andy-goryachev-oracle Can you be the primary reviewer on this?

@andy-goryachev-oracle
Copy link
Contributor

It looks like the scope of this change went beyond the original problem in JBS.
Would it be possible to update the JBS ticket with the information in this PR description?

@beldenfox
Copy link
Contributor Author

@andy-goryachev-oracle You're right, this PR also deals with the problem of the map retaining stale entries when the user switches layout. I added details about this to the JBS ticket. We could treat it as a separate bug if you want, to me it's just the sort of standard housekeeping one does when caching information.

I also added a note about the KeyboardTest app. This PR scales back that test a bit but this is the first time we've been able to run the KeyCharacterCombinations on the Mac so that portion is basically a new test.

Let me know if you want further information in the JBS ticket.

@andy-goryachev-oracle
Copy link
Contributor

andy-goryachev-oracle commented Nov 14, 2023

@beldenfox thank you. I still not entirely sure what issue is being tested and how to fix it, between this PR and JBS.
For example, when I run the application listed in JBS description with your fix, I get one "HI" - is it expected? (I also get 1 "HI" on Windows)

On macOS, new KeyCodeCombination(KeyCode.MINUS, KeyCombination.CONTROL_DOWN) does not equal to new KeyCharacterCombination("-", KeyCombination.CONTROL_DOWN) (same on Windows)

Interestingly, Ctrl+- on macOS with the US keyboard produces

KeyEvent{type=KEY_PRESSED, character=<\u00>, text=, code=CONTROL, control}
KeyEvent{type=KEY_PRESSED, character=<\u00>, text=, code=MINUS, control}
KeyEvent{type=KEY_TYPED, character=<\u1F>, text=, code=UNDEFINED, control}
KeyEvent{type=KEY_RELEASED, character=<\u00>, text=, code=MINUS, control}
KeyEvent{type=KEY_RELEASED, character=<\u00>, text=, code=CONTROL}

but on Windows, we don't get KEY_TYPED event:

KeyEvent{type=KEY_PRESSED, character=<\u00>, text=, code=CONTROL, control, shortcut}
KeyEvent{type=KEY_PRESSED, character=<\u00>, text=-, code=MINUS, control, shortcut}
KeyEvent{type=KEY_RELEASED, character=<\u00>, text=-, code=MINUS, control, shortcut}
KeyEvent{type=KEY_RELEASED, character=<\u00>, text=, code=CONTROL}

is this expected?

And yes, it would help if we can extract any other issues into their own tickets (unless they all being fixed by this change).

If I may make one suggestion - for the sake of reviewers - to re-phrase the JBS ticket in the format of

  • steps to reproduce
  • expected outcome
  • observed outcome

it would help a lot!

Thanks!

@beldenfox
Copy link
Contributor Author

@andy-goryachev-oracle What keyboard layout do you normally use on Windows? And Mac? When talking about the behavior of punctuation this is a crucial bit of information.

but on Windows, we don't get KEY_TYPED event:
is this expected?

When you hold down Ctrl and press a key the platform may or may not generate a low-ASCII control code. JavaFX has never tried to make the platforms consistent here, Glass just passes on whatever the OS generates. In any case the TextInputControls treat these TYPED events as nuisances, if you look in TextInputControlBehavior.java and read defaultKeyTyped you'll see how it tries to filter out control codes and anything else that looks like it was associated with a shortcut.

@andy-goryachev-oracle
Copy link
Contributor

What keyboard layout do you normally use on Windows?

US layout on both, at least when the events were captured in the previous comment.

@beldenfox
Copy link
Contributor Author

@beldenfox thank you. I still not entirely sure what issue is being tested and how to fix it, between this PR and JBS. For example, when I run the application listed in JBS description with your fix, I get one "HI" - is it expected? (I also get 1 "HI" on Windows)

When you run the application in the JBS description and press Ctrl-'-' on the main keyboard (not the numeric keypad) you should see "HI" printed twice, once for the MINUS KeyCode combination and once for the '-' KeyCharacter combination. I'm not sure why that's not working for you on Windows with a U.S. layout. With that said, on Windows KeyCharacter combinations for punctuation can be uncertain unless you're running a build with PR #1264 which was just integrated.

@andy-goryachev-oracle
Copy link
Contributor

let me test with #1264 integrated.

@andy-goryachev-oracle
Copy link
Contributor

andy-goryachev-oracle commented Nov 15, 2023

All right, so I see "HI" printed twice on macOS and Windows. I guess we are good here.

A couple of other observations:

  1. Using French keyboard on mac (and windows), Ctrl-- is expected to produce the key code RIGHT_PARENTHESIS, correct?
KeyEvent{type=KEY_PRESSED, character=<\u00>, text=, code=CONTROL, control}
KeyEvent{type=KEY_PRESSED, character=<\u00>, text=, code=RIGHT_PARENTHESIS, control}
KeyEvent{type=KEY_TYPED, character=<\u1B>, text=, code=UNDEFINED, control}  <-- missing on Windows, as expected
KeyEvent{type=KEY_RELEASED, character=<\u00>, text=, code=RIGHT_PARENTHESIS, control}
KeyEvent{type=KEY_RELEASED, character=<\u00>, text=, code=CONTROL}
  1. Using KeyboardTest, I am getting:
Layout w/o combinations w/o keypad combinations with all combinations
US 0 0 0
French 15 27 28
German 4 13 14
Spanish 7 17 18
Latin 0 0 0
non-Latin 0 0 0

Is this a problem?

(Also, as a side note - would it make sense to change the KeyboardTest to to run all the individual tests in one go?)

@beldenfox
Copy link
Contributor Author

  1. Using French keyboard on mac (and windows), Ctrl-- is expected to produce the key code RIGHT_PARENTHESIS, correct?

Yes, that key turns into right parenthesis with a French layout.

  1. Using KeyboardTest, I am getting:

You should not see any failures as long as the test and your selected keyboard match. It looks like you were running the French, German, and Spanish tests while your keyboard was set to U.S. English. Unfortunately you have to change the keyboard layout manually, there's no API in Java that allows me to do it. That's also why I can't run all the tests in one go.

@andy-goryachev-oracle
Copy link
Contributor

as long as the test and your selected keyboard match.

ah! my fault, did not read the javadoc. let me try again.

@andy-goryachev-oracle
Copy link
Contributor

andy-goryachev-oracle commented Nov 16, 2023

0 failures with German.

Do you think some other layouts should be added to the keyboard test?
(Suggestion: Arabic, Hebrew, Japanese, French, Russian)

Another suggestion: maybe add the instructions to the KeyboardTest window?

@kevinrushforth
Copy link
Member

Do you think some other layouts should be added to the keyboard test? (Suggestion: Arabic, Hebrew, Japanese, French, Russian)

Another suggestion: maybe add the instructions to the KeyboardTest window?

Those both sound like good suggestions. I recommend taking them in a follow-up issue rather than as part of this PR.

@andy-goryachev-oracle
Copy link
Contributor

definitely as a follow-up. this PR looks good to me.

@beldenfox
Copy link
Contributor Author

I created JDK-8320216 for adding instructions to the test app.

It would be nice to have a non-Latin language in the test so I will look into adding Russian, Arabic, or Hebrew.

I have no idea where to begin with IME-based languages like Japanese and am concerned that the resulting tests would be very platform-specific. In any case that would be an entirely different test app.

Copy link
Member

@kevinrushforth kevinrushforth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I tested it only on a US keyboard layout, but Andy has done more extensive testing.

@openjdk
Copy link

openjdk bot commented Nov 16, 2023

@beldenfox This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8087700: [KeyCombination, Mac] KeyCharacterCombinations behave erratically

Reviewed-by: angorya, kcr

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 74 new commits pushed to the master branch:

  • 2e73304: 8319996: Update to GCC 13.2.0 on Linux
  • 59c86bb: 8303478: DatePicker throws uncatchable exception on tab out from garbled text
  • c6783b1: 8319341: [Linux] Remove operation to show or hide children because it is unnecessary
  • 986ec4f: 8319669: [macos14] Running any JavaFX app prints Secure coding warning
  • 2d4b494: 8319762: Update to Visual Studio 2022 version 17.6.5 on Windows
  • c8b44be: 8274967: KeyCharacterCombinations for punctuation and symbols fail on non-US keyboards
  • d24e96a: 8318984: Update to Xcode 14.3.1 on macOS
  • 8dd3c37: 8318841: macOS: Memory leak with MenuItem when Menu.useSystemMenuBar(true) is used
  • c8641e2: 8284445: macOS 12 prints a warning when a function key shortcut is assigned to a menu
  • 29f9c4d: 8319147: Add regression test for JDK-8317836
  • ... and 64 more: https://git.openjdk.org/jfx/compare/8998b2d84ccdc48194abc0ae4184b2b0c5658fc3...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@andy-goryachev-oracle, @kevinrushforth) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

@openjdk openjdk bot added the ready Ready to be integrated label Nov 16, 2023
@andy-goryachev-oracle
Copy link
Contributor

I have no idea where to begin with IME-based languages like Japanese

good point. IME is probably out of scope then.

@beldenfox
Copy link
Contributor Author

/integrate

@openjdk openjdk bot added the sponsor Ready to sponsor label Nov 16, 2023
@openjdk
Copy link

openjdk bot commented Nov 16, 2023

@beldenfox
Your change (at version bb02b08) is now ready to be sponsored by a Committer.

@kevinrushforth
Copy link
Member

/sponsor

@openjdk
Copy link

openjdk bot commented Nov 16, 2023

Going to push as commit cda623d.
Since your change was applied there have been 74 commits pushed to the master branch:

  • 2e73304: 8319996: Update to GCC 13.2.0 on Linux
  • 59c86bb: 8303478: DatePicker throws uncatchable exception on tab out from garbled text
  • c6783b1: 8319341: [Linux] Remove operation to show or hide children because it is unnecessary
  • 986ec4f: 8319669: [macos14] Running any JavaFX app prints Secure coding warning
  • 2d4b494: 8319762: Update to Visual Studio 2022 version 17.6.5 on Windows
  • c8b44be: 8274967: KeyCharacterCombinations for punctuation and symbols fail on non-US keyboards
  • d24e96a: 8318984: Update to Xcode 14.3.1 on macOS
  • 8dd3c37: 8318841: macOS: Memory leak with MenuItem when Menu.useSystemMenuBar(true) is used
  • c8641e2: 8284445: macOS 12 prints a warning when a function key shortcut is assigned to a menu
  • 29f9c4d: 8319147: Add regression test for JDK-8317836
  • ... and 64 more: https://git.openjdk.org/jfx/compare/8998b2d84ccdc48194abc0ae4184b2b0c5658fc3...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated Pull request has been integrated label Nov 16, 2023
@openjdk openjdk bot closed this Nov 16, 2023
@openjdk openjdk bot removed ready Ready to be integrated rfr Ready for review sponsor Ready to sponsor labels Nov 16, 2023
@openjdk
Copy link

openjdk bot commented Nov 16, 2023

@kevinrushforth @beldenfox Pushed as commit cda623d.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

@yososs
Copy link

yososs commented Nov 25, 2023

!! This PR causes block-level problems in Japanese environment.

When Japanese IME is turned on,
key input is ignored when Japanese IME is turned on after merging this PR.
I verified that the problem does not reproduce by reverting to the code before applying this PR.

@kevinrushforth @andy-goryachev-oracle

@beldenfox
Copy link
Contributor Author

@yososs I can't reproduce this with my test apps or with the MonkeyTester. I tried both Kana and Romaji on macOS 14 but didn't try every possible input mode.

  • Which version of macOS are you using?
  • Which input method and mode are you using?
  • What KeyCharacterCombinations does your app use?

@yososs
Copy link

yososs commented Nov 26, 2023

The environment where I confirmed the problem is as follows.

  • macOS Ventura 13.6 (arm64)
  • Xcode 15.0.1

The IME settings are as follows.

  • ABC
  • 日本語

This issue causes text input and shortcut operations to be ignored when using the IME. Almost all apps are affected.

@beldenfox
Copy link
Contributor Author

I can reproduce this on 13.6. The charactersByApplyingModifiers call is modifying the event in some way which is disturbing the IME. This is surprising given how that call is documented and that it's been around since 10.15. It looks like the fix is to do all this processing after we call handleEvent: on the NSTextInputContext.

@beldenfox
Copy link
Contributor Author

The problem occurs on macOS 12 and 13 which is as far back as I can test. Calling charactersByApplyingModifiers modifiers the NSEvent in some way that causes IMEs to malfunction. I've verified that Chinese - Traditional is also affected.

The solution is to use the same Carbon calls that I used in PR #425. I've written the code and will perform some more tests before submitting a PR. I'm not sure if the right approach is to re-open the original bug and rejuvenate this PR or submit a separate bug. The fix for the original bug will need to be re-tested.

@yososs
Copy link

yososs commented Nov 27, 2023

In language environments that use IMEs, the side effects of this change will be considered block-level. For the app developer, I hope that by his next OpenJFX release there will be a plan to keep this side effect out of the release's binaries.

Currently, JavaFX macOS support for both x86_64 and arm64 is specified for minOS 11.0. I would like to see testing on macOS 11.

@andy-goryachev-oracle
Copy link
Contributor

Do we want to revert the fix? In any case, we need a new bug report ticket.

@beldenfox
Copy link
Contributor Author

I entered a new ticket JDK-8320773. I will submit a fix today.

I can test macOS 12, 13, and 14 on Apple Silicon but can't go as far back as 11.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integrated Pull request has been integrated
4 participants