feat(web): delayed modipress completion 🐵 #9973

jahorton · 2023-11-08T14:04:48Z

This aims to cover a few "fun" edge cases that can easily arise during quick typing that involves modipresses:

For longpresses triggered during a modipress that has since been released, the underlying layer for the OSK will not revert until subkey-selection is complete.
- It felt "weird" when it would swap back early, underneath the subkey menu.
- Granted, it also feels weird to me when it stays in place, but it's a better representation of the system's state.
For flicks triggered during a modipress that has since been released, the underlying layer for the OSK will not revert until the flick is complete.
- Note: flick completion may be triggered if a new, incoming tap is received. In such a case:
  1. The flick completes as normal, for the original key from the modipressed layer.
  2. The modipress is then fully resolved, including reversion of the layer to the one that was originally active before the modipress
  3. And then the new, incoming tap is processed - for the pre-modipress layer.

The main idea behind point 2 above: during quick typing, it's very natural to lift the modipress finger to prepare to type a new key. This could easily trigger earlier than intended, so we keep the modipress active while the flick is still active.

Point 1 above then provides consistency with point 2 - in both cases, the layer "sticks" until the pending gesture is completed.

In case the user accidentally presses a new key either slightly before or simultaneously with release during this state, we can still allow the flick to complete. Since this is generally during quick typing, and the user did lift the modipress key, they've likely has anticipated the layer to return - and so we adjust for that expectation.

I will say that point 2 above was an interesting niche case to implement - it involved a second coming of the TWo caps problem (#7173 / #9802) from a slightly different angle. While it does add some complexity, it should help make flicks more robust to certain natural 'optimizations' we often subconsciously make while typing quickly.

Notes for future revisitation

❗ ❗ ❗

I ended up throwing in quite a number of other fixes into this one, as it held the big user test suite and was used as the anchor-point for demo builds. Not exactly ideal, but it is what it is at this point. FWIW, I did aim to keep the commits reasonably clean, so going commit by commit might help... though there was a merge from the base (including a master update) that may make certain aspects interesting to follow.

User Testing

Note that most of these tests will be repeats of user tests from previous PRs. Consider this largely a feature-branch regression-test suite... and thus tests worthy of consideration for becoming permanent regression tests once the feature-branch lands.

TEST_10_KEY_ROTA: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Classic 10-key".
Verify that all of the keys numbered 1-9 aside from 7 have three or more lowercase letters as key hints.
Tap the 8 key once, then wait for a second.
Tap the 8 key twice in quick succession, then wait for a second.
- Expected output: 8a.
Tap the 8 key continuously for at least 2 seconds.
- it should rotate through: 8a8, 8aa, 8ab, 8ac, 8aA, 8aB, 8aC, then back to 8a8 (and continue from there)
- stop tapping when the new character is a capital letter.
Verify that the key hints are now capitalized.
Double-tap the 9 key once. A D should be emitted as the result.
Repeatedly tap 9 for at least five seconds, paying attention to the key hints.
- Whenever a lowercase letter is the result, the key hints should be lowercase
- Whenever an uppercase letter is the result, the key hints should be uppercase

TEST_10_KEY_DIACRITICS: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select the text-area element - the first input-accepting element on the page.
- Placeholder text: "Type here".
- Is directly underneath "Type in your language in this text area:"
Select "English...", then "Diacritic 10-key Rota".
Tap the '◌̀' key. (The key between backspace [above] and enter [below], representing a diacritic)
Double-tap the 8 key.
- Expected result: à
Triple-tap the '◌̀' key - a circumflex diacritic should appear over the à.
Triple-tap the 9 key.
- Expected total result: àê
Tap the '◌̀' key. A diacritic should appear over the ê.
Deselect the text-area element.
Reselect the text-area element, placing the point of text entry (the caret) at the end, after the ê.
- Note for future test-maintenance: the point is to trigger a context-reset.
Double-tap the 8 key.
- Expected final result: àềa
- For clarity:
  - With a copy of the à's accent-mark on top the original ê, as in the cropped screenshot above.
    - It looks different in raw-text form here due to GitHub's font / rendering.
  - Note: there is no matching accent-mark on the second a.

TEST_APP_10_KEY_DIACRITICS: Using the Keyman for Android test artifact...

Download this KMP: https://jahorton.github.io/diacritic_rota.kmp
- Can also be found via the main page at https://jahorton.github.io.
Open / install it for use within the Keyman app.
Tap the 7 key twice; verify that 77 is output.
Tap the '◌̀' key. (The key between backspace [above] and enter [below], representing a diacritic)
Double-tap the 8 key.
- Expected total result: 77à
Triple-tap the '◌̀' key - a circumflex diacritic should appear over the à.
Triple-tap the 9 key.
- Expected total result: 77àê
Tap the '◌̀' key. A diacritic should appear over the ê.
Open the in-app Settings menu.
- Note, for test maintenance: the point of this step and the next one is to trigger a context-reset.
Close the in-app Settings menu.
Double-tap the 8 key.
- Expected final result: 77àềa
- For clarity:
  - With a copy of the à's accent-mark on top the original ê, as in the cropped screenshot above.
    - It looks different in raw-text form here due to GitHub's font / rendering.
  - Note: there is no matching accent-mark on the second a.

TEST_BASIC_SIMPLE_SHIFT: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "Norwegian" (for the SIL EuroLatin keyboard)
Tap the spacebar. (The keyboard should be on the default [lowercase] layer after this.)
Tap the shift key once.
Verify that the keyboard changes to, and stays on, the shift layer.

TEST_BASIC_MODIPRESS: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "Norwegian" (for the SIL EuroLatin keyboard)
Tap the spacebar. (The keyboard should be on the default [lowercase] layer after this.)
Tap and hold the shift key.
Verify that the keyboard changes to the shift layer.
Tap the A key, verifying that it produces an A.
Release the shift key.
Verify that the keyboard changes to the default layer as a result of step 6.
Repeat steps 3 through 7 quickly, holding shift just long enough to tap the shift layer's A key, then releasing quickly.

TEST_BASIC_MODIPRESS_HOLD: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "Norwegian" (for the SIL EuroLatin keyboard)
Tap the spacebar. (The keyboard should be on the default [lowercase] layer after this.)
Tap and hold the shift key.
Verify that the keyboard changes to the shift layer.
Wait at least one second.
Release the shift key.
Verify that the keyboard changes to the default layer as a result of step 6.

TEST_NUMERIC_FROM_SHIFT: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "Norwegian" (for the SIL EuroLatin keyboard)
Tap the spacebar. (The keyboard should be on the default [lowercase] layer after this.)
Tap the shift key, making the shift layer active.
Tap and hold the numeric key (123) underneath the shift key.
Verify that the keyboard changes to the numeric layer - numbers in the top row, mathematical symbols elsewhere.
Longpress the % key and select the ‱ subkey.
Verify that ‱ is emitted as text.
Release the numeric key.
Verify that the keyboard changes back to the shift layer.
Tap the numeric key, releasing it immediately.
Repeat steps 6 and 7, longpressing the % key and select the ‱ subkey.
Verify that the keyboard automatically changes back to the default (lowercase) layer.

TEST_DELAYED_SUBKEY: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "Norwegian" (for the SIL EuroLatin keyboard)
Tap the spacebar. (The keyboard should be on the default [lowercase] layer after this.)
Tap the shift key, making the shift layer active.
Tap and hold the numeric key (123) underneath the shift key.
Verify that the keyboard changes to the numeric layer - numbers in the top row, mathematical symbols elsewhere.
Longpress the % key, but do not select a subkey yet.
Release the numeric key.
Verify that the keyboard changes remains on the numeric layer, with the subkey menu still displayed.
Select the ‱ subkey.
Verify that ‱ is emitted as text and that the layer has changed back to the shift layer.

TEST_DOUBLETAP_CAPS: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "Norwegian" - you should then see "Norwegian - EuroLatin (SIL)" in the space bar.
Type All .
Double-tap the shift key. Afterward, the shift key should be highlighted, and its up-arrow should have a thin line underneath, as if underlining it.
Without touching the shift key at any point, type CAPS .
- If this is impossible - i.e., the layer shifted out from underneath you - FAIL the test.
Single-tap the shift key. Afterward, you should be returned to the base layer.
Type work .
Tap the shift key once.
Without touching the shift key at any point, type Well.
- If this is impossible - i.e., the layer did not drop the the base layer when expected - FAIL the test.
- Expected final result, in total: All CAPS work Well

TEST_CUSTOM_MULTITAP_MODIFIER: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Gesture Prototyping".
Tap and hold the ◌́ key. (The key between the spacebar [left] and the Enter key [right].)
A number of letters, including all of the vowels, should change to versions using the ◌́ accent mark.
Tap the á and verify that the corresponding letter is output.
Release the ◌́ key; you should be returned to the default layer.
Note the hints on the ◌́, then triple-tap it, holding on the third tap.
A number of letters, including all of the vowels, should change to versions using the ◌̂ mark
- This corresponds to the second diacritic from the ◌́-key's hint.
Tap the â and verify that the corresponding letter is output.
Release the ◌́ key; you should be returned to the default layer.

TEST_ALTERNATING_SHIFT_AND_KEY: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Gesture Prototyping".
Tap and hold the SHIFT key
Tap the H key and release it.
Release the SHIFT key.
Verify that an H is output and that the layer returns to default after step 4.
Repeat steps 2-5 five times.
Repeat steps 2-4 quickly for at least 10 seconds.
- Do not bother checking the result layer; the priority is to be quick here.
- FAIL this test if any output text is replaced or removed during these repetitions.
- FAIL this test if text (either h or H) is not output each time you press the h / H key.
When done with step 7's repetitions, tap the numeric (123) key and verify that the layer switches properly to keys with numbers in the top row and symbols in the middle two rows.

TEST_FLICKS_BASIC: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Gesture Prototyping".
Tap and hold the u key.
Drag it in an northwest direction (up and to the left) for at least a key-width.
Release it.
Expected result: ù
Tap the u key again and drag it in a northeast direction for at least a key-width before releasing it.
Expected result: ú (does not replace the previous output)
Tap u again and drag north (straight up) in the same manner.
Expected result: û.

TEST_FLICK_CORRECTION: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Gesture Prototyping".
Tap and hold the u key.
Drag it in an purely west direction (straight left).
Release it.
Expected result: ù
Tap and hold the u key.
Drag it purely south (straight down).
Release it.
Expected result: u. The flick animation and key preview should disappear.
Tap and hold the u key again.
Drag it up just a few pixels / millimeters - a very short distance - then release it.
Expected output: u.

TEST_FLICK_LOCKING: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Gesture Prototyping".
Tap and hold the u key.
For the following steps, pay attention to the animated preview.
Drag it purely north (straight up) - you should see the û key in the preview area (assuming it's not blocked by your finger).
- If the animation was "jumpy" - if there was a lack of motion at the same time you moved your finger, then a big "jump" to catch up - FAIL this test.
- If any such "jumps" were directly tied to you moving your finger quickly, do not fail this test.
Return your finger to its original position; the 'u' key should slide back into place.
- Note: "original position" is kind of important; if you return slightly to the left of the original position, the next step may not work well.
Now, drag your finger in a up-and-right motion: you should also see a preview for ú as you do so.
- Again, if the animation felt "jumpy" and not well connected to your input, FAIL this test.
While the ú is visible and in the center, lift your finger and end the gesture.
Verify that a ú is output.
Repeat steps 2 through 6 for the a key; you should see similar previews and text there. (â, á)

TEST_MODIPRESS_MULTITAP_FLICK_PREVIEW: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Gesture Prototyping".
Double-tap and hold the SHIFT key, keeping it stationary for all following steps.
Tap and hold the U key.
Verify that the key preview is visible.
Drag the finger on U purely north (straight up) - you should see the Û key in the preview area (assuming it's not blocked by your finger).
Verify that the flick-animation in the preview scrolls to a Û.
With the Û fully in-view, release the flick.
Verify that a Û is output.
Repeat steps 2 through 6 quickly three times, including the "verify" steps.

TEST_FLICK_DURING_MODIPRESS: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Gesture Prototyping".
Tap and hold the SHIFT key, keeping it stationary for all following steps.
Tap and hold the U key.
For the following steps, pay attention to the animated preview.
Drag it purely north (straight up) - you should see the Û key in the preview area (assuming it's not blocked by your finger).
- If the animation was "jumpy" - if there was a lack of motion at the same time you moved your finger, then a big "jump" to catch up - FAIL this test.
- If any such "jumps" were directly tied to you moving your finger quickly, do not fail this test.
Release the SHIFT key (held since step 2).
Verify that the shift layer (all uppercase) remains visible and that the flick animation is still active.
Return your finger to its original position; the 'U' key should slide back into place.
Now, drag your finger in a up-and-right motion: you should also see a preview for Ú as you do so.
- Again, if the animation felt "jumpy" and not well connected to your input, FAIL this test.
While the Ú is visible and in the center, tap on the Q key.
Verify that the keyboard returns to the default (lowercase) layer and that either Ú or Úq is output.
- Úq is ideal, but if your a tap is quick, you may only get the Ú.

I will note that sometimes, when doing the input quickly, I get only Ú (and not the q). Not sure exactly why that is, but I felt it may be worth noting... and worth a future investigation. It is likely typing-speed related, since there's a fair bit of await-async there.

keymanapp-test-bot · 2023-11-08T14:04:52Z

User Test Results

Test specification and instructions

✅ TEST_10_KEY_ROTA (PASSED): Tested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Opened "English - Classic 10-key" keyboard. 2. Verified that all of the keys numbered from 1 to 9 aside from 7 have three or more lowercase letters as key hints. 3. Followed steps 3 and 4 and verified that the expected output is 8a. 4. Tapped the 8 key continuously for at least 2 seconds and verified that it output different combination of letters and numbers. Stopped tapping when the new character is a capital letter. Verified that the key hints are now capitalized. 5. Double-tapped the 9 key once. A 'D' key appeared in the text area. 6. Repeatedly tapping 9 for at least 5 seconds and verified that the key hits change from lowercase letter to uppercase letter or uppercase letter to lowercase letter. (notes)
✅ TEST_10_KEY_DIACRITICS (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "English - Diacritic 10-key Rota" keyboard. 2. Tapped the ◌̀ key. Then, double-tapped the 8 Key. Verified that the expected result à appears in the text area. 3. Triple-tapped the ◌̀ key. Triped tapped the 9 key. Verified that the àê letters appeared on the screen. 4. Tapped the '◌̀' key. Then, tapped the 4 key once. Hit backspace. Double-tapped the 8 key showed the expected result àềa. (notes)
✅ TEST_APP_10_KEY_DIACRITICS (PASSED): Retested using the Keyman for Android test artifact.. from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Installed the attached Keyman build 17.0.209-alpha-test-9973 in the device. 2. Downloaded and then installed diacritic_rota.kmp file in Keyman App. 3. Tapped the 7 key twice. Verified that 77 appears on the screen. 4. After running step 5, able to see 77à as a result. 5. After running step 7, able to see the 77àê as a result. 6. After running step 11, able to see 77àềa as a result. (notes)
✅ TEST_BASIC_SIMPLE_SHIFT (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. After running step 3, it is verified that the keyboard remains on the shift layer. (notes)
✅ TEST_BASIC_MODIPRESS (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. After running step 8, it is verified that the keyboard changed to the default layer. (notes)
✅ TEST_BASIC_MODIPRESS_HOLD (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. After running step 6, it is verified that the keyboard changed to the default layer due to step 6. (notes)
✅ TEST_NUMERIC_FROM_SHIFT (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. Verified that ‱ is emitted as text and the keyboard changed back to the shift layer, after running steps 6 and 8. 3. Verified that the keyboard automatically reverted to the default layer, after running step 11. (notes)
✅ TEST_DELAYED_SUBKEY (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. Verified that the keyboard keyboard changed to the numeric layer after running step 4. 3. Also, verified that the keyboard changes remain on the numeric layer, with the subkey menu still displayed, after running step 8. 4. Verified that ‱ is emitted as text and that the layer has changed back to the shift layer. (notes)
✅ TEST_DOUBLETAP_CAPS (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. Verified that it is showing the expected outputs after running from 2 to 8. (notes)
✅ TEST_CUSTOM_MULTITAP_MODIFIER (PASSED): In which I update a test spec again; had the wrong output specified for a step, but the test results were correct for actual (and intended!) behavior.
✅ TEST_ALTERNATING_SHIFT_AND_KEY (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. Verified that an H is output and the keyboard layer returns to default layer after running step 4. 3. Verified that there is no deviations in the output after running step 7. 4. Verified that the layer switches properly to keys with numbers in the top row and symbols in the middle two rows, after clicking the 'num' key. (notes)
✅ TEST_FLICKS_BASIC (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. After running step 4, it is verified that it outputs ù on the screen. 3. After running step 6, it is verified that it outputs ù on the screen. 4. After running step 8, it is verified that it outputs û on the screen. (notes)
✅ TEST_FLICK_CORRECTION (PASSED): In which it turns out another test spec needed updating to match current behavior. I think I wrote that test before the locked-flicks stuff was fully implemented; it makes sense for step 8 to emit that 'u', now that I inspect it more closely. (notes)
✅ TEST_FLICK_LOCKING (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. Tapped and hold the 'u' key. 3. Dragged it purely straight-up position, and I was able to see the letter û. Observed that there was no lack of motion after dragging it. 4. After moving back my finger to the original position, the 'u' key slides back into place. 5. Tapped and hold the 'u' key, then drag it in an up-and-right motion. I was able to see a preview of ú key. Observed that no animated preview happened. Lift my finger and end the gesture. Verified that a ú outputs on the screen. 6. Followed steps 2 through 6 for the 'a' key and I am able to see the similar previews and text there. Seems to be working fine. Thanks. (notes)
✅ TEST_MODIPRESS_MULTITAP_FLICK_PREVIEW (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. Double tapped then hold the Shift key. Tapped and hold the U key. Verified that the key preview is visible. 3. Dragged the finger on U purely north (straight up) and I was able to see Û key in the preview area. 4. Verified that the flick-animation in the preview scrolls to a Û. 5. Release the flick, when the Û key is fully visible. Verified that Û is output. 6. Repeated steps 2 through 6 quickly three or four times and verified that the Û appears repeatedly as the output. Seems to be working fine. (notes)
✅ TEST_FLICK_DURING_MODIPRESS (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. Tapped and hold the Shift key. Tapped and hold the U key. 3. Dragged it to straight-up position. I was able to see the Û key without "jumpy" in the preview area. Released the Shift key. Verified that the shift layer remains visible and that the flick animation is still active. 4. Returned the finger to its original position, able to see the U key slide back into place. 5. Dragged the finger in an up-and-right motion and I was able to see the Ú key in the preview. 6. While the Ú is visible in the center, tapped on the q key with the other finger. Verified that the keyboard returns to the default layer and Úq is output on the screen. Seems to be working fine. (notes)

Test Artifacts

Android
Developer
iOS
- Keyman for iOS (simulator image)
- FirstVoices Keyboards for iOS (simulator image)
- TestFlight internal PR build version - 17.0.209 (0.9973.9813)
Keyboards
- Test Keyboards
Web
- KeymanWeb Test Home
Windows

…letion-triggering source

…ing)

…ed-modipress-completion

mcdurdin · 2023-11-15T02:26:33Z

common/web/gesture-recognizer/src/engine/headless/touchpointCoordinator.ts

+    // For multitouch gestures, only report the gesture **once**.
+    const sourceIDs = selection.matcher.allSourceIds;
+    for(let sequence of this._activeGestures) {
+      if(!!sequence.allSourceIds.find((id1) => !!sourceIDs.find((id2) => id1 == id2))) {


O(m*n)? Are these arrays short?

A person only has ten fingers (corresponding to allSourceIds), and I'd be quite impressed if someone actually managed to use each of them independently while building concurrent gestures (which corresponds to the for... let part). Technically, completed touches can be included in the allSourceIds array - it's needed for multitap history.

Secondly, with the way Web currently works, I believe the maximum count for concurrent gestures is 2; a modipress and whatever the current input is on the modipress layer. Starting a third will immediately resolve the previously-current input (simple tap, flick, multitap) or be blocked (longpress). There's a chance of it actually being 3 - I'm not 100% sure that a held backspace will be auto-cancelled on new input. But yeah, that should be it.

This does depend on proper state management of the 'active gestures' array. If errors happen that prevent removal of finished gesture objects from that array, then over time it may become "less short" than intended.

The potential for most growth is in allSourceIds if someone goes crazy with constant 'rota' key mashing, but even then, it's still practically just O(n) due to the low cap on this._activeGestures's practical length.

That's what I figured but helpful info on the potential edge cases, thanks!

bharanidharanj · 2023-11-15T12:45:39Z

Test Results

TEST_10_KEY_ROTA (PASSED): Tested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Opened "English - Classic 10-key" keyboard. 2. Verified that all of the keys numbered from 1 to 9 aside from 7 have three or more lowercase letters as key hints. 3. Followed steps 3 and 4 and verified that the expected output is 8a. 4. Tapped the 8 key continuously for at least 2 seconds and verified that it output different combination of letters and numbers. Stopped tapping when the new character is a capital letter. Verified that the key hints are now capitalized. 5. Double-tapped the 9 key once. A 'D' key appeared in the text area. 6. Repeatedly tapping 9 for at least 5 seconds and verified that the key hits change from lowercase letter to uppercase letter or uppercase letter to lowercase letter.

..1to9 numbers with key hints

..Tap the 8 key twice in quick succession

..Tap the 8 key continuously for at least 2 seconds

..Repeatedly tap 9 for at least five seconds..

bharanidharanj · 2023-11-15T15:07:09Z

Test Results

TEST_10_KEY_DIACRITICS (PASSED): Tested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "English - Diacritic 10-key Rota" keyboard. 2. Tapped the ◌̀ key. Then, double tapped the 8 Key. Verified that the expected result à appear on the text area. 3. Triple-tapped the ◌̀ key. Triped tapped the 9 key. Verified that the àê letters appeared on the screen. 4. Tapped the '◌̀' key. Then, tapped the 4 key once. Hit backspace. Double-tapped the 8 key showed the expected result àềa.

..result 1

..result 2

..result 3

bharanidharanj · 2023-11-16T07:02:32Z

Test Results

TEST_CUSTOM_MULTITAP_MODIFIER (FAILED): Tested using the standard "Test unminified KeymanWeb" test page from the mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "English - Gesture Prototyping" keyboard. 2. Single-tapped the ◌́ key. Noticed that the number of letters, including all of the vowels are changed to versions using the wrong output of the accent mark. Seems to be an issue.

jahorton · 2023-11-22T02:13:31Z

@keymanapp-test-bot retest TEST_10_KEY_DIACRITICS

@bharanidharanj That failed test is on me; I forgot that we changed a related behavior with #9944. I've updated the test instructions accordingly.

Turns out that I discovered a bug on master while doing so; for the sake of user testing, I've gone ahead and fixed it here... and 🍒-picked it as #10039 for master. It'd require a retooling of the test keyboard and heavier retooling of the instructions otherwise.

Also, something probably worth note/making an issue: TEST_APP_10_KEY_DIACRITICS will fail if performed on an iOS device if the 7 key step is removed... if the text area is empty before starting the test. My intuition tells me that this is due to one of the iOS quirks we noticed a while back: the iOS equivalent to a text-services framework signals a context reset any time the active text-input receiver has its text cleared. Well... multitapping on a context-initial diacritic-output-only key probably isn't helping things there, possibly causing a context-desync between the app and the keyboard engine. That doesn't seem entirely right, but it's probably quite related.

As it appears to be due to something iOS-specific and does not occur on Android, I think this issue should be documented and pursued separately.

On Android, the context-initial diacritic doesn't seem to get displayed properly in the emulator setup I use most frequently, but it is still present and does still combine properly according to keyboard rules. (Notably, placeholder text disappears when it should be visible, indicating the presence of text... even if it - the isolated diacritic - is not being displayed properly.) That'd be more of a rendering issue within the app and/or on Android, I guess.

jahorton · 2023-11-22T03:31:23Z

Also, something probably worth note/making an issue: TEST_APP_10_KEY_DIACRITICS will fail if performed on an iOS device if the 7 key step is removed... if the text area is empty before starting the test. My intuition tells me that this is due to one of the iOS quirks we noticed a while back: the iOS equivalent to a text-services framework signals a context reset any time the active text-input receiver has its text cleared. Well... multitapping on a context-initial diacritic-output-only key probably isn't helping things there, possibly causing a context-desync between the app and the keyboard engine. That doesn't seem entirely right, but it's probably quite related.

As it appears to be due to something iOS-specific and does not occur on Android, I think this issue should be documented and pursued separately.

Eh, why not.

Did a little investigation: our old enemy, U+200E (https://unicode-explorer.com/c/200E) has returned to haunt us. It's being automatically inserted at the start of the context whenever a diacritic is added in that position or directly after a different diacritic meeting this compound condition (e.g., it can chain).

This is occurring via setKeymanVal calls to the ios-host.js script:

keyman/ios/engine/KMEI/KeymanEngine/resources/Keyman.bundle/Contents/Resources/ios-host.js

Lines 292 to 295 in 6b8a06a

    
           if(keyman.context.getText() != text) { 
        
               keyman.context.setText(text); 
        
               keyman.resetContext(); 
        
           }

This has been verified via breakpoint when performing a repro for the conditions described above - that if-condition will trigger, resulting in, among other things, a deadkey-wipe.

The Swift-side interface that calls this:

keyman/ios/engine/KMEI/KeymanEngine/Classes/Keyboard/KeymanWebViewController.swift

Lines 231 to 251 in 6b8a06a

    
           func setText(_ text: String?) { 
        
             var text = text ?? "" 
        
             // Remove any system-added LTR/RTL marks. 
        
             text = text.replacingOccurrences(of: "\u{200e}", with: "") // Unicode's LTR codepoint 
        
             text = text.replacingOccurrences(of: "\u{200f}", with: "") // Unicode's RTL codepoint (v1) 
        
             text = text.replacingOccurrences(of: "\u{202e}", with: "") // Unicode's RTL codepoint (v2) 
        
             do { 
        
               let encodingArray = [ text ]; 
        
               let jsonString = try String(data: JSONSerialization.data(withJSONObject: encodingArray), encoding: .utf8)! 
        
               let start = jsonString.index(jsonString.startIndex, offsetBy: 2) 
        
               let end = jsonString.index(jsonString.endIndex, offsetBy: -2) 
        
               let jsonText = jsonString[start..<end] 
        
               self.currentText = String(jsonText) 
        
               webView!.evaluateJavaScript("setKeymanVal(\"\(jsonText)\");", completionHandler: nil) 
        
             } catch { 
        
               SentryManager.captureAndLog(error.localizedDescription) 
        
             } 
        
           }

Well, huh. That should handle it, right? Well... here's the thing...

So... there's no match, the byte-header doesn't get filtered, do not pass Go, do not collect $200, etc. Of course, that did work before... so a recent-ish version of iOS must have changed how the byte-header gets encoded. The filter-block was written in #1756 for version 14.0 in May 2019, and there have been quite a few new versions of iOS since then.

Also, I'd noticed weirdness with predictive-text even on master - how an extra backspace was needed after reaching cleared text at the start of context for predictions to work right. It's definitely the same thing - our pred-text engine doesn't filter out byte-order marks either.

jahorton · 2023-11-22T03:35:00Z

Given the investigation and related behaviors I've noted, it's a bug on the current master and thus should be handled separately.

jahorton · 2023-11-22T04:13:03Z

And, of course, fixing the fact that context-resets weren't resetting deadkeys (a la #10039)... caused me to run into a different context-sync issue, at least on iOS. Now checking if it also happens on Android.

Core issue there: there are two text updates during a multitap: 'restore original context' and then 'apply new text effects'. If text is updated from the app based on the first and not the second, while the JS side has done both, that causes a context desync.

…multitap context-reversion

jahorton · 2023-11-22T06:06:16Z

As that last comment is something only relevant for feature-gestures - because of multitaps - I went ahead and fixed that here.

bharanidharanj · 2023-11-22T10:31:40Z

Test Results

TEST_10_KEY_DIACRITICS (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "English - Diacritic 10-key Rota" keyboard. 2. Tapped the ◌̀ key. Then, double-tapped the 8 Key. Verified that the expected result à appears in the text area. 3. Triple-tapped the ◌̀ key. Triped tapped the 9 key. Verified that the àê letters appeared on the screen. 4. Tapped the '◌̀' key. Then, tapped the 4 key once. Hit backspace. Double-tapped the 8 key showed the expected result àềa.

..Double-tap 8 key

..Triple-tap 9 key

..Double-tap the 8 key after the change

bharanidharanj · 2023-11-22T11:33:44Z

Test Results

TEST_APP_10_KEY_DIACRITICS (PASSED): Retested using the Keyman for Android test artifact.. from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Installed the attached Keyman build 17.0.209-alpha-test-9973 in the device. 2. Downloaded and then installed diacritic_rota.kmp file in Keyman App. 3. Tapped the 7 key twice. Verified that 77 appears on the screen. 4. After running step 5, able to see 77à as a result. 5. After running step 7, able to see the 77àê as a result. 6. After running step 11, able to see 77àềa as a result.

..Double-tap the 8 key

..Triple-tap the 9-key

..Double-tap the 8th key

bharanidharanj · 2023-11-22T13:09:39Z

Test Results

TEST_BASIC_SIMPLE_SHIFT (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. After running step 3, it is verified that the keyboard remains on the shift layer.
TEST_BASIC_MODIPRESS (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. After running step 8, it is verified that the keyboard changed to the default layer.
TEST_BASIC_MODIPRESS_HOLD (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. After running step 6, it is verified that the keyboard changed to the default layer due to step 6.

bharanidharanj · 2023-11-22T13:26:24Z

Test Results

TEST_NUMERIC_FROM_SHIFT (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. Verified that ‱ is emitted as text and the keyboard changed back to the shift layer, after running steps 6 and 8. 3. Verified that the keyboard automatically reverted to the default layer, after running step 11.

..After running step 7

..After running step 12

bharanidharanj · 2023-11-22T13:48:47Z

Test Results

TEST_DELAYED_SUBKEY (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. Verified that the keyboard keyboard changed to the numeric layer after running step 4. 3. Also, verified that the keyboard changes remain on the numeric layer, with the subkey menu still displayed, after running step 8. 4. Verified that ‱ is emitted as text and that the layer has changed back to the shift layer.

bharanidharanj · 2023-11-22T13:57:47Z

Test Results

TEST_DOUBLETAP_CAPS (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Norwegian" keyboard. 2. Verified that it is showing the expected outputs after running from 2 to 8.

bharanidharanj · 2023-11-22T14:11:42Z

Test Results

TEST_CUSTOM_MULTITAP_MODIFIER (FAILED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. Noticed that after running step 4, it is showing the wrong output in the text screen.

bharanidharanj · 2023-11-22T14:42:28Z

Test Results

TEST_ALTERNATING_SHIFT_AND_KEY (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. Verified that an H is output and the keyboard layer returns to default layer after running step 4. 3. Verified that there is no deviations in the output after running step 7. 4. Verified that the layer switches properly to keys with numbers in the top row and symbols in the middle two rows, after clicking the 'num' key.

..H is the output after running step 5

..No deviation in the output after running step 7

..Numeric key appears after running step 8

bharanidharanj · 2023-11-22T15:02:52Z

Test Results

TEST_FLICKS_BASIC (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. After running step 4, it is verified that it outputs ù on the screen. 3. After running step 6, it is verified that it outputs ù on the screen. 4. After running step 8, it is verified that it outputs û on the screen.

bharanidharanj · 2023-11-22T15:35:50Z

Test Results

TEST_FLICK_CORRECTION (FAILED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. After running step 4, it is verified that it outputs ù on the screen. 3. After running step 8, it is observed that the u letter appears on the screen. seems to be an issue. 4. After running step 11, it is verified that it outputs u on the screen.

jahorton · 2023-11-23T03:31:21Z

Test Results

TEST_CUSTOM_MULTITAP_MODIFIER (FAILED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. Noticed that after running step 4, it is showing the wrong output in the text screen.

Could you please elaborate on what the error is here? In the screenshot above... isn't that the expected output from the a key from the layer reached by triple-tap?

To be clear - from this layer:

... wait a second. Step 4 - the first key. And yep, test-spec error again. That output is correct for the layer you should be reaching. Man, I did not proofread the tests with the varied diacritic forms correctly.

TEST_CUSTOM_MULTITAP_MODIFIER (PASSED): In which I update a test spec again; had the wrong output specified for a step, but the test results were correct for actual (and intended!) behavior.

jahorton · 2023-11-23T03:39:23Z

Test Results

TEST_FLICK_CORRECTION (FAILED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. After running step 4, it is verified that it outputs ù on the screen. 3. After running step 8, it is observed that the u letter appears on the screen. seems to be an issue. 4. After running step 11, it is verified that it outputs u on the screen.

TEST_FLICK_CORRECTION (PASSED): In which it turns out another test spec needed updating to match current behavior. I think I wrote that test before the locked-flicks stuff was fully implemented; it makes sense for step 8 to emit that 'u', now that I inspect it more closely.

I've gone ahead and adjusted the test-spec accordingly.

bharanidharanj · 2023-11-23T09:40:01Z

Test Results

TEST_FLICK_LOCKING (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. Tapped and hold the 'u' key. 3. Dragged it purely straight-up position, and I was able to see the letter û. Observed that there was no lack of motion after dragging it. 4. After moving back my finger to the original position, the 'u' key slides back into place. 5. Tapped and hold the 'u' key, then drag it in an up-and-right motion. I was able to see a preview of ú key. Observed that no animated preview happened. Lift my finger and end the gesture. Verified that a ú outputs on the screen. 6. Followed steps 2 through 6 for the 'a' key and I am able to see the similar previews and text there. Seems to be working fine. Thanks.

..u text animated preview

..a text-animated preview

bharanidharanj · 2023-11-23T10:46:38Z

Test Results

TEST_MODIPRESS_MULTITAP_FLICK_PREVIEW (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. Double tapped then hold the Shift key. Tapped and hold the U key. Verified that the key preview is visible. 3. Dragged the finger on U purely north (straight up) and I was able to see Û key in the preview area. 4. Verified that the flick-animation in the preview scrolls to a Û. 5. Release the flick, when the Û key is fully visible. Verified that Û is output. 6. Repeated steps 2 through 6 quickly three or four times and verified that the Û appears repeatedly as the output. Seems to be working fine.

bharanidharanj · 2023-11-23T11:07:17Z

Test Results

TEST_FLICK_DURING_MODIPRESS (PASSED): Retested using the standard "Test unminified KeymanWeb" test page from a mobile device (Samsung Galaxy A23 - Android 13) and here is my observation: 1. Selected "Gesture Prototyping" keyboard. 2. Tapped and hold the Shift key. Tapped and hold the U key. 3. Dragged it to straight-up position. I was able to see the Û key without "jumpy" in the preview area. Released the Shift key. Verified that the shift layer remains visible and that the flick animation is still active. 4. Returned the finger to its original position, able to see the U key slide back into place. 5. Dragged the finger in an up-and-right motion and I was able to see the Ú key in the preview. 6. While the Ú is visible in the center, tapped on the q key with the other finger. Verified that the keyboard returns to the default layer and Úq is output on the screen. Seems to be working fine.

keymanapp-test-bot bot added the user-test-missing User tests have not yet been defined for the PR label Nov 8, 2023

keymanapp-test-bot bot added the epic-gestures label Nov 8, 2023

keymanapp-test-bot bot changed the title ~~feat(web): delayed modipress completion~~ feat(web): delayed modipress completion 🐵 Nov 8, 2023

keymanapp-test-bot bot added this to the A17S25 milestone Nov 8, 2023

github-actions bot added web/ common/ common/web/ feat labels Nov 8, 2023

mcdurdin modified the milestones: A17S25, A17S26 Nov 13, 2023

jahorton added 4 commits November 13, 2023 15:49

feat(web): delayed modipress completion with nested subkeys

a213531

feat(web): delayed modipress completion with nested flicks

fabf5a4

fix(web): modipress-flick robustness; properly updates layer for comp…

1dc94fc

…letion-triggering source

fix(web): resetting locked flicks during a modipress (without cancell…

f47fd0d

…ing)

jahorton changed the base branch from change/web/polish-pass to feat/web/flick-lock-polish November 13, 2023 08:50

jahorton force-pushed the feat/web/delayed-modipress-completion branch from deb35e2 to f47fd0d Compare November 14, 2023 02:56

jahorton mentioned this pull request Nov 14, 2023

change(web): locked flicks 🐵 #9957

Merged

jahorton changed the base branch from feat/web/flick-lock-polish to change/web/polish-pass November 14, 2023 07:11

chore(web): Merge branch 'change/web/polish-pass' into feat/web/delay…

e44e5f3

…ed-modipress-completion

jahorton mentioned this pull request Nov 14, 2023

change(web): gestures polish pass 🐵 #9972

Merged

keymanapp-test-bot bot added has-user-test user-test-required User tests have not been completed and removed user-test-missing User tests have not yet been defined for the PR labels Nov 14, 2023

jahorton marked this pull request as ready for review November 14, 2023 08:28

jahorton requested a review from mcdurdin as a code owner November 14, 2023 08:28

mcdurdin approved these changes Nov 15, 2023

View reviewed changes

keymanapp-test-bot bot removed the user-test-failed label Nov 22, 2023

jahorton mentioned this pull request Nov 22, 2023

fix(web): app/webview did not clear deadkeys on context-reset #10039

Merged

jahorton mentioned this pull request Nov 22, 2023

bug(ios): LTR / RTL mark not filtered when diacritics are attached #10040

Closed

change(web): app/webview - single transform per keystroke, even with …

31cc1f8

…multitap context-reversion

keymanapp-test-bot bot added the user-test-failed label Nov 22, 2023

keymanapp-test-bot bot removed the user-test-failed label Nov 23, 2023

keymanapp-test-bot bot removed the user-test-required User tests have not been completed label Nov 23, 2023

jahorton merged commit 6fdd966 into feature-gestures Nov 24, 2023
18 of 19 checks passed

jahorton deleted the feat/web/delayed-modipress-completion branch November 24, 2023 04:36

jahorton mentioned this pull request Nov 24, 2023

epic: new gesture engine 🐵 #7324

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(web): delayed modipress completion 🐵 #9973

feat(web): delayed modipress completion 🐵 #9973

jahorton commented Nov 8, 2023 •

edited

keymanapp-test-bot bot commented Nov 8, 2023 •

edited

mcdurdin Nov 15, 2023

jahorton Nov 15, 2023 •

edited

mcdurdin Nov 15, 2023

bharanidharanj commented Nov 15, 2023

bharanidharanj commented Nov 15, 2023

bharanidharanj commented Nov 16, 2023

jahorton commented Nov 22, 2023 •

edited

jahorton commented Nov 22, 2023 •

edited

jahorton commented Nov 22, 2023

jahorton commented Nov 22, 2023

jahorton commented Nov 22, 2023

bharanidharanj commented Nov 22, 2023 •

edited

bharanidharanj commented Nov 22, 2023 •

edited

bharanidharanj commented Nov 22, 2023

bharanidharanj commented Nov 22, 2023

bharanidharanj commented Nov 22, 2023

bharanidharanj commented Nov 22, 2023

bharanidharanj commented Nov 22, 2023

bharanidharanj commented Nov 22, 2023

bharanidharanj commented Nov 22, 2023

bharanidharanj commented Nov 22, 2023

jahorton commented Nov 23, 2023 •

edited

Test Results

jahorton commented Nov 23, 2023

Test Results

bharanidharanj commented Nov 23, 2023 •

edited

bharanidharanj commented Nov 23, 2023

bharanidharanj commented Nov 23, 2023

feat(web): delayed modipress completion 🐵 #9973

feat(web): delayed modipress completion 🐵 #9973

Conversation

jahorton commented Nov 8, 2023 • edited

Notes for future revisitation

User Testing

keymanapp-test-bot bot commented Nov 8, 2023 • edited

User Test Results

Test Artifacts

mcdurdin Nov 15, 2023

Choose a reason for hiding this comment

jahorton Nov 15, 2023 • edited

Choose a reason for hiding this comment

mcdurdin Nov 15, 2023

Choose a reason for hiding this comment

bharanidharanj commented Nov 15, 2023

Test Results

bharanidharanj commented Nov 15, 2023

Test Results

bharanidharanj commented Nov 16, 2023

Test Results

jahorton commented Nov 22, 2023 • edited

jahorton commented Nov 22, 2023 • edited

jahorton commented Nov 22, 2023

jahorton commented Nov 22, 2023

jahorton commented Nov 22, 2023

bharanidharanj commented Nov 22, 2023 • edited

Test Results

bharanidharanj commented Nov 22, 2023 • edited

Test Results

bharanidharanj commented Nov 22, 2023

Test Results

bharanidharanj commented Nov 22, 2023

Test Results

bharanidharanj commented Nov 22, 2023

Test Results

bharanidharanj commented Nov 22, 2023

Test Results

bharanidharanj commented Nov 22, 2023

Test Results

bharanidharanj commented Nov 22, 2023

Test Results

bharanidharanj commented Nov 22, 2023

Test Results

bharanidharanj commented Nov 22, 2023

Test Results

jahorton commented Nov 23, 2023 • edited

Test Results

jahorton commented Nov 23, 2023

Test Results

bharanidharanj commented Nov 23, 2023 • edited

Test Results

bharanidharanj commented Nov 23, 2023

Test Results

bharanidharanj commented Nov 23, 2023

Test Results

jahorton commented Nov 8, 2023 •

edited

keymanapp-test-bot bot commented Nov 8, 2023 •

edited

jahorton Nov 15, 2023 •

edited

jahorton commented Nov 22, 2023 •

edited

jahorton commented Nov 22, 2023 •

edited

bharanidharanj commented Nov 22, 2023 •

edited

bharanidharanj commented Nov 22, 2023 •

edited

jahorton commented Nov 23, 2023 •

edited

bharanidharanj commented Nov 23, 2023 •

edited