Enable spatial navigation into and within Qt elements with arrow keys #264
Conversation
This commit does several things: - Introduce central arrow Up/Down arrow handling into MainWidget for spatial navigation - Translate focus-shift:exhausted events into Qt key events - Make kiosk button styling uniform and focusing more visible - Prevent kiosk buttons from handling spatial navigation - Remove the multi-action dialog that is redundant now
It would be possible to split this script into multiple more specific ones, but 1) we use all of it on all pages unconditionally; 2) it would require more complicated QWebChannel handling (e.g. via a CustomEvent) for sharing among the sub-scripts, making the JS code more complex.
Currently it errors out with
js: Uncaught ReferenceError: qt is not defined
when there are any sub-frames in the page.
Not sure what is the root cause/ and fix.
This implies that opaque mode will not work in sub-frames, however
sub-frames are anyway non-focusable with the remote control anyway, so
it does not matter.
This would lead to some None's along the way, causing a crash. This seems to be purely synthetic / dev env related (e.g. while navigating inside of Chrome DevTools).
After digging a bit deeper into QWebEngine's and Chromium userscript
guts, it's more clear what is going on:
- `window.qt.webChannelTransport` is the Chromium IPC transport and is
created by QWebEngine itself when you call `setWebChannel`. It will be
available only in the specified worldId (which is MainWorld by
default) and ONLY in the top-level window (not on iframes). The latter
seems to be a strange limitation in the implementation.
- "structured clone" is not allowed between worlds, which is why scripts
running in different worlds can see the dispatched events, but cannot
read `event.detail` if it is an object. `event.detail:
JSON.stringify(...)` is fine though
- If two (user)scripts expect to communicate, they have the following
options:
1. Run in the same worldId
2. Use window.dispatchEvent, but with event.detail of type
string (avoids structured clone)
3. Use runtime.sendMessage
4. Use window.postMessage
- The above also applies if a user script wants exchange events with a
script running on the page (e.g. focus-shift embedded in Play, or
kiosk dispatching events to Play)!
These points imply that we either run FocusShiftBridge (and hence
FocusShiftScript) on MainWorld or... if we want to run
in a more isolated setup (e.g. in ApplicationWorld), then we have the
following options:
a. JSON.stringify all `event.details`
b. Create a separate "Proxy" script that runs on MainWorld and
handles event forwarding/translation between the page and the
user scripts. This can be done in several ways (again, via
window.postMessage, runtime.sendMessage, or even regular DOM
events, but with detail as string).
Since it does not seem like we need the extra isolation currently (e.g.
i.e. don't expose anything dangerous via the QWebChannel), MainWorld is
probably good enough.
|
Discovered one minor issue/quirk: if you focus back into the web view and the focus is restored on a text/numeric input field, then it will not be possible to open the virtual keyboard without focus-shifting inside the web view. There are no events emitted by Qt, so it's not a bug in our code. It is very similar to the "refocus" Qt bug reported a while ago. We can try to handle it with various hacks, but since this currently can only happen on pages where the top-most element is a text/numeric input (i.e. not currently on Play or PlayOS, might be some captive portals) and since it is recoverable (just move focus around), I would say it can be left as is. |
Good hygiene, also avoids strange interactions with some pages that somehow re-eval user-scripts.
|
@knuton discovered a case of
Note that it seems to work fine in all other pages, so this is something quite specific. |
|
Looks good, yet to perform practical tests. |
|
Works nicely overall. When I open the kiosk on my laptop with check URL for (I have not tested on a physical device yet, will do.) 
Edit: When I made the screenshot, focus seems to have moved to the button. 🙃 Anyway, this is the screen I am talking about. |
Good catch.
Yes, most likely. I will do some testing with an empty page and an infinitely loading page to check what can be done.
Classic scenario 💀 |
knuton
added
changes suggested
Ensures that the currently visible view has focus and key events are properly propagated. This fixes situations where network errors or pages that are stuck in loading prevent the user from being able to shift focus to the other Qt widgets.
Should be fixed in 750ca7d. Tested with both an infinitely loading page and a network error scenario. |
yfyf
added
reviewable
2 participants
...and also deal with a bunch orthogonal annoyances, including:
As usual with Qt, this turned out to be quite elaborate and quite silly, but is the best I can come up with. It should be fairly robust to various re-arrangements and already handles situations like "double stacking" when in PlayOS settings (i.e. dialog) and captive portal message is visible.
I tried and/or considered several different approaches, but they seemed even messier:
focusNextPrevChildbehaviour to prevent wrapping - this seemed way more cumbersome and requires to track/propagate info about the "top" widgetRemaining questions/issues, a bit orthogonal:
focus-shift:exhaustedevents do not trigger on many pages I tried, making it impossible to focus on the close button in Network login (captive portals).Checklist
Changelog updatedUser manual updated