Add element-based navigation mode #1
Conversation
The sandbox isn't compatible with the Carbon Accessibility API (so, for example, you can't query for `AXUIElement`s from other apps). Thanks to tmandry/AXSwift#16 for the tip.
(As well as GridWindowController and GridViewController to JumpWindowController and JumpViewController, respectively.)
(This complements the existing grid-based navigation functionality.) Applicable elements are determined using the underlying accessibility APIs provided by macOS (c/o AXSwift).
There's no benefit to drawing multiple copies of the same frame -- this just adds visual noise and clutters the decision tree.
(There are still legibility issues to work through, but this seems a bit better in most cases.)
|
Legibility is usually OK-ish, but far from perfect. (Help making this better is appreciated!) Here's what HN looks like, for example...
The combination of a small font size, with many unique anchors in close proximity, is problematic. It might make sense to ignore elements that are clustered too closely together (using size as a proxy for importance). |
|
Another issue: if the focused window in the frontmost app has a scroll view, and you pull up Scoot (in element-based nav mode), and then start scrolling (either using Scoot's keyboard shortcuts, or using your physical mouse or trackpad), the regions that Scoot has drawn won't update. In the case where you're scrolling with Scoot, it should be possible to properly handle this. However, the latter might be a problem: are there any accessibility-related hooks to detect if the user is scrolling? |
As per #1, `⇧⌘J` now activates Scoot in the element-based nav mode. This particular demo uses grid-based nav, which has a new keyboard shortcut: `⇧⌘K`.
This PR introduces "element-based" navigation. 🎉
When this element-based nav mode is active, Scoot will use MacOS accessibility APIs to find the user interface elements on the screen (specifically, those in the focused window of the frontmost app at the time that Scoot is invoked).
Here's what this looks like, in Safari on example.com:
Scoot identifies the only link on the page ("More information..."), and assigns it the key sequence "aa". (As always, type "aa" to teleport the mouse cursor directly there.) Scoot doesn't just find the content on the web page though -- it also identifies the browser chrome/ UI elements. For example, type "k" to move the cursor to the "show tab overview" button.
Another example, on a web page with (a few) more links:
Here's Music's MiniPlayer: