Towards a Mouse Free UX for Most APPs on macOS (WIP)

This tiny tool aims to ease the pain of

1. Mouse selection (mainly text)
2. Selecting actions in the pop-up menu after right clicking
3. Recurring tasks given selected text

• You can think of it as a purely keyboard and open-source version of PopClip with some extra utility features.
• And it allows you to interact with some UI text that is not even possible to select using a mouse. Like those in a video game or in a button.

Demo

Text Manipulation

Text can be extracted from either UI elements or Apple VisionKit OCR results

demo.mp4

Image Copying and Input Text Editing

For users who are not satisfied with the default text editing experiences of editable text fields, this app allows you to edit them in your favorite editor, and automatically sync the saved content back to the UI element.

demo2.mp4

Multi-selection

1. Toggle multi-selection mode on (Shift key)
2. Select starting/ending piece of text
3. Select the other side, and the program will automatically guess the paragraph of intention

Here's an example of how to select and translate the lyrics in Apple Music.

demo3.mp4

Precise selection

If you want to select a specific sub-piece of text in an identified element, you can

1. Split the whole context into pieces, the interface of word picker will pop up
2. Toggle multi-selection mode on within the word picker
3. Select both sides according to the hint keys

Other Features

• UI element tree exploring mode (E)
  • Useful for debugging and screenshot taking
• Apple Dictionary support in simple pop-up window
  • Avoids the hassle of opening the dictionary app, like what PopClip will do
  • Dictionary CSS is respected to make the text more readable

• Easily extensible text actions, please refer to the Configuration section
  • Avoids the hassle of plugin management, easier to share across devices
• Act on text from clipboard
• Customizable theme

Getting Started

1. Install with brew tap and start the service

brew tap blindfs/tools
brew install glyphlow
brew services start glyphlow

2. Grant accessibility permission to it

3. Press the global trigger (defaults to "ALT + g") to start

Purging

This app is designed to be lean and clean, it only generates 2 files:

1. A configuration file $XDG_CONFIG_HOME/glyphlow/config.toml or ~/.config/glyphlow/config.toml if the env-var is not set.
2. A cache file for temporary editing: $XDG_CACHE_HOME/glyphlow/tempfile.md or ~/.cache/glyphlow/tempfile.md.

Configuration

Here's how I configure it to perform those actions shown in the demo videos. A comprehensive configuration file is generated when you run this app at the first time.

colored_frame_min_size = 100
element_min_width = 15
element_min_height = 15
ocr_languages = [
  "zh-Hans",
  "ja-JP",
  "en-US",
]
dictionaries = [
  "牛津英汉汉英词典",
  "New Oxford American Dictionary",
]

[[text_actions]]
display = "󰊭 Google Search"
key = 'G'
command = "nu"
args = ["-c", "r#'{glyphlow_text}'# | url encode | ^open $'https://google.com/search?q=($in)'"]

[[text_actions]]
display = "󰖬 Wikipedia Search"
key = 'W'
command = "nu"
args = ["-c", "r#'{glyphlow_text}'# | url encode | ^open $'https://en.wikipedia.org/wiki/Special:Search/($in)'"]

[[text_actions]]
display = "󰊿 Goolge Translate -> zh_cn"
key = 'T'
command = "nu"
args = ["-c", "r#'{glyphlow_text}'# | url encode | ^open $'https://translate.google.com/?sl=auto&tl=zh_cn&text=($in)&op=translate'"]

[editor]
display = " Editor"
key = 'V'
# command = "tmux"
# args = ["new-window", "-t", "dev", "^open -a Ghostty; ^nvim {glyphlow_temp_file}"]
command = "open"
args = ["-a", "Zed", "{glyphlow_temp_file}"]

[theme]
hint_font = "AndaleMono:16"
menu_font = "IosevkaTerm Nerd Font Mono:26"

Roadmap

1. nix-flake
2. menu bar icon