Skip to content
BranchesTags

Repository files navigation

Whisper Screen Tools

System-wide voice input, voice commands, focus protection, screen recording, screenshots, and an Electron image annotator for Windows.

The app is built around global hotkeys: hold a key to dictate text, tap a key to capture the screen, or open the screenshot editor and save annotated PNG + JSON metadata for later AI review.

Features

  • Push-to-talk voice typing with faster-whisper.
  • Voice command hotkey for spoken commands such as task kill <app>.
  • Focus-lock hotkey to reduce foreground focus stealing by apps such as Revit.
  • Plain screenshot capture with frozen-screen region selection.
  • Screenshot image editor with notes, drawing tools, shapes, text boxes, colors, grid snap, zoom, pan, selection, copy/cut/paste, undo/redo, and radial tool wheel.
  • Annotated screenshot export as screenshot.png plus annotations.json.
  • Video region recording with recording.gif, recording.mp4, and a labeled frames.png grid.
  • Clipboard integration: paths are copied after successful voice/screen operations.
  • Activity logging in the screen output directory.

Requirements

Requirement Details
OS Windows 10/11 or Linux (Wayland-Hyprland tested, X11 works for most subsystems)
Python 3.10+
Node.js Needed for the Electron annotator
Microphone Any input device PortAudio can see
GPU Optional NVIDIA CUDA for faster transcription

Installation (Windows)

git clone https://github.com/KisliPlug/whisper_tool.git
cd whisper_tool

python -m venv venv
.\venv\Scripts\activate
pip install -r requirements.txt

npm install
npm run build:annotator

For NVIDIA GPU acceleration, install CUDA runtime wheels as needed:

pip install nvidia-cublas-cu12 nvidia-cudnn-cu12

The first Whisper model load downloads the selected model automatically.

Installation (Linux)

On Arch, run the bundled installer — it pulls system packages with pacman, builds a venv, installs pip + npm deps, and builds the Electron annotator bundle:

git clone https://github.com/KisliPlug/whisper_tool.git
cd whisper_tool

./install.sh             # full install (asks for sudo when pacman is needed)
./install.sh --no-cuda   # skip the NVIDIA wheels

To do it by hand on any distro:

git clone https://github.com/KisliPlug/whisper_tool.git
cd whisper_tool

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

npm install
npm run build:annotator

System packages used by the Linux backend (Arch package names, your distro may differ):

sudo pacman -S wl-clipboard grim slurp ydotool libnotify ffmpeg
# Optional: tk if you want the Tk fallback for the screenshot region picker
sudo pacman -S tk
# Optional CUDA wheels (same as Windows):
pip install nvidia-cublas-cu12 nvidia-cudnn-cu12

ydotool needs a running daemon and access to /dev/uinput:

# Run ydotoold as your user (one of the usual recipes):
sudo modprobe uinput
sudo chown $USER /dev/uinput
ydotoold &

Without ydotool, voice transcription still works — text lands on the clipboard, but auto-paste into the focused window is disabled.

Run

# Windows
python main.py

# Linux foreground mode (Ctrl+C stops it)
./run.sh

# Linux background mode (detached, logs to run.log)
./run.sh --bg
./run.sh status
./run.sh stop
./run.sh restart --bg

Run in the background with the tray supervisor:

.\Start WHISPER.cmd

The tray icon keeps Whisper alive without an open terminal. Its menu opens activity.log, stdout/stderr logs under logs\, restarts Whisper, or stops it. If the Python process crashes, the tray restarts it automatically up to five times per minute before giving up and turning the icon red.

With a custom config:

python main.py --config my_config.yaml
./run.sh --config my_config.yaml --bg

List microphone devices without starting Whisper:

python main.py --list-input-devices

Linux hotkeys via Hyprland

Global key polling like Windows does isn't available on Wayland. Bind the same keys in ~/.config/hypr/hyprland.conf to trigger the daemon over its Unix socket. Start the daemon first with ./run.sh --bg:

# Hold-to-talk voice input (F21)
bind   = , F21, exec, ~/path/to/whisper_tool/venv/bin/python ~/path/to/whisper_tool/main.py --trigger f21 press
bindr  = , F21, exec, ~/path/to/whisper_tool/venv/bin/python ~/path/to/whisper_tool/main.py --trigger f21 release

# Hold-to-talk voice command (F19)
bind   = , F19, exec, ~/path/to/whisper_tool/venv/bin/python ~/path/to/whisper_tool/main.py --trigger f19 press
bindr  = , F19, exec, ~/path/to/whisper_tool/venv/bin/python ~/path/to/whisper_tool/main.py --trigger f19 release

# Tap-to-toggle screen captures
bind   = , F20,        exec, ~/path/to/whisper_tool/venv/bin/python ~/path/to/whisper_tool/main.py --trigger f20 tap
bind   = , F17,        exec, ~/path/to/whisper_tool/venv/bin/python ~/path/to/whisper_tool/main.py --trigger f17 tap
bind   = , F18,        exec, ~/path/to/whisper_tool/venv/bin/python ~/path/to/whisper_tool/main.py --trigger f18 tap
bind   = CTRL, F17,    exec, ~/path/to/whisper_tool/venv/bin/python ~/path/to/whisper_tool/main.py --trigger "ctrl+f17" tap

The trigger name is whatever you put in config.yaml for that action (hotkey, command_hotkey, focus_lock_hotkey, screen.*_hotkey) — the daemon treats the spec as an opaque label on Linux.

The socket path defaults to $XDG_RUNTIME_DIR/whisper-tool.<uid>.sock (or /tmp if XDG_RUNTIME_DIR is unset). Override with WHISPER_TOOL_SOCKET=/path/to/socket if you run multiple instances.

Default Hotkeys

Action Default
Voice typing Hold F21, release to transcribe and paste
Voice command Hold F19, release to execute command
Focus lock Tap Ctrl+F17
Screen recording Tap F20 to select/start, tap F20 again to stop
Plain screenshot Tap F17, select region
Annotated screenshot Tap F18, select region, edit, tap F18 again to commit
Exit app Ctrl+C in the terminal, or tray icon -> Stop Whisper

Hotkeys are configured in config.yaml.

Voice Input

Hold the voice hotkey, speak, then release. The recognized text is copied to the clipboard and pasted into the previously focused window.

Important config keys:

hotkey: "f21"
input_device: null
input_sample_rate: "auto"
language: "auto"
allowed_languages: ["ru", "en"]
model_size: "small"
device: "auto"
compute_type: "int8"
beam_size: 1
insert_mode: "clipboard"

input_device can be null for the Windows default microphone, an integer device index from python main.py --list-input-devices, or a device name substring. input_sample_rate: "auto" opens the microphone at its default capture rate and resamples to sample_rate for Whisper.

Supported model sizes include tiny, base, small, medium, and large-v3.

Speed vs quality

The three inference knobs — model_size, compute_type, beam_size — set the trade-off. Pick a preset as a starting point, then tune.

Preset model_size compute_type beam_size Notes
CPU laptop (fast) small int8 1 Default. ~10× faster than medium/CUDA.
CPU laptop (quality) small int8 5 Slightly better on noisy audio.
CPU + medium medium int8 1 If small drops Russian words.
NVIDIA GPU (fast) medium float16 5 Recommended desktop default.
NVIDIA GPU (best) large-v3 float16 5 Only viable with CUDA.

Notes:

  • tiny / base are unusable for Russian — skip them.
  • float16 on CPU silently falls back to float32 in ctranslate2, which is ~2× slower than int8 for the same quality. Always set int8 on CPU.
  • beam_size: 1 is greedy decode — ~3-4× faster than beam_size: 5 with ~95% of the quality on clean microphone input.
  • Models are auto-downloaded on first use and cached in ~/.cache/huggingface/hub. Switching model_size only re-downloads the first time you use a new size.

Voice Commands

Hold the command hotkey, speak a command, and release. Current command support is intentionally narrow:

task kill <app>

Examples:

  • task kill revit
  • task kill chrome
  • Russian app-name aliases are handled in app/commands.py.

Important config keys:

command_hotkey: "f19"
command_language: "en"

Focus Lock

Focus lock reduces unwanted focus stealing while you are capturing or dictating.

  • Windows: cranks SPI_SETFOREGROUNDLOCKTIMEOUT so SetForegroundWindow calls from other apps get downgraded to a taskbar flash.
  • Linux/Hyprland: toggles misc:focus_on_activate via hyprctl (the closest analogue — activatewindow requests turn into a subtle highlight instead of a focus steal).
focus_lock_hotkey: "ctrl+f17"
focus_lock_auto_enable: true

Screen Recording

Tap the video hotkey, select a region, and recording starts. Tap the same hotkey again to stop.

Each recording creates a timestamped folder containing:

  • recording.gif
  • recording.mp4
  • frames.png

The output folder path is copied to the clipboard.

Relevant config:

screen:
  enabled: true
  video_hotkey: "f20"
  video_fps: 15
  output_dir: "~/Documents/records"

Screenshots

Plain screenshot:

  1. Tap screenshot_hotkey.
  2. Select a region on a frozen snapshot.
  3. The app saves screenshot.png.
  4. The PNG file path is copied to the clipboard.

Annotated screenshot:

  1. Tap screenshot_edit_hotkey.
  2. Select a region on a frozen snapshot.
  3. The Electron annotator opens above other windows.
  4. Draw notes, shapes, arrows, text, and markups.
  5. Click Save in that annotator window.
  6. The app saves screenshot.png and annotations.json.
  7. The output folder path is copied to the clipboard.

You can open multiple annotator windows at the same time. Each window saves independently when it closes; the clipboard is updated by the most recently saved screenshot or annotation folder.

All screenshot save paths update the clipboard with the saved file or folder path. If multiple annotator windows are open, the one saved last wins.

Relevant config:

screen:
  screenshot_hotkey: "f17"
  screenshot_edit_hotkey: "f18"

Image Annotator

The editor is an Electron + React app launched by the Python screen controller.

Main capabilities:

  • Multiple notes per image.
  • Note names and note text.
  • Select, pen, line, arrow, rectangle, ellipse, and text tools.
  • Radial tool wheel instead of a permanent tool sidebar.
  • Color picker, stroke size, text size, grid size, and snap settings.
  • Move, resize, delete, copy, cut, paste, duplicate, and multi-select.
  • Undo/redo.
  • Zoom, fit, 100%, and middle-mouse pan.
  • Inline multi-line text boxes.
  • Freehand pen ignores grid snap for intermediate points so strokes stay smooth.

Annotator controls:

Action Control
Open radial wheel Right mouse button over the image, or top tool pill
Pan view Hold middle mouse button
Zoom Ctrl+wheel, Ctrl++, Ctrl+-, Ctrl+0
Undo/redo Ctrl+Z, Ctrl+Y
Copy/cut/paste objects Ctrl+C, Ctrl+X, Ctrl+V when not editing text
Delete selection Delete or Backspace when not editing text
Save annotated screenshot Click Save
Save clean PNG without JSON Click Save as PNG
Copy saved result path Any save action

The Electron window title is Whisper Screenshot Annotator. On this machine, GlazeWM is configured to ignore that title so the editor floats instead of being tiled.

Tool shortcuts use physical key codes as a fallback, so they still work when the keyboard layout is not English.

Annotation Metadata

Annotated screenshot folders contain:

screenshot.png
annotations.json

annotations.json uses original screenshot pixels:

{
  "version": 1,
  "image": "screenshot.png",
  "size": { "width": 1280, "height": 720 },
  "coordinate_space": "screenshot_pixels",
  "notes": [
    {
      "id": 1,
      "name": "Note 1",
      "text": "User note text",
      "items": [
        {
          "id": 1,
          "type": "rectangle",
          "bbox": [420, 180, 610, 240],
          "color": "#ff2a2a",
          "color_name": "red",
          "width": 4
        }
      ]
    }
  ]
}

Supported item types:

  • rectangle
  • ellipse
  • line
  • arrow
  • freehand
  • text

Configuration Reference

Top-level keys:

Key Description
hotkey Voice typing hold hotkey
command_hotkey Voice command hold hotkey
command_language Language forced for command recognition
focus_lock_hotkey Focus lock toggle hotkey
focus_lock_auto_enable Enable focus lock on startup
language Voice recognition language, or auto
allowed_languages Allow-list for auto language detection
input_device Optional microphone index or name
input_sample_rate Microphone capture sample rate, or auto
input_keepalive_ms Keep mic stream warm after recording
model_size Whisper model size
device auto, cuda, or cpu
compute_type faster-whisper compute precision
beam_size Decode beam size
sample_rate Audio sample rate
min_duration Ignore shorter recordings
stream_interval Streaming transcription interval
sound_feedback Beeps for record start/stop
insert_mode clipboard or typing

Screen keys:

Key Description
screen.enabled Enable screen tools
screen.video_hotkey Toggle region recording
screen.screenshot_hotkey Plain screenshot
screen.screenshot_edit_hotkey Screenshot editor
screen.video_fps Recording frame rate
screen.output_dir Capture output root

Development

Build the annotator:

npm run build:annotator

Run basic Python compile validation:

python -m compileall -q app main.py

Use Vite during annotator UI work:

npm run annotator:dev

Set ANNOTATOR_DEV_URL before launching the Python flow if you want Electron to load the dev server instead of dist/index.html.

Project Structure

whisper_tool/
  main.py                         App entry point and hotkey wiring
  config.yaml                     User configuration
  requirements.txt                Python dependencies
  package.json                    Electron/Vite/React dependencies
  app/
    commands.py                   Voice command parser/executor
    config.py                     Config loader
    focus_lock.py                 Focus-lock public shim → app/platform
    hotkey.py                     Windows GetAsyncKeyState hotkey poller
    notifier.py                   Tk-based notification popup (Windows)
    recorder.py                   Microphone capture
    transcriber.py                faster-whisper wrapper
    platform/
      __init__.py                 Platform-dispatching facade
      base.py                     Shared protocols / dataclasses
      windows.py                  Win32 backend (winsound, keybd_event, …)
      linux.py                    Linux backend (wl-copy, ydotool, hyprctl, grim/slurp)
      linux_ipc.py                Unix-socket trigger transport for Hyprland hotkeys
      _focus_lock_win.py          Win32 SPI_SETFOREGROUNDLOCKTIMEOUT impl
    screen/
      controller.py               Screen capture state machine
      electron_annotator.py       Python/Electron bridge
      annotator.py                Tk fallback annotator
      exporter.py                 GIF/MP4/frame-grid exporter
      selector.py                 Region picker
      video.py                    Region video recorder
  electron/
    annotator/
      main.js                     Electron main process
      preload.cjs                 IPC bridge
      src/App.jsx                 React annotator
      src/styles.css              Annotator styling

Troubleshooting

  • If the annotator opens in a GlazeWM tile, reload GlazeWM config with Alt+Shift+R or:
glazewm.exe command wm-reload-config
  • If the annotator does not open, run npm install and npm run build:annotator.
  • If a screenshot image path contains spaces or non-ASCII characters, the Electron bridge should use a proper file:// URL. This is handled in electron/annotator/main.js.
  • If video export fails, check that imageio-ffmpeg is installed from requirements.txt.
  • If the microphone hangs at startup, check input_device and Windows microphone permissions.
  • Linux: if python main.py --trigger ... reports "socket not found", check ./run.sh status. The daemon may not be running, or XDG_RUNTIME_DIR differs between the Hyprland session and the daemon's environment — set WHISPER_TOOL_SOCKET explicitly to pin the path.
  • Linux: if voice transcription works but text never auto-pastes, ydotool is missing or ydotoold isn't running. The text still lands on the clipboard — paste with Ctrl+V manually.
  • Linux/Wayland: if mss can't grab the screen (some compositors restrict X11 desktop reads), the controller transparently falls back to grim. Make sure grim and slurp are installed.

License

MIT

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages