Skip to content

Screen Translator

Jump to bottom
P3DROVFX edited this page Oct 12, 2026 · 1 revision

Screen Translator

Press Super + Shift + T and the screen freezes; a moment later every piece of foreign text on it has been replaced, in place, by its translation — same position, same angle, and (as far as it can tell) the same background and text colour, so a menu, a manga page, a game screenshot or a PDF reads as if it had been written in your language. It is the shell's equivalent of Google Lens "translate", running as an overlay on your own desktop.

It reads the screen with Google Cloud Vision and translates with Google Cloud Translation, authenticated with a service account key you provide. Unlike the free Translator (which needs no account), this one does need that one-time setup — the screen has to be understood as an image, which is the Vision API's job.

Quick start

  1. Set up Google Cloud once (below). The first time you press Super + Shift + T the overlay opens with a key button (and an error asking you to Set your Google Cloud service account key); paste the service-account JSON and press Confirm. It is stored in the system keyring.
  2. Focus the monitor you want and press Super + Shift + T (or run qs -c ii ipc call screenTranslator translate, or bind the Translate Screen Content action to a touch gesture, see Tablet Family). The current monitor is captured and frozen.
  3. Wait through Uploading image → Reading image → Getting ready to translate → Translating. The translated text appears over the original.
  4. Esc or the close button leaves. Scroll zooms the frozen screen (0.1× to 5×) around the pointer and drag pans it, so you can inspect small text; the toolbar has a language picker (System Language, English, Português, Español, Français, Deutsch, 日本語, 中文, 한국어, Русский) to translate into something other than your UI language, a key button to change the credentials, and Close.

One-time Google Cloud setup

You need a Google Cloud project (the free tier covers casual use — pricing is per image and per character; see Google's pages) with:

  1. The Cloud Vision API and the Cloud Translation API enabled.
  2. Billing enabled on the project (Google requires it for these two APIs even within the free allowance).
  3. A service account (IAM & Admin → Service accounts → Create) with permission to call both APIs, and a JSON key (Keys → Add key → JSON).

Paste the JSON into the panel. It must be valid JSON (a key with no project_id is reported as an error), and is stored in the keyring as googleCloud.serviceAccountKey, and from then on exchanged for short-lived access tokens: services/gCloud/token_from_key.py (Google's google-auth library) runs inside the shell's private Python virtualenv, returns a token and its expiry, and is called again only when the token has expired. The account is independent of the OAuth client used for Gmail / Tasks / Calendar / Drive — see Google Cloud Setup.

How it works

frozen screenshot ──▶ base64 ──▶ Vision  images:annotate  (DOCUMENT_TEXT_DETECTION)
                                      │  blocks / paragraphs with rotated bounding boxes
                                      ▼
                         coherent paragraphs  (GCloudVisionResult, confidence ≥ 0.5)
                                      │  each paragraph = one string
                                      ▼
                       Translation v3  projects/<id>:translateText  (text/plain, one call, many strings)
                                      │
   original screenshot, blurred under each box ◀── translated paragraph drawn in the box
  • Capture. The overlay holds a still ScreencopyView of the monitor that had focus when you pressed the key; a screenshot of it is written to a temp folder and base64-encoded.
  • Reading. Vision's DOCUMENT_TEXT_DETECTION returns pages, blocks and paragraphs with four-point polygons. The shell merges them into coherent paragraphs (GCloudVisionResult, with a 0.5 confidence threshold) and keeps the polygon, so rotated or slanted text is drawn at the same angle.
  • Translating. All paragraph strings go in a single Translation request. A paragraph that comes back unchanged (it was already in your language) is left alone.
  • Painting. Under each translated box the original text is blurred away with a masked blur, then the translation is drawn in a squeezed-to-fit text item. For each box a small helper (scripts/images/text_color.py, via ImageMagick magick … -crop of just that box) estimates the background and text colour from the pixels, so the replacement matches; the box fill is a semi-transparent tint of that background. Changing the language re-runs only the translation, not the reading.
  • Privacy. The screenshot of the focused monitor is sent to Google Cloud Vision and the recognized text to Google Cloud Translation, both under your project and credentials; nothing goes through any other server. The screenshot never leaves your temp folder otherwise. Don't use it on a screen with something you must not send to Google (a password manager, banking).

Requirements

  • Hyprland with screencopy (wlr-screencopy — standard), curl, jq, ImageMagick (magick), and the shell's Python virtualenv (created by the installer; it contains google-auth).
  • A working network; every use costs two API calls to Google.

Troubleshooting

Symptom Fix
Set your Google Cloud service account key Paste a valid JSON key with the key button
Error panel with a link to "setup instructions" The message under Screen Translator names what failed (invalid key, API not enabled, billing). Fix it in the Google Cloud console
Token error after the key was fine The key was deleted or the project's APIs were disabled; the shell re-reads the key and retries when the token becomes ready
Nothing is translated though text is visible The language already matched the target, or Vision's confidence was too low on stylised text
Colours look off on gradients or video The colour estimate samples each box once; it is a heuristic

Under the hood

Piece Role
modules/ii/screenTranslator/ScreenTranslator.qml Global shortcut screenTranslate, IPC screenTranslator translate, one panel on the focused monitor
ScreenTranslatorPanel.qml Frozen screen, pan/zoom, toolbar, language picker, key input
ScreenTextOverlay.qml Vision → paragraphs → translation → blurred background and text items
modules/common/models/gCloud/ GCloudApi (state machine), GCloudVision, GCloudVisionResult, GCloudTranslate
services/GoogleCloud.qml, services/gCloud/token_from_key.py Service-account key and tokens
scripts/images/text_color.py Per-box colours

Related pages

← Back to Home

Clone this wiki locally