Surgical Tool
The Surgical Tool checks the on-image text that appears in your captions. When a caption transcribes writing that is visible in the image — a shop sign, a book cover, a slogan on a shirt — the tool reads that same text with optical character recognition (OCR) and compares the two. Where the reading is confident it can confirm or correct the caption automatically; where it is uncertain it hands the decision to you.
Note: In the app this tool is titled Text Surgeon. This page uses the maintainer's name, Surgical Tool, but the two refer to the same tool.
Note: The Surgical Tool is an early, experimental tool and is available to Patreon subscribers. It appears in the toolbar only once an active Patreon membership is linked in Settings, and its background jobs will not run without it. If you do not see the tool, no membership is currently linked.
The tool works on the text already written in your captions, so it complements rather than replaces the Caption Tool. It is a separate tool from the Tagger Tool, which predicts descriptive tags for an image; the Surgical Tool is concerned only with verbatim on-image text.
The tool always works over your active dataset and the folder currently selected within it. A Folder scope bar at the top of the landing screen shows exactly which folder will be processed, and an Include subfolders toggle lets you extend the job to everything beneath it. You do not enter a path by hand; the scope follows the dataset and folder you already have open.
The landing screen offers three cards, each opening its own screen. You can move between them freely, and background jobs keep running while you are elsewhere in Studio.
- Label: A fast, keyboard-first review queue for checking on-image text by hand.
- Auto-Correct: A background job that reads each image with two OCR models and corrects the transcribed text in captions where both models agree.
- Tag: The same dual-model job, but instead of correcting captions it writes the confirmed on-image text into the image's tags.
Auto-Correct and Tag both rely on dual-model agreement. Two OCR models read the same image. A primary model provides the reading that is adopted when a correction is made, and a second model acts as a corroborating reader.
A caption's transcription is only changed automatically when the evidence is strong:
- Both models must read the same text.
- Each reading must match the caption's text closely enough to be trusted.
- Neither reading may be ambiguous — that is, neither model may be torn between two similar candidates.
When all of these hold, the reading either confirms the caption (the text is already correct and is left untouched) or produces a correction that both models support. Anything short of full agreement — the models diverge, only one of them reads anything, or a reading is ambiguous — is never edited silently. It is queued for you to adjudicate in the Label screen instead.
The thresholds behind this are deliberately conservative. A wrong automatic edit to a training caption is more costly than an extra image to review by hand, so the tool errs towards asking.
Auto-Correct runs as a background job on your GPU over the folder in scope. It reports its progress as it works, and because it runs in the background you can leave the screen and return later — the run resumes where it left off.
As each image is processed, the results table fills in with a per-image verdict:
| Verdict | Meaning |
|---|---|
| Approved | The on-image text was confirmed, or corrected with the agreement of both models. No further review is needed. |
| Queued | The models disagreed or were unsure. The image has been added to the Label queue for a manual decision. |
| No text | Neither model found on-image text worth acting on. |
A running summary tallies how many images were auto-approved, how many need Label review, how many corrections were applied, and how many had no text. When the run finishes you can jump straight to the Label queue to work through anything that was set aside.
Corrections are written to your caption files as each image is processed, so stopping the job midway keeps the work already done. If you change your mind about a completed run, Undo this run restores the captions it corrected.
Note: If your GPU cannot hold both models at once, the tool falls back to running them one after another and shows a short notice that it is doing so. The results are the same; the run simply takes longer.
Tag runs the same dual-model agreement job, but rather than editing captions it records the confirmed on-image text as tags on each image. This is useful when you want the text present as structured tags — for example so it feeds into the tag-style captions Studio maintains through Background Sync (see Captioning). Its results table reports how many images were tagged, how many tags were written, and how many were queued or had no text.
Label is a manual review queue for working through on-image text quickly and by hand. It is the place disagreements from Auto-Correct and Tag are sent, but you can also open it on its own. Running Auto-Correct first gives the best queue ordering, because the most confident items are brought to the front and presumed false positives are pushed to the back.
The screen is split in two. On the left, the image is shown with the detected text regions overlaid, and you can zoom and pan to inspect them. On the right, the caption is displayed with each transcribed span highlighted:
- Green spans are ones the OCR reading agrees with.
- Red spans are ones it disagrees with.
Each red span is given a correction block: the caption's wording struck through, the OCR reading proposed as its replacement, and a zoomed crop of that region so you can check it against the image. Selecting a highlighted span frames the matching region on the image.
For each image you can:
- Accept correction: Replace the caption text with the OCR reading.
- Keep original: Approve the caption as it stands.
- False positive: Send the image to the back of the queue when the caption claims text the image does not really contain.
The queue is designed to be driven from the keyboard.
| Key | Action |
|---|---|
| Enter | Stamp (approve the current image) |
| Space | Skip |
| F | Mark as a false positive |
| ← / → | Previous / next image |
| Ctrl+Z | Undo the last decision |
| Esc | Back |
A running counter shows how many images are left and how many you have reviewed. Any decision can be undone, so you can move quickly without fear of losing work.
Sometimes OCR finds no text on an image even though the caption clearly describes some. A common cause is a horizontally mirrored image, where the writing reads back-to-front. When the tool detects this situation — OCR came up empty but the caption names on-image text — it offers a Mirrored text? prompt with a Flip horizontally button. Flipping corrects the image so the text reads normally and can be verified.
Corrections made through Auto-Correct or the Label screen are written back into your caption files, and text confirmed through the Tag mode is stored as tags on the image. Both follow the image around Studio, so the work carries over into the rest of your dataset — the corrected captions appear wherever captions are shown, including in Gallery View, and confirmed text tags feed into the tag-style captions described on the Captioning page.