RFC: Persistent lastSeen cache for findText / click(text) — ×25 to ×150 speedup on repeated OCR lookups

[julienmerconsulting]

@RaiMan @adriancostin6 — story first, design questions second.

🎯 The story

Two weeks ago I shipped #353 — the persistent lastSeen cache for image find via a PNG ancillary chunk. The idea was honest: the second time you look for the same image, you have already paid the bill once, so the second lookup should not pay it again. The chunk holds the last-seen coordinates inside the PNG itself, the next find() checks the cached ROI first, falls back to the full region if the element has moved. Measured speedup on cold-start find: ~×6.5.

It worked. Then yesterday I started a session on a notepad with the word "Espagne" on screen, ran a script that calls findText("Espagne") eight times in a row, and timed it. Each call took roughly six seconds. Eight calls, forty-eight seconds. The text never moves between two consecutive calls of the same test step. Tesseract was scanning the entire 1920×1080 screen, every single time, just to confirm something it had already located one second earlier.

I asked myself the obvious question: why don't we have the same lastSeen pattern for text that we have for images? The data shape is the same — a coordinate, a width, a height, a screen resolution stamp. The fallback semantics are the same — if the cached ROI no longer contains the target, scan the full region and update. The only thing #353 has that this one cannot reuse is the PNG ancillary chunk as a carrier: text has no PNG file to attach to.

So I built it with a JSON sidecar instead. PoC is now validated on a live notepad with two scenarios. Measurements below.

🔧 What the patch does

Single funnel injection in Region.doFind — the private method through which every text-finding API converges (findText, existsText, waitText, hasText, click(text), doubleClick(text), rightClick(text), hover(text), find(text), dragDrop(text, ...)). One modification at the funnel = full coverage.

A new persistence class org.sikuli.support.TesseractLastSeen exposes three public methods: getRegion(searchRegion, text), put(searchRegion, text, match), invalidate(searchRegion, text). The JSON file tesseract-lastseen.json lives in System.getProperty("user.dir") — the JVM working directory, which colocates the cache with the user's test project so it can be committed to a repo, shared across CI runners, and survives across runs. Out of ~/.oculix/ on purpose: this is a project asset, not a per-user setting.

The cache key is a pair (text, context) where context is the absolute bounds of the search region ("x,y,w,h"). This matters because the same text can live in two different sub-regions of the screen at the same time: notepad.findText("Save") and chrome.findText("Save") must not overwrite each other. Each pair gets its own cache slot.

Before (cold scan on every call):

if (findingText) {
  log(logLevel, "doFind: Switching to TextSearch");
  finder = new Finder(this);
  finder.findText(someText);
}

After (cached ROI first, fallback on miss):

if (findingText) {
  log(logLevel, "doFind: Switching to TextSearch");
  Region cachedRoi = TesseractLastSeen.getRegion(this, someText);
  if (cachedRoi != null) {
    finder = new Finder(cachedRoi);
    finder.findText(someText);
    if (!finder.hasNext()) {
      TesseractLastSeen.invalidate(this, someText);
      finder = new Finder(this);
      finder.findText(someText);
      shouldCacheTextHit = true;
    } else {
      cachedRoiUsed = cachedRoi;
    }
  } else {
    finder = new Finder(this);
    finder.findText(someText);
    shouldCacheTextHit = true;
  }
}

And after match = finder.next(), a coordinate translate so the Match returned by a cached-ROI scan is expressed in the same frame of reference as a full-region match:

if (cachedRoiUsed != null) {
  match.x += cachedRoiUsed.x - this.x;
  match.y += cachedRoiUsed.y - this.y;
}
if (shouldCacheTextHit && findingText) {
  TesseractLastSeen.put(this, someText, match);
}

JSON shape:

{
  "version": 1,
  "entries": {
    "Save": {
      "100,100,600,400":   { "x":280,  "y":315, "w":40, "h":18, "screenW":1920, "screenH":1080, "lastSeen":"2026-06-22T14:18:00Z", "hits":3 },
      "700,100,800,600":   { "x":750,  "y":120, "w":40, "h":18, "screenW":1920, "screenH":1080, "lastSeen":"2026-06-22T14:18:00Z", "hits":5 }
    },
    "Logout": {
      "0,0,1920,1080":     { "x":1880, "y":12,  "w":60, "h":20, "screenW":1920, "screenH":1080, "lastSeen":"2026-06-22T14:18:00Z", "hits":12 }
    }
  }
}

📊 Measurements

Live test on Windows 10, 1920×1080, full-HD, Tesseract 5.5.0 via Legerix, OculiX 3.0.5-SNAPSHOT (branch claude/log-modes), Java 17 Temurin. The target word "Espagne" was displayed in a plain Notepad window. Wall-clock timing was captured around Screen().existsText("Espagne") with System.nanoTime().

Test 1 — empty cache → 8 sequential runs (sparse screen):

Run	x	y	w	h	Duration	Code path
1	671	773	87	19	6404 ms	cache empty → full scan + put
2	672	773	86	19	236 ms	cache hit on cached ROI
3	672	773	86	19	223 ms	cache hit
4	672	773	86	19	343 ms	cache hit
5	672	773	86	19	230 ms	cache hit
6	672	773	86	19	230 ms	cache hit
7	672	773	86	19	229 ms	cache hit
8	672	773	86	19	249 ms	cache hit

Run-2-to-8 average: 249 ms. Speedup vs Run 1: ×25.7.

Test 2 — stale cache (window moved between Test 1 and Test 2) → 8 sequential runs (dense screen):

Run	x	y	w	h	Duration	Code path
1	1721	198	115	24	12881 ms	cache stale → cached ROI empty → invalidate → fallback full + put
2	1721	198	114	24	208 ms	cache hit on new ROI
3	1721	198	114	24	202 ms	cache hit
4	1721	198	114	24	198 ms	cache hit
5	1721	198	114	24	209 ms	cache hit
6	1721	198	114	24	204 ms	cache hit
7	1721	198	114	24	206 ms	cache hit
8	1721	198	114	24	192 ms	cache hit

Run-2-to-8 average: 203 ms. Speedup vs Run 1: ×63.4.

Key observation — the speedup grows with screen text density. Run 1 of Test 2 was twice as long as Run 1 of Test 1 (12881 ms vs 6404 ms) because the screen was visibly fuller of text at that moment, and Tesseract scales linearly with the amount of text it has to OCR. Cached-ROI calls are constant ~200 ms regardless of the rest of the screen, because they only OCR the tight ROI. Extrapolated asymptote: on a very dense dashboard screen where a full scan can climb to 30-40 s, the cached path remains ~200 ms — that's ×150 and growing.

The fallback overhead is not what you might fear. In Test 2, Run 1 took 12881 ms; ~250 ms of that was the failed cached-ROI scan (Tesseract on 87×19 pixels — negligible), ~5 ms was the JSON invalidate write, the remaining ~12.6 s was the full scan that any non-cached path would have paid anyway. The proper fallback overhead is around 2 %, dominated entirely by the underlying full scan.

🌿 Why this matters

This isn't a niche optimization. The dominant OculiX workflow in the wild is automation scripts that replay the same UI navigation against the same target screens — login pages, kiosk interfaces, AS/400 terminal sessions, embedded HMI dashboards. In all these cases, the same text is searched repeatedly at the same place. The current behavior pays the full OCR cost every single time. The PoC removes that cost from call 2 onward, deterministically, with zero new API surface and zero behavior change for callers.

It also extends a pattern OculiX has already adopted (lastSeen for images, #353) to a complementary dimension (lastSeen for text). The two caches coexist independently: image find uses the PNG chunk, text find uses the JSON sidecar. Same shape, same semantics, two different carriers because the two data structures don't have the same natural attachment point.

❓ Open design questions

#	Question	My current PoC pick	Alternatives worth debating
1	Catalog file name	tesseract-lastseen.json	text-lastseen.json / oculix-text-cache.json
2	Catalog location	user.dir (JVM working dir)	~/.oculix/ (per-user) / configurable
3	Stale entry behavior	auto-invalidate then fallback	keep + flag stale / configurable
4	hits counter	bumps on put only (cache write)	bump on every hit (extra disk write per call)
5	Scope on findAllText	excluded (cache holds one bbox, not a list)	cache the first occurrence / cache all occurrences
6	Activation	always-on by default	opt-in CLI flag / opt-out / config setting
7	Padding around cached ROI	0 px (strict, deterministic)	small fixed pad / adaptive pad (size of text)

I'd love feedback on these specifically. The PoC works as-is, but the right answers here shape what 3.0.5 (or 3.0.6) actually ships.

🔗 Related

• [Feature]: Persistent lastSeen via PNG ancillary chunk — ~×6.5 find speedup, zero config #353 — persistent lastSeen for image find (PNG ancillary chunk, the conceptual sibling of this proposal)
• [Feature]: OCR perf — cap upscale to 0.8 on large search regions (full-screen findText ~2× faster, zero accuracy loss) #378 — OCR findText perf — cap upscale to 0.8 on large search regions (~×2 OCR speedup, complementary, not competing)

---

🦎 — feedback welcome from anyone who has a heavy OCR workflow and would put this through its paces.

[RaiMan]

@julienmerconsulting
Great idea - clapping hands.

your questions:

• at 1: oculix-text-cache.json (open for additions in the future)
• at 2: default ~/.oculix/ and configurable (see below)
• at 3: your choice is ok
• at 4: what does it bring for the usage - I don’t see any value - obsolete
• at 5: cache all occurrences (json sample already has 2 entries for Save), but might also be an option for the future, if asked for
• at 6: keep it in mind for the future, based on usage feedbacks

configurable: IMHO all text related options should live in OCR.options

generally: as far as I understand: the scope of the .json currently is the user. The png.lastseen implementation is per project or even per script (depends on how the user organises the images). I think, this should be possible for the text.lastseen also.
But you might also decide to not implement this now and wait for reactions.