Two paintings of one page are never the same bytes — what that does to caching, golden tests and replay

[xizhuomengcontin]

A measurement that I think matters for anything that tries to make a vision-driven browser run repeatable, Stagehand's action caching included. It is not about my tooling — it falls out of what a screenshot is.

The measurement

A vision agent puts the screenshot in the request. I recorded one such request, then replayed it twice against the recording: once sending the identical image, once sending a repaint — same page, same size, different pixels, which is the only thing a browser can actually do.

Identical image:

info replay.done reused=1/1 exact=0 divergences=1 unmatched=0 exit=0
warn divergence  request 0 is identical apart from 16 redacted values

Matched.

Repaint:

openai.BadRequestError: 400 - replay halted — request 0 does not match the recording
                              (distance 472, 1 recorded vs 1 replayed messages)
info replay.done reused=0/1 exact=0 divergences=0 unmatched=1 exit=1

Did not match, and the difference was 472 characters on a 16×16 image. On a real viewport screenshot the distance is enormous.

That is the whole point: the page did not change, the intent did not change, the pixels changed — and any byte-level comparison of the prompt is therefore a coin flip you always lose.

What this implies for repeatability features

Anything keyed on "was this request the same as last time" has to decide what it does about the screenshot:

• Key on the screenshot → the key never repeats, and the cache is dead weight on anything vision-driven.
• Exclude the screenshot from the key → the key repeats, but two genuinely different screens can now collide, and you are trusting the DOM/instruction part to carry all the meaning.
• Normalize it (perceptual hash, downscale, quantize) → works, and buys a threshold you now have to tune and explain.

I do not know which Stagehand does for act/observe caching, and I could not tell from the outside — that is really the question behind this post. If the answer is "we key on the accessibility tree and instruction, not the screenshot", that is worth saying loudly in the caching docs, because the failure mode when someone assumes otherwise is silent: a cache that never hits, or one that hits when it should not.

The second hazard: a screenshot looks exactly like a leaked credential

This one cost me real debugging time. A base64 screenshot is a long, high-entropy, random-looking string — which is precisely the shape every secret scanner is built to catch. My own redactor treated one as a leak and shredded it into tens of thousands of placeholders before it reached disk. Each half was individually correct: the entropy sweep is right that a long random-looking run is what a leaked key looks like, and the comparison is right to compare what it was given. Together they made a vision agent impossible to record.

The generalisation, which is the part worth having: if you pipe vision-agent traffic through anything that scans for secrets — DLP, log redaction, a scrubbing proxy, a compliance filter — expect the screenshot to be mangled or to trip an alert, and expect the tool to be technically right about why. Worth a line in any "logging Stagehand traffic" guidance.

Even after fixing that, note the exact=0 in the first result above: the per-run salting of redaction digests means identical input still compares as minor-divergent rather than byte-identical. Byte-equality across runs is harder to get than it looks once anything in the path rewrites values.

Limits of what I measured

• Driven with a synthetic noise PNG through the OpenAI chat-completions vision shape, not through Stagehand itself. The image is deliberately incompressible so it looks like a secret to an entropy heuristic — that is the property under test. I did not run a Stagehand session, so nothing here is a claim about Stagehand's own behaviour; it is a claim about screenshots in prompts, which Stagehand sends.
• 16×16 image, one request, one model shape. The direction of the result is robust; the magnitude scales with image size.

Tooling was OrcaReplay (Apache-2.0), which I maintain — disclosed so you can weigh it, though the finding is about image bytes and holds whatever records them.

[rhkiswani]

Matches what we found building replay for browser runs: the only durable key is the action and its target, never the pixels. Our replay records what the agent did as a deterministic script (effectively generated Playwright) and reruns it with new inputs. Screenshots are kept as evidence per step, not as part of any key. We tried a perceptual hash, the threshold ends up site specific and the failure is silent in exactly the way you describe.

For anyone storing screenshots in a trail: keep them as separate binary artifacts referenced by content hash and keep the log line to the hash. Inline base64 in logs is what trips every secret scanner, and the scanner is right. If the trail has to be tamper evident, chain the artifact hash into the log entry.

On what act/observe caching keys on, I don't know either. Agree it belongs in the caching docs in one sentence, both failure modes are invisible from outside.