Skip to content

ClipMind v1.1.0 — transcript novelty on every visual state

Choose a tag to compare

@yy31104 yy31104 released this 02 Sep 15:52
· 132 commits to main since this release

Adds a deterministic signal describing how much of a visual state's on-screen
text the speech never carried, and gives preview derivation one safety net.
Evidence Pack schema moves to 1.1.0; packs written by 1.0.0 stay readable.

Why

Burned-in captions repeat what is being said, so the transcript already holds
that information. A slide, document or code pane shows text nobody reads
aloud, and the frame is the only place it exists.

Measured on the 64-second interview video: document frames carry 68–344
characters the nearby speech never covers, caption frames carry about 7.

New fields on every visual state

field meaning
ocr_char_count characters across recognised tokens
transcript_novelty_char_count characters the nearby speech does not cover
transcript_overlap_ratio 1 - novelty / total

Comparison is a multiset over tokens — one per CJK ideograph, one per latin
word — after NFKC normalisation and case folding, windowed around the state's
own interval, so a term shown three times but spoken once is not treated as
fully covered.

What this is not

Not importance. A silent video makes every frame novel, code identifiers are
novel because nobody says them out loud, and an ASR slip manufactures novelty.
The field is a signal for a downstream agent to weigh, not a verdict.

Evidence stays untouchable

Annotation runs after canonical retention, never before, so an ASR or OCR
mistake cannot delete evidence. The preview safety net requires enough
unspoken text and no already-selected state showing substantially the same
terms; on all five real packs it adds zero frames, because every high-novelty
state was already selected. It is insurance against a brief document inside
one long visual scene, demonstrated by a synthetic case rather than asserted.

Compatibility

The published schema accepts 1.0.0 and 1.1.0. Existing packs remain readable
and reusable by the cache; scripts/rebuild_preview.py upgrades one in place,
adding the measurements without re-downloading or re-running OCR.

Also fixed

A failed resync silently restored the stuck state the resync exists to escape.
refreshJobs() now retries with backoff and surfaces a message.

Verification

91 tests (11 new), CI green, linear history.

Full changelog: v1.0.1...v1.1.0