Skip to content

Releases

ClipMind v1.6.0

Choose a tag to compare

@yy31104 yy31104 released this 18 Nov 21:26
02c842b

ClipMind v1.6.0

Find the moments when on-screen information changes. The full pack still keeps the screenshots, OCR and transcript.
Open Frames → Information changes to browse the marked moments. OCR now checks dark code panels and light overlays.

On four complete held-out recordings, Mac recall was 88.51%, precision 56.20% and code CER 10.21%. Docker CPU reached 88.51%, 56.83% and 7.82%. Both met all three pooled targets.
Separate source-only passes of the same AI model agreed at 96.28% F1 before adjudication.
Interactive demos still produce extra boundaries, and code CER covers two recordings.

Results and source credits · Install
Saved packs stay available. Reprocess a video to apply the new rules.

ClipMind v1.5.0

Choose a tag to compare

@yy31104 yy31104 released this 18 Nov 08:05
300cf93

ClipMind v1.5.0

ClipMind extracts searchable speech, screen text and timestamped images from videos.

Screen states are compared with the last information screen, so speaker, camera, pointer and transition states stay saved but are no longer counted as new information.
Code OCR keeps trailing comments on their line, recovers thin small text on Mac, and reads dark code panels inside light slides with Tesseract.
Development set (ten excerpts plus the former holdout): recall 99.28% → 94.20% on both; precision Mac 23.42% → 41.27%, Docker 23.46% → 34.48%; code CER Mac 15.95% → 10.20%, Docker 17.40% → 12.69%.
New holdout, three complete talks (69 min, English and Mandarin), one run per backend: recall Mac 92.86%, Docker 95.05%; precision 33.87% / 24.96%; code CER 22.45% / 13.18%.
Recall meets 85% on both and Docker code CER meets 15%. Precision misses 50% on both; Mac code CER misses 15%. No tuning or rerun after the result.
AI-assisted annotations; human review pending. The holdout review package has a blank CSV.
Results and causes · Holdout protocol · Install
Reprocess existing videos to apply the new rules; saved packs are preserved.

ClipMind v1.4.0

Choose a tag to compare

@yy31104 yy31104 released this 17 Nov 00:15

ClipMind v1.4.0

ClipMind extracts searchable speech, screen text and timestamped images from videos.

Progressive builds keep fuller images and actual capture times. Supported speaker motion is labeled separately. Code OCR improves contrast and reading order; native worker failures have bounded cleanup.
Speech language defaults to detection. Tesseract prefers English first while retaining Chinese models; overrides remain available. Tasks show an analysis-time range.
Development capture: 44/60 → 59/60. Code CER: Mac 39.29% → 6.24%; Docker 54.30% → 23.61%.
Single holdout campaign, three complete talks: both backends capture 73/78 (93.59%). Precision is Mac 20.39% / Docker 20.28%; code CER is 19.00% / 16.08%.
Recall meets 85%; precision misses 50% and code CER misses 15%. All results and failures remain public; no post-result tuning or replacement run.
AI-assisted annotations; human review pending. Known timing/transcription discrepancies remain in the frozen references. All 15 code ROIs are from one English talk; Chinese code OCR is N/A.
Measurements, limitations and comparisons · Review downloads and facts
Install with the versioned GitHub or Docker command, then run clipmind serve.
Reprocess existing videos to apply the new extraction rules; saved packs are preserved.
Temporary media are cleaned. Results, model weights, exports and recoverable trash remain on disk.

ClipMind v1.3.0 — searchable video evidence and one-command setup

Choose a tag to compare

@yy31104 yy31104 released this 16 Nov 13:48

ClipMind turns videos into searchable speech, screen text and timestamped images for people and AI agents.

Install with uv tool install --python 3.12 git+https://github.com/yy31104/clipmind.git@v1.3.0, then run clipmind serve.
For Docker, run docker run --rm -p 127.0.0.1:8420:8420 -v clipmind-data:/data ghcr.io/yy31104/clipmind:v1.3.0.
See platform dependencies; the unrelated PyPI package named clipmind is not this project.
Set CLIPMIND_ASR_LANGUAGE=en for English. This version defaults to Chinese.

Changes: safer acquisition cleanup and cancellation, Bilibili part-aware reuse, Douyin browser fallback, recoverable Library removal, complete-frame browsing and reading exports.
The real demo searches a complete 39-minute talk and uses MCP to retrieve its evidence.
The ten-clip benchmark exposes missed screen changes and weak code OCR. AI-assisted annotations; human review pending.
Temporary media are cleaned; results, model weights, exports and recoverable trash remain. Docker uses CPU inference and has no Chrome fallback.
Wheel, source, demo and benchmark inputs are attached with checksums. Media retain their source attribution and licenses.
The MCP Registry entry is active. No PyPI upload or signed desktop installer is included.

ClipMind v1.2.0 — universal local-first evidence extraction

Choose a tag to compare

@yy31104 yy31104 released this 04 Sep 15:59

Source release. ClipMind grows from a Douyin-only macOS tool into a
local-first video evidence layer with multiple sources, portable providers and
agent-native interfaces. The canonical path still requires no API key.

Sources

Verified built-in adapters for YouTube, Douyin and local files,
plus a clipmind.sources entry point so a third party can add one without
patching the core. A generic URL fallback runs after the built-ins on a
best-effort basis.

Complete or refuse

A low-cost preflight estimates visual states, OCR time and pack size before
any expensive work. Within budget it processes normally; over budget it stops
and reports actionable numbers. It never truncates a pack, never silently
lowers evidence resolution, and never emits something that looks complete but
is not. An explicit override processes the full source.

Portable providers

ASR and OCR sit behind protocols. Apple Silicon uses MLX Whisper and Apple
Vision; Linux and Windows use faster-whisper and Tesseract. Runtime settings
and providers are injected rather than fixed at import, so a job can carry its
own configuration.

Interfaces

Python SDK, REST resources, a full CLI and a stdio MCP server exposing six
tools over the same Evidence Packs — one source of truth, four ways in.

Extraction fidelity

Word-level timing, OCR layout boxes, scene and scroll grouping, stability
measurements, and optional pyannote diarization. All of it is measured and
derived; none of it decides canonical membership. Evidence Pack schema moves
to 1.3.0, additively — every v1 minor version still reads.

Quality gate

make eval audits existing packs offline. make eval-reextract runs real
sources through the current pipeline. Exact canonical counts can veto a change
only when the downloaded source SHA-256 matches the baseline, because the
same URL can return different bytes — a difference this project once mistook
for a code regression. Packs now record which dedupe algorithm chose their
canonical states, so a changed algorithm is distinguishable from changed bytes.

This gate rejected a perceptual-hashing change written for this release. That
change is not in this release.

Also

Browser uploads, local evidence search, productized Inbox/Library/detail views,
Python packaging, cross-platform CI, Docker, and a documented macOS app build
path.

Install

Not on PyPI. pip install from a checkout — see
docs/INSTALL.md.

No .dmg is attached. The macOS build path exists and is documented, but
it produces unsigned development artifacts; shipping one would make a
first-time user's first action be bypassing Gatekeeper. A public desktop
release needs signing, hardening and notarization first.

Full changelog: v1.1.0...v1.2.0

ClipMind v1.1.0 — transcript novelty on every visual state

Choose a tag to compare

@yy31104 yy31104 released this 02 Sep 15:52

Adds a deterministic signal describing how much of a visual state's on-screen
text the speech never carried, and gives preview derivation one safety net.
Evidence Pack schema moves to 1.1.0; packs written by 1.0.0 stay readable.

Why

Burned-in captions repeat what is being said, so the transcript already holds
that information. A slide, document or code pane shows text nobody reads
aloud, and the frame is the only place it exists.

Measured on the 64-second interview video: document frames carry 68–344
characters the nearby speech never covers, caption frames carry about 7.

New fields on every visual state

field meaning
ocr_char_count characters across recognised tokens
transcript_novelty_char_count characters the nearby speech does not cover
transcript_overlap_ratio 1 - novelty / total

Comparison is a multiset over tokens — one per CJK ideograph, one per latin
word — after NFKC normalisation and case folding, windowed around the state's
own interval, so a term shown three times but spoken once is not treated as
fully covered.

What this is not

Not importance. A silent video makes every frame novel, code identifiers are
novel because nobody says them out loud, and an ASR slip manufactures novelty.
The field is a signal for a downstream agent to weigh, not a verdict.

Evidence stays untouchable

Annotation runs after canonical retention, never before, so an ASR or OCR
mistake cannot delete evidence. The preview safety net requires enough
unspoken text and no already-selected state showing substantially the same
terms; on all five real packs it adds zero frames, because every high-novelty
state was already selected. It is insurance against a brief document inside
one long visual scene, demonstrated by a synthetic case rather than asserted.

Compatibility

The published schema accepts 1.0.0 and 1.1.0. Existing packs remain readable
and reusable by the cache; scripts/rebuild_preview.py upgrades one in place,
adding the measurements without re-downloading or re-running OCR.

Also fixed

A failed resync silently restored the stuck state the resync exists to escape.
refreshJobs() now retries with backoff and surfaces a message.

Verification

91 tests (11 new), CI green, linear history.

Full changelog: v1.0.1...v1.1.0

ClipMind v1.0.1 — live progress and library readability

Choose a tag to compare

@yy31104 yy31104 released this 02 Sep 15:41

Maintenance release. No Evidence Pack, persistence or extraction changes; the
v1.0.0 pack contract is unchanged.

Fixed

Stalled progress that never recovered (#1)

When a viewer's tab was backgrounded, its notification queue filled and the
subscriber was silently dropped from the broadcast set. The SSE connection
stayed open and kept emitting keepalives, so the page looked connected while
receiving nothing further — processing continued and the Evidence Pack
completed, but the UI froze at whatever it had last seen.

Overflow now drains the stale notifications, enqueues a single resync
control event and keeps the subscription. The client answers resync the same
way it answers hello: by refetching the authoritative snapshot from
GET /api/jobs. The queue stays bounded and publishing never awaits a slow
subscriber.

A home page that opened on failures (#2)

Failure cards sat between the composer and the library with raw URLs as
titles, and every reprocessing generation appeared as its own card, so three
videos read as seven near-identical entries.

Failures now fold into a single line below the library, packs group by source
video with an older-version count, card metadata shows the date instead of
processing time, and an unfetchable job is labelled rather than titled with
its URL.

Verification

78 tests, CI green on both pull requests, linear history.

Full changelog: v1.0.0...v1.0.1

ClipMind v1.0.0 — Local-first Evidence Packs

Choose a tag to compare

@yy31104 yy31104 released this 02 Sep 14:03

ClipMind v1.0.0 turns Douyin share links into durable, local Evidence Packs on Apple Silicon Macs.

Highlights

  • Free canonical path with no API key or per-use fee
  • Local MLX Whisper transcription and macOS Vision OCR
  • Uncapped canonical visual states plus content-driven previews
  • Readable 1280 px evidence with measured OCR gains over 640 px
  • Versioned, deterministic Evidence Pack schema and ZIP export
  • One-way knowledge-base Inbox handoff
  • Durable jobs, restart recovery, cache reuse, bounded concurrency, and actionable failures

Validation

  • 72 deterministic tests
  • Three real-world extraction profiles covering code/UI, talking-head captions, documents, and slides
  • Generated silent four-slide fixture covering no-audio degradation and the full packaging path
  • Bounded batch benchmark: peak concurrency 4/4 and approximately 3.7× simulated speedup

See the architecture, real-world evaluation, privacy boundary, and known limitations.

The canonical workflow is currently macOS/Douyin-specific. Protected or expired sources may require a usable local Chrome session and a fresh share link.