Skip to content

v2.51.0

Choose a tag to compare

@samuelgursky samuelgursky released this 12 Jun 02:29

CLAP audio embeddings — the final phase of the post-program improvements (v2.46.1 → v2.51.0). Detection-based like every other backend (never auto-installed); local compute, so nothing touches the caps ledger.

Added

  • Audio backend detection: CLAP via transformers (laion/clap-htsat-unfused, preferred) or the laion_clap package, plus torch + ffmpeg. capabilities gains an audio block with install guidance and a clap_audio Tools-page entry. Activate with pip install transformers (model weights ~600 MB download on first use).
  • build_embeddings(kinds=["audio"]): one CLAP window per shot (center-cropped to ~10 s, piped from the source media as raw PCM — read-only on source media, no temp files) plus a clip-level mean vector, stored as embedding_kind="audio" rows. Idempotent via content hashes; offline media is reported in skipped_missing_media; a missing backend is a graceful skip with install guidance.
  • find_similar(kind="audio"): shot/clip queries over the audio vectors, and free-text queries ("engine revving") via the CLAP text encoder.

Validation

  • Backend mocked in 7 new unit tests (build plumbing, idempotency, clip mean vector, graceful skip, missing media, audio similarity, kind validation); offline suite at 1184.
  • Graceful-skip path verified live on a machine without a CLAP backend: clean errors with install guidance from build_embeddings, find_similar, and the capabilities/Tools surface.