Skip to content

v2.6.0

Latest

Choose a tag to compare

@github-actions github-actions released this 14 Nov 05:31
· 3 commits to main since this release
aaa3a86

A file can now be taken apart into a checked tree of its parts and rebuilt byte for byte, a folder of CMMNS saves becomes tables, and a video bundle gains frame sheets and a faster run.

  • gather video-bundle writes contact sheets of frames across the whole video without downloading it. The frames come from the site's preview storyboard (the thumbnails the scrub bar shows), read from the info yt-dlp already returns: 24 to 96 frames chosen evenly across the duration, 6 to a row and 24 to a sheet, the time printed on each, as <id>.frames-NN.jpg (about 2000 pixels wide) with <id>.frames.tsv (time, sheet, row, column, pixel box, frame number). Storyboard thumbnails are small, 320 by 180 pixels at best on YouTube, and <id>.video.json states their real size under frames; a video with no storyboard records frames: none offered. At most 24 sprite images are read per video, 1 second apart. Cutting needs Pillow, the new optional extra frames (pip install "gather-engine[frames]"); without it the record says the sheets were not cut. --no-frames skips the stage.
  • Add gather video-bundle --fast. Measured on one 234-second video, the 3-second sleeps the batch used were about 90% of a 162-second run, and 45 of its 47 comment requests read replies: 300 comments held 3 top-level comments and 297 replies. --fast keeps yt-dlp's requests 1 second apart, gives replies a tenth of the comment budget (at most 5 to a thread, set with --max-replies), and reads the caption track and the storyboard while the same yt-dlp call is still reading comments. One extraction still serves everything: yt-dlp writes its info to a file once the page is read (--print-to-file at pre_process). The same video took 37 seconds with frame sheets, 270 of its 300 comments top-level. A rate limit is retried with the same bounded backoff as before, and a bot check or a refused sprite image stops its stage without a retry.
  • gather video-bundle --json reports timings (seconds per stage) and frames. Add gather.pacing.HostLimiter: a rate limit by host that waits only for the time still owed since the last request to the same host.
  • Add gather saves FOLDER --out DIR: a folder of posts and pages saved by the CMMNS browser extension (daylight.sidecar/1 Markdown sidecars, their JSON sidecars and media files) becomes items, entities and media tables as CSV and JSON Lines, with a manifest.json (gather.saves-tables/1) that holds row counts, table hashes and per-file problems. Fixed rules only: no model call, no network call, no subprocess. The same folder gives byte-identical output on every run. The language column is a rule over Unicode scripts and stop words, entities are regular expressions, and a media field that cannot be read is empty.
  • Add gather saves --probe FILE and gather.media_probe: container from magic bytes, size, picture size, duration, sample rate and channels for PNG, JPEG, GIF, WebP, MP4, MOV, M4A, WebM, Matroska, WAV, FLAC, Ogg and MP3, read with the standard library by seeking. --no-media-hash skips the media SHA-256.
  • Add gather decompose FILE --out DIR and gather rebuild DIR --out FILE: one local file becomes a tree (gather.decompose-tree/1) with one record per part (kind, offset, length, SHA-256, parameters, handler), and the tree rebuilds the file byte for byte with every node checked. The handler is chosen from the file's bytes. Structural handlers: ZIP (entries decoded into nested trees within recorded depth and size limits; entry names never become paths), PNG, WAV, JSON, text and the CMMNS Markdown sidecar with its JSON companion. Bytes no handler understands stay as raw ranges, so every file rebuilds exactly.
  • gather decompose also writes a content level, kept apart from the structure: components.csv and components.jsonl plus tables. For PCM WAV: sample peak, RMS, crest factor, DC offset, integrated loudness (ITU-R BS.1770, gated), and tempo, key, spectrum and decay estimates. For PNG: a palette table. For text, Markdown and HTML: headings and references. Compressed media and tool-dependent components get a row that says why they did not run.