Repository navigation
A file can now be taken apart into a checked tree of its parts and rebuilt byte for byte, a folder of CMMNS saves becomes tables, and a video bundle gains frame sheets and a faster run.
gather video-bundlewrites contact sheets of frames across the whole video without downloading it. The frames come from the site's preview storyboard (the thumbnails the scrub bar shows), read from the info yt-dlp already returns: 24 to 96 frames chosen evenly across the duration, 6 to a row and 24 to a sheet, the time printed on each, as<id>.frames-NN.jpg(about 2000 pixels wide) with<id>.frames.tsv(time, sheet, row, column, pixel box, frame number). Storyboard thumbnails are small, 320 by 180 pixels at best on YouTube, and<id>.video.jsonstates their real size underframes; a video with no storyboard recordsframes: none offered. At most 24 sprite images are read per video, 1 second apart. Cutting needs Pillow, the new optional extraframes(pip install "gather-engine[frames]"); without it the record says the sheets were not cut.--no-framesskips the stage.- Add
gather video-bundle --fast. Measured on one 234-second video, the 3-second sleeps the batch used were about 90% of a 162-second run, and 45 of its 47 comment requests read replies: 300 comments held 3 top-level comments and 297 replies.--fastkeeps yt-dlp's requests 1 second apart, gives replies a tenth of the comment budget (at most 5 to a thread, set with--max-replies), and reads the caption track and the storyboard while the same yt-dlp call is still reading comments. One extraction still serves everything: yt-dlp writes its info to a file once the page is read (--print-to-fileatpre_process). The same video took 37 seconds with frame sheets, 270 of its 300 comments top-level. A rate limit is retried with the same bounded backoff as before, and a bot check or a refused sprite image stops its stage without a retry. gather video-bundle --jsonreportstimings(seconds per stage) andframes. Addgather.pacing.HostLimiter: a rate limit by host that waits only for the time still owed since the last request to the same host.- Add
gather saves FOLDER --out DIR: a folder of posts and pages saved by the CMMNS browser extension (daylight.sidecar/1Markdown sidecars, their JSON sidecars and media files) becomesitems,entitiesandmediatables as CSV and JSON Lines, with amanifest.json(gather.saves-tables/1) that holds row counts, table hashes and per-file problems. Fixed rules only: no model call, no network call, no subprocess. The same folder gives byte-identical output on every run. The language column is a rule over Unicode scripts and stop words, entities are regular expressions, and a media field that cannot be read is empty. - Add
gather saves --probe FILEandgather.media_probe: container from magic bytes, size, picture size, duration, sample rate and channels for PNG, JPEG, GIF, WebP, MP4, MOV, M4A, WebM, Matroska, WAV, FLAC, Ogg and MP3, read with the standard library by seeking.--no-media-hashskips the media SHA-256. - Add
gather decompose FILE --out DIRandgather rebuild DIR --out FILE: one local file becomes a tree (gather.decompose-tree/1) with one record per part (kind, offset, length, SHA-256, parameters, handler), and the tree rebuilds the file byte for byte with every node checked. The handler is chosen from the file's bytes. Structural handlers: ZIP (entries decoded into nested trees within recorded depth and size limits; entry names never become paths), PNG, WAV, JSON, text and the CMMNS Markdown sidecar with its JSON companion. Bytes no handler understands stay as raw ranges, so every file rebuilds exactly. gather decomposealso writes a content level, kept apart from the structure:components.csvandcomponents.jsonlplus tables. For PCM WAV: sample peak, RMS, crest factor, DC offset, integrated loudness (ITU-R BS.1770, gated), and tempo, key, spectrum and decay estimates. For PNG: a palette table. For text, Markdown and HTML: headings and references. Compressed media and tool-dependent components get a row that says why they did not run.