SuperDuperMegaCrunch: roll pack pipeline, editorial signals + hardening#33
Merged
Conversation
… TSV) The foundation for click-precise + denoised OCR: returns each word with its pixel box and confidence, plus reading-order text reassembled from a confidence floor (default 50) that drops dense-UI garble. Boxes are in screen.mp4 pixel space — which roll now also emits click/cursor coords in — so a click resolves to the exact word under it.
The screen side, denoised and made precise (edator's #1 ask): - ProcessPack now word-OCRs each frame via the new /ocr/words (tesseract TSV). - CrunchAssembler builds screen spans from confidence-filtered reading-order lines (drops the dense-UI garble), and adds ocr_at_click: the exact OCR line under the cursor — resolvable now that roll emits click coords in screen.mp4 pixel space. - on_screen_text stays the ax.label (stable for edator's roll-actions binding). OcrClient::words() + the sidecar /ocr/words endpoint. 107 tests green, phpstan L7 clean.
…hrome filter
Root-caused the garbled crunch.json on-screen text to two things, proven side-by-side on
real pack frames:
1. FrameExtractor wrote lossy JPEG — ringing artifacts on sharp UI text turned a clean
"x402" into "x42" and invented stray leading quotes. Now lossless PNG.
2. The pack pipeline read screen text via word-level image_to_data with a conf-50 filter,
which splits/garbles dense UI text. Replaced with line-level OCR: join every word in a
tesseract line (== image_to_string quality on a clean frame) with a line bounding box.
Plus two index-quality fixes in CrunchAssembler:
- fuzzy span-merge: dedup lines on a normalised key so OCR/punctuation jitter collapses to
one span instead of N near-duplicates (kills the span inflation), cleanest read wins.
- drop top-of-frame OS chrome (macOS menu bar) via a relative pixel cutoff.
/ocr/words now returns { lines:[{text,conf,box}], text, image_height }. ocr_at_click resolves
a click to the line whose box covers it. 109 tests green, phpstan+pint clean.
…ored)
Moments were purely mechanical (app-switches + record-button clicks) — no signal for where
the *content* matters. Add three scored, source-tagged moment streams an editor can rank or
threshold:
- audio/emphasis: vocal loudness peaks via ffmpeg EBU R128 momentary loudness (perceptual,
no new dep). 'You leaned in here.'
- audio/pause: silence runs >=600ms — natural cut points, timed at the pause start.
- transcript/cue_phrase: spoken editorial cues ('the important thing', 'look at this',
'here's the thing'...), word-boundary matched, strongest-per-beat, scored by cue strength.
New AudioProsody support class (ffmpeg I/O thin; parse + peak/pause detection pure & unit-tested
against a synthetic envelope). CrunchAssembler merges telemetry + transcript + prosody moments,
each now tagged with a 'source'. 115 tests green, phpstan+pint clean.
…, CPU-only) Turns the flat firehose into a story outline. Boundaries from signals we already detect (an app switch = hard scene change, a long pause = soft one), min 8s per segment. Each beat labelled with NO generative model: - title: the window it mostly lived in (fallback app, then keyword) - keywords: most distinctive spoken words (frequency, stopwords stripped) - summary: the spoken sentence densest in those keywords New Segmenter support class (pure, unit-tested); CrunchAssembler emits a top-level 'segments'. Default-constructed so it stays DI-free. All tests green, phpstan+pint clean.
…mers-php) A 'give it a go' generative capability: distilbart-cnn-6-6 ONNX via transformers-php, same warm-engine pattern as rerank/sentiment. Small model, CPU-only (no GPU, no heavy LLM) — rough gist, not polished prose, by design. Standalone endpoint; deliberately NOT wired into the proven pack pipeline yet. Mocked endpoint test. All green.
DB_QUEUE_RETRY_AFTER defaulted to 90s, but a pack job (OCR + transcribe a whole recording) runs several minutes with an 1800s timeout. Laravel deems the still-running job stale after 90s, another worker re-reserves it, and with tries=1 it's instantly failed. Every recent multi-minute pack run hit this. Default bumped to 2100s (> the job timeout) so the visibility window clears the runtime.
…, ocr_at_click) The async result is documented as prose (Scramble can't infer the job payload), so it drifted behind a day of changes — it still described only click_on/app_switch moments and no segments. Now lists the current shape: transcript, screen, events (+ocr_at_click), scored+sourced moments (telemetry/transcript/audio), and segments.
The pack pipeline used the OCR sidecar's line-level read (text + pixel box + confidence per line) directly, so it wasn't callable standalone — only flat page text was public via /ocr. Expose it as POST /ocr/lines (tesseract, the box-emitting engine): reading-order lines with boxes+conf, joined text, image height. Same OcrClient::lines() the pack uses, now first-class + documented + tested. Prosody and segmentation stay internal by design — they're pack-specific signal/logic, not encoders.
…CI gates feature/** - config/telescope.php: default TELESCOPE_ENABLED to false (opt-in). Telescope must never record in prod — it bloats the DB and floods the error tracker under Octane. (Prod already has the env set; this makes it safe-by-default for any fresh deploy.) - README: drop the knife, document /pack (the roll Cruncher: transcript × screen × events + scored moments + segment outline), /summarize, /ocr/lines; refresh the roadmap. - CI: also run tests + lint on feature/** pushes, not just PRs to main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidates a sprint of work into
main. Everything is green (121 tests, phpstan L7, Pint) and already validated live on the Netcup box.Roll pack pipeline (the "Cruncher")
POST /pack→ asynccrunch.json: the join of transcript × on-screen-text × events on one shared clock, plus scoredmomentsand asegmentsoutline. Controller → Job → Action; hardened archive unpack (path-traversal + symlink guards).OCR quality (root-caused + proven on real frames)
image_to_stringquality) with per-line boxes for click hit-testing, replacing the conf-50 word path that garbled dense UI text.Editorial signals (CPU-only, no GPU, no generative LLM)
momentsnow scored +source-tagged:telemetry(click/app-switch),transcript(spoken cues),audio(vocal emphasis + pauses via ffmpeg EBU R128).segments— beat-by-beat outline (extractive title/keywords/summary), cut at app switches + pauses.New composable endpoints
POST /summarize— abstractive summary (distilbart-cnn, CPU, rough-but-fun).POST /ocr/lines— the pack's line-level OCR, surfaced standalone (text + box + conf).Reliability + hygiene
retry_after90 → 2100 — long pack jobs were being deemed stale after 90s and failed as "attempted too many times". Root cause of the recent pack failures./packOpenAPI description refreshed (segments, richer moments, ocr_at_click). README covers the pack pipeline. CI now gatesfeature/**.Validated on the notrobo pack (before/after measured) and processing at ~1.5× recording length on CPU.