Repository navigation
Follow-up to issue #1:
points 5 (workspace containment), 8 (provenance / local-only), 1 (generic
transcript importer) and the provider-neutral Stage F contract. Point 2 (Apple
Silicon / MLX profile) still needs someone with a Mac — see below.
Added
scripts/import_transcript.py— an external transcript (MLX Whisper, a hosted
STT service, creator captions, a human transcript) becomes the pipeline's
transcript.json+transcript.txt(docs/transcript-contract.md), so Stage 0 is
skipped. Reads SubRip, WebVTT (NOTE/STYLE blocks, cue settings,<v>voices kept as
speaker) and whisper-style JSON ({"segments": [...]}or a bare list, with word
timings when every word has them); other JSON shapes via--list-key/
--start-key/--end-key/--text-key/--time-unit ms;--offsetfor a later
chunk. Follows the contract's six rules: the source is only read (importing a file onto
itself is refused), an existing transcript needs--force(--backupkeeps it), times
become float seconds, segments are sorted, and a segment withend < startis an error
with nothing written — never a silent fix. Caption styling is stripped; the words are
not touched. Recordsengine/whisper_modelinmetadata.jsonandruns.jsonllike
the local engine, plusprovenance.json.provenance.jsonin every lecture dir (scripts/provenance.py). Each stage that
writes an artifact records, per artifact: stage, tool, pipeline version, run id,
provider(local/groq/gemini/ …),engine,model,network
(none/loopback/internet), its parameters, the artifact's own sha256 and the
sha256 of every input (media files over 256 MB by size + first/last MiB; a directory
such asslides/by one hash over its files). `python scripts/provenance.py verify ` reports an artifact as `modified` (not what its stage wrote) or `stale` (an input changed after it was built — e.g. grounding that predates a re-transcription); exit 1 on either. Paths outside the lecture dir are stored as `external:`, never absolute. `export_web.py` writes its own into the support folder. Stage F has no script, so `provenance.py record` lets the agent record which model wrote the note. Recording never fails a stage.- Local-only mode:
LECTURE_LOCAL_ONLY=1orconfig.yamlprivacy.local_only: true
makestranscribe_video.py --engine groq,groq_asr.py,gemini_asr.pyand
asr_gemini_primary.pyexit before anything is uploaded (fail closed). No stage ever
fell back from local to cloud; this closes the "mistyped flag" case. docs/stage-f-contract.md— Stage F without any provider in it: inputs, the
output files and their keys (slides_final.json,note_draft.md,L3/files), the
rules no script can check, and which script fails on what afterwards.SKILL.md
is one adapter of it (Claude Code).
Security
- Names from data files can no longer leave the directory you gave. New
_common.contained_path/safe_basename/is_within: a plan slug, Stage F's
attachment_name, an embed target in an L3 note, afilenamein a slides JSON, a
manifest clip that would be moved — none may be absolute, carry a drive,..,:
or a control character, or resolve (symlinks and junctions followed) outside the
approved root. A valid name passes through unchanged (outputs stay
byte-identical); an invalid one stops the stage with exit 2 instead of being rewritten
into some other name. Before this release,finalize_to_vault.pycopied a slide to
attach_dir / attachment_nameas written by the LLM, so../../x.jpglanded outside the
vault's attachment folder;export_web.pyresolved![[../../x.jpg]]relative to the
figure folder and copied whatever it found into the shareable support folder. export_web.pymoves or recycles only clips inside the course root.
Changed
- Every stage that writes an artifact goes through the helper for its output paths and
for the data-derived input names it opens (Stages A, B, B2, C, D, E, the PDF / image
bridges, ASR suspects, transcription and collapse repair, the Gemini route,
render_embeds,finalize_to_vault,build_single_talk_web,export_web,
batch/build_L1). docs/transcript-contract.md: the importer,provenance.json, refreshed file:line
references; an Apple Silicon / MLX section that says plainly it is untested._log.__version__(the build identity outside a git checkout) now follows the release.
Verified
No regression on real data, compared byte for byte against 0.9.0:
export_web.pyon all 84 course work dirs of this machine's archive (cloned:
text copied, binaries hard-linked, manifest root pointed away from the delivered
folders),--no-remux: 84/84 identical — 4,567 files including the page, timeline,
media map, markdown, images,分享說明.txtand the exporter's own stdout/stderr;
32,839 note blocks and 35,077 slide blocks. 83/83 new provenance files verifyok.- Per-lecture stages on 24 real lecture dirs (copies), each stage hashed after it ran: 160 stage runs —
grounding 24, ASR suspects 24,render_embeds16,finalize_to_vault16 (into a scratch vault),
build_single_talk_web23, the PDF bridge 8, quick OCR 9 and dedup 9 (decks up to 300 frames),
Stage D and B2 with--resumewhere no model call was needed (15 and 16). Same exit codes and
same files in all 160 (5,864 files in the final trees; the only difference is the absolute course
pathbuild_single_talk_webwrites into_raw/manifest.json, which is the copy's own location);
console output identical once timings are masked.provenance.py verifyon the results flagged
exactly the artifacts the test run had made stale by re-running dedup after Stage D. transcribe_video.py(breeze25, GPU) andextract_slides.pyon a 90-second real
clip: transcript (51 segments) and frames identical.- The importer on 23 real transcripts (10,206 segments) as JSON, as generated SRT and
as generated WebVTT: 23/23 identical each way;transcript.txtequals the canonical
writer's output. - 72 new hermetic tests (
tests/test_path_containment.py,test_provenance.py,
test_import_transcript.py); the symlink cases skip on Windows without the privilege
and use an NTFS junction instead where possible.
Not done here
- Point 2, Apple Silicon / MLX — no Mac to verify on. The importer is the adapter
interface (--format json --engine mlx-whisper, by mlx-whisper's documented output
format), but it has not been run on real MLX output, andgpu_check.pyhas no
unified-memory / Metal profile. Left for a contributor with the hardware. - Lockfile (point 7): still none, for the reason given in 0.8.0 — CI exercises none
of the heavy dependencies, so a lock frozen on one machine would claim a tested
combination nobody tested.