0.6.2 - Reproducible transcription

Makes transcription reproducible. No change to the decode, the selection rules or the retention guard.

Seeding

• The rescue pass decodes at a temperature where the model samples rather than beam-searches, and nothing seeded it. The same recording could publish a different transcript on each run. Measured previously: selection differed on two of eleven recordings between two runs of the same build, and a 67 word swing between two draws of one rescue changed what got published.
• The decoder is now seeded from a stable hash of the job identifier, not from a constant. Draws stay as varied across the archive as before, while each individual recording becomes reproducible. Set RESCUE_SEED_MODE to off to restore the previous behaviour without a rebuild.
• The seed is logged on every pass and on the completion line, so any published transcript can be reproduced exactly.
• Verified on the GPU host: the same recording submitted twice under the same identifier, with the queue discarded between runs, produced byte-identical transcripts and byte-identical timings, on a recording where the rescue fires and is selected.

Retained transcripts

• RESCUE_TRANSCRIPT_DIR keeps both the published and the discarded transcript whenever a rescue runs, along with both sets of counts. Off by default. This is what made the last analysis impossible to finish from stored data.

Why this matters more than it looks

• While the input to a quality rule is an unrecorded random draw, no quality rule can be validated. The seven recoveries measured in 0.6.1 came from draws that will never be drawn again, so they are not a valid baseline for anything measured on this build. A fresh corpus run re-establishes it.

Deliberately not included

• The selection ordering. Analysis of nine refused rescues found that six of them discarded scripture readings the service had already transcribed, and that ranking corroborated content ahead of raw word count would publish five more of them while losing none. That change alters the selection rule and has not been measured on the corpus, which is exactly what this release makes possible.

The public API is unchanged.

0.6.1 - Gap-triggered rescue

Adds one rescue trigger, measured across the same 100 recordings used to validate 0.6.0.

The change

• A recording whose largest contiguous uncovered stretch reaches 8 seconds now fires one rescue pass, alongside the existing low word-rate trigger. It cannot requeue or quarantine, and the 20 second threshold that feeds the quarantine path is untouched.
• The same measurement was already present but priced as a gate, where a false positive costs a whole transcript, so it sat at 20 seconds and caught 3 of 7 confirmed content losses. Priced as a trigger, where a false positive costs one extra decode, 8 seconds catches all 7. The detector was never missing. Nobody had re-priced it for a cheaper consequence.
• The longest run of words the unpublished pass holds and the published one lacks is now logged as a review marker. It needs no extra decoding, since both transcripts are already in memory wherever the rescue ran.

Measured on 102 recordings

• The rescue fires on 20, is selected on 11, and 7 of those measurably recover real content against the archive. Two are complete scripture readings restored to sermons that were missing them: Matthew 12:15-16 and Matthew 6:19-21, both going from partial to essentially complete.
• Cost is 13.8 percent more decode time, not the 24 percent predicted. The prediction priced a rescue as a second full decode; it is closer to two thirds of one, because it samples rather than beam-searches.
• Every one of the 11 selected rescues is strictly additive. The discarded pass held nothing the published one lacks, on all 11.
• No recording lost more than 5 percent against 0.6.0. No selected rescue lost content. The retention guard was exercised and refused a rescue that would have lost 1.25 percent. Alignment applied on every affected recording, with and without the rescue.

Known and recorded

• One recording fires the trigger and still publishes about 15 percent short. The trigger now sees it; the remedy does not fix it.
• On one recording the rescue recovers a 156 word reading from Exodus 24 but is discarded on a tie, because the selection rule cannot see that one pass holds a large contiguous passage the other lacks. Under investigation rather than changed blind, since that rescue is also several hundred words shorter overall and it is not yet established whether those words are content or repetition.
• The rescue samples without a seed, so recovery is an expectation across the corpus rather than a promise for any one sermon. Selection differed on 2 of 11 recordings between two runs of the same build.

The public API is unchanged.

0.6.0 - Single-pass decode, rescue pass, per-utterance alignment

The first release to change transcription output. Every change below is backed by measurement across 100 recordings from the archive plus a 6-file reference set; the method, the numbers and the rejected ideas are in docs/QUALITY_PROPOSAL.md and docs/PUNCHLIST.md.

Decode

• One beam-5 pass with the full temperature fallback ladder replaces the five-pass scheme. Median word count across 99 recordings moved +0.18 percent, with two regressions above 5 percent, and the reference recording now completes in 45 seconds against 138 before.
• The old winner was chosen by duration-weighted word probability, which is computed by teacher-forcing the model over text it already produced. It rewards loops, cannot see omissions, and on one recording chose a pass 458 words short. It is gone.
• Three of the five old passes were sampling rather than beam search, because faster-whisper ignores beam size at any temperature above zero. Their beam settings never did anything.

Recovering dropped passages

• A rescue pass fires when any minute of speech falls below 1.5 words per second. It re-decodes once with previous-text conditioning off and the ladder starting above the rung that already failed, and is selected only if it retains the primary's words. Measured firing rate is about 12 percent of recordings, selected on 8 percent.
• This targets a specific failure: passages read aloud, scripture and quotations, were being dropped confidently, so no decode threshold tripped and no fallback rung was ever attempted. Two confirmed cases now recover, including Matthew 6:19-21 and Matthew 12:15-16.

Alignment

• Per-utterance alignment replaces whole-file alignment. It applied on 100 percent of recordings at the first attempt. The old path failed both attempts on a 23 minute recording and wasted five minutes doing it.
• Audio is converted to 16 kHz mono, words are assigned by sequence matching with monotonic enforcement, and Whisper word timestamps are the fallback where the aligner does not reach.

Quality signals

• A low word-rate window now fires the rescue rather than only counting toward a penalty. No quality signal can quarantine a job any more; quarantine is reserved for output that is unusable on its face. A recording that produced 8292 plausible words was previously black-holed by a gate that retried a deterministic decode three times.
• Boundary de-duplication is disabled. Across 64 trims it produced roughly 38 clear false positives, including four consecutive dropped segments on the acclamation in 1 Kings 18:39, and no observed true positives. The temporal implementation remains behind a switch.
• Anomaly counts, flagged segments, speech seconds and rescue outcomes are published as additive fields on both endpoints.

Known and recorded

• One confirmed omission remains, a passage from Exodus 24, where the rescue recovers the reading but loses more elsewhere so the retention guards correctly refuse it.
• One recording publishes about 15 percent short of its previous transcript with no signal detecting it.
• The profile selector is keyed on level, which does not predict fragmentation. Sample rate and bit rate do.

The public API is unchanged. New fields are additive.

0.5.4 - Residual CVE pass

Residual CVE pass. No application code, dependency pin, or CUDA change. Local Trivy scan of the candidate image: zero CRITICAL or HIGH fixable findings on every target.

• libnghttp2-14 pulled to the focal-security build in the existing apt layer.
• The mambaforge bootstrap environment under /opt/conda, which the service never runs, had 2023-era certifi, cryptography, pyOpenSSL, setuptools and urllib3. They are upgraded in place with that environment's own pip, followed by a pip check.
• The remaining msgpack and setuptools findings were pip's own vendored copies, declared in pip's bundled software bill of materials, not installed packages. pip is removed from the finished image; nothing at runtime uses it, and the hotfix overlay (Dockerfile.release) does not need it either.

Validated on the GPU host against the reference file before release; per-pass numbers are in docs/SESSION_KNOWLEDGE.md.

0.5.3 - CI model download rate limit fix

CI fix for the image build. No application code, dependency pin, or CUDA change.

• The Dockerfile's mfa model download step resolves models through the GitHub releases API. Unauthenticated calls from the shared GitHub Actions runner pool exceed the 60 per hour limit, which is why the 0.5.1 and 0.5.2 release builds failed at that layer while builds on the GPU host succeed.
• The workflow now passes its own GITHUB_TOKEN to the build as a BuildKit secret mount, and the Dockerfile adds --github_token only when that secret is present. Local builds run unauthenticated exactly as before. Secret mounts are not image layers, so the token never lands in the image or its history.
• runDocker.sh exports DOCKER_BUILDKIT=1 defensively.

Validated on the GPU host against the reference file before release.

0.5.2 - CUDA runtime slimming and launch-blocking removal

CUDA runtime slimming and a throughput fix. No application code or dependency pin changed.

• cuda-toolkit-12-2 replaced by cuda-libraries-12-2. The toolkit brought the compiler, Nsight Systems, dev headers and documentation, none used at runtime. ctranslate2 loads libcudart, libcublas and libcudnn dynamically and torch carries its own CUDA libraries from pip, so cuda-libraries-12-2 plus libcudnn9-cuda-12 is the complete runtime set. cuda-runtime-12-2 was deliberately not used: it depends on cuda-drivers.
• CUDA_LAUNCH_BLOCKING=1 removed. It forced every kernel launch to be synchronous on all five Whisper passes.
• Expected side effect: the Trivy gate loses the 64 findings that lived in the Nsight Systems Go binary.

Validated on the GPU host against the reference file; per-pass timings versus the 0.4.0 baseline are in docs/SESSION_KNOWLEDGE.md.

0.5.1 - Base image v3.4.2

Base image bump only.

• mmcauliffe/montreal-forced-aligner moved from v3.4.1 to v3.4.2 (Dependabot PR #8). Upstream 3.4.2 is bug fixes: the YAML config loader and tokenizer cleanup. The base's Ubuntu, Python, and torch layers are unchanged.
• runDocker.sh names images after lightweight release tags (git describe --tags).

No application code or dependency pin changed. Validated on the GPU host against the reference file before release; numbers are in docs/SESSION_KNOWLEDGE.md.

0.5.0 - Service hardening

Service hardening release. No change to the decode parameters validated in 0.4.0; the reference file transcribes at the same 2.75 words/sec. Public API contract unchanged; new fields and the /health endpoint are additive.

Queue and worker

• Cleanup now actually runs: once an hour on every worker wake, keyed on completion time, deleting files and rows for the same jobs, plus a sweep of orphaned files older than two days that no live job owns.
• MFA's per-job working corpus under /mfa is removed after alignment (the aligner runs with --clean), closing a container-layer disk leak.
• Jobs stuck in 'processing' are recovered on every wake, not only at startup.
• Transient failures (CUDA OOM, I/O) are retried through the attempt counter with a poll interval between attempts instead of erroring the job on the first exception.
• The worker moves straight to the next queued job instead of sleeping 30 s between jobs.
• The dead second garbage check is gone; empty alignment output falls back to Whisper timings without counting an attempt.

Upload and status

• Duplicate-GUID uploads are serialized and return 409 (case-insensitive) instead of racing to a 500.
• Undecodable uploads return 400 and leave no file behind; the row is claimed before the file is written so the worker cannot see a half-uploaded job.
• Extension allowlist (.mp3 .wav .m4a .flac .ogg .aac .mp4) and a 1 GiB body cap.
• Duration is probed with ffprobe instead of decoding the whole file to PCM in the request thread.
• Upload and status use the same ETA math, counting the in-flight job, and status never reports a completion time in the past.
• GUIDs must be canonical hyphenated form.
• Additive fields: processing_seconds, words_per_second, attempt_count, mfa_applied on /transcriptions rows and completed /status bodies.

Operability

• GET /health with worker liveness, last wake, queue depth and model state; 503 when the worker is dead or stale. Docker HEALTHCHECK added.
• runDocker.sh builds before stopping anything, keeps a transcription-api:rollback tag, and waits for /health.
• Quieter logs: idle transition logged once, wake/sleep at DEBUG, MFA output only on failure, GUID on every job line.

Repo and CI

• Test job (pytest and ruff) runs on push and pull request, and the image publish job depends on it.
• 127 tests, app.py at 93 percent coverage (was 13 tests, 42 percent).
• Stale SQLite file removed from git and from the image; host tooling excluded from the image; requirements-dev.txt added.
• free-disk-space action pinned to a commit.
• docs/PUNCHLIST.md tracks the full service and pipeline review with per-item status.

0.4.0 - Decode collapse fix, pinned dependencies, diarization removed

Fixes the transcription collapse that produced 0.15 to 1.24 words/sec on some recordings. Validated on the GPU host against the 755 s reference file at 2.75 words/sec with per-pass confidence 0.97 to 0.98.

Decode

• Removed the audio preprocessing stage. Its noise gate measured the speech itself, so every file went through an unprofiled denoise and a second lossy encode.
• Temperature is now passed as a fallback ladder, so faster-whisper can recover from a collapsed decode instead of being locked to one attempt.
• Removed the single-speaker initial prompt.
• Pass 5's condition_on_previous_text setting is now honored.

Quality gate

• Added a words-per-second floor (MIN_WORDS_PER_SEC, default 1.0) alongside the alphanumeric check, routed through the same retry and quarantine path.

Build

• Direct dependencies pinned to the package set captured from the working container (torch 2.12.1, ctranslate2 4.8.0, faster-whisper 1.2.1).
• Speaker diarization removed. It had been failing silently in production and the multi-speaker path never ran. The build no longer needs a Hugging Face token.
• LD_LIBRARY_PATH puts the conda lib directory first; the v3.4.1 base's numpy needs a newer libstdc++ than Ubuntu 20.04 provides.
• Flask debug mode off by default.
• Dockerfile.release added as the documented source-only hotfix path.

Docs

• docs/SESSION_KNOWLEDGE.md with the diagnosis, the validation recipe, and the validation record.

0.3.1 - Queue hardening + reconciled timings/diarization production code

Supersedes 0.3.0. Reconciles the queue-hardening (quarantine + crash recovery) with the production-live features that had been running uncommitted on devmachine:

• pyannote speaker diarization + skip MFA forced-alignment for multi-speaker audio
• local Whisper/pyannote model cache baked at build (no runtime HF token)
• audio preprocessing (_processed.mp3) for transcription quality + timings
• queue: quarantine after MAX_GARBAGE_RETRIES garbage results; reset orphaned 'processing' jobs on startup

runDocker.sh now reads HUGGINGFACE_TOKEN from the environment (no hardcoded secret). 13 tests pass.