Repository navigation
Release 0.50.19
Install
Docker
docker pull ghcr.io/jedwards1230/earmark:0.50.19Helm
helm install earmark oci://ghcr.io/jedwards1230/charts/earmark --version 0.50.19v0.50.19 Release Notes (Patch)
This patch release hardens the ASR runner with transient-failure retries, resume from the last finished window, and mid-job yielding to GPU arbitration. It also adds a database migration (00018) and read-only status fields for gpu-arbiter's yield request.
Features
ASR runner: transient retry and classified failures
The Python runner now classifies each job failure as transient (DB connection loss, CUDA out of memory, unreachable books share, network I/O errors, ffmpeg/ffprobe timeouts) or permanent.
- Transient failures are retried in place with exponential backoff and jitter. Each class has its own retry budget.
- When retries run out, the job returns to
pendingwith its attempt kept and the error recorded. The claim that uses the last attempt fails the job instead, so jobs can't get stuck at the attempt cap. - After an I/O release, the runner holds new claims until the books share answers again. Heartbeat, self-update, and the GPU gate keep running during the hold.
- Permanent failures (for example, a missing file on a reachable share or undecodable audio) mark the job failed, as before.
- ffmpeg and ffprobe calls now have timeouts, so a hung read no longer stalls a job indefinitely.
ASR runner: resume from the last finished window
Chunked jobs write a local checkpoint after each window. A restart, crash, or retry resumes at the next window instead of 0 s.
- Checkpoints are stored in
RUNNER_CHECKPOINT_DIR(offdisables them) and are invalidated when the source file, model, or settings change. - Checkpoints are deleted when the job completes or fails permanently, and pruned after 7 days.
- The transcript is still written once, at the end of the job.
ASR runner: mid-job yield to gpu-arbiter
A new runner_control.arbiter_yield flag lets gpu-arbiter ask the runner to leave the GPU. The runner checks this between windows of a chunked job.
- An eviction costs at most one window of work.
- Pausing the runner now also stops a long job at its next window boundary, not after the whole file. The job resumes from its checkpoint when unpaused.
- Single-pass files, and chunked jobs with checkpointing disabled, still finish before parking.
Status API: arbiter yield visibility
/api/v1/status now includes:
pipeline.arbiterYield: whether gpu-arbiter is holding the runner off the GPUpipeline.arbiterYieldAt: the timestamp of the arbiter's last write, omitted until it has written once
These fields are read-only and independent of the batch phase.
Improvements
- Database migration 00018 adds
arbiter_yieldandarbiter_yield_attorunner_control. The change is additive and uses a constant default, so it rewrites no rows. - Heartbeat reliability: the per-job heartbeat reconnects after a failed connection or stamp, instead of going silent for the rest of the job. A silent heartbeat could let the stale sweep re-pend a running job.
- Claim-guarded job writes: the heartbeat,
_mark_done, and failure writes now check that the runner still holds the claim. A stale write can no longer overwrite a job that another claim has finished or reset._mark_donealso tolerates a replayed write when the job is alreadydone. - Fix: the runner's provenance-column check no longer fails when the connection returns rows as dictionaries.
Upgrade notes
- Apply the Earmark release containing migration 00018 before updating the runner pin (
asr_runner_windows_source_ref) in homelab-ansible. - The gpu-arbiter
gpu_gatechange that writesarbiter_yieldmust ship with or after that pin bump. The gate does not work against a schema without the column. - New runner settings are optional. Defaults are listed in
runner/README.mdanddocs/CONTRACT.md.
Commits
- chore(earmark): bump chart version to 0.50.19 (525e95f)
- feat(runner): ASR runner hardening — transient retry, resume from last window, mid-job yield (00018) (#216) (66164fe)
Full Changelog: v0.50.18...v0.50.19