Skip to content

Release 0.50.19

Choose a tag to compare

@github-actions github-actions released this 03 Nov 17:19
· 9 commits to main since this release

Install

Docker

docker pull ghcr.io/jedwards1230/earmark:0.50.19

Helm

helm install earmark oci://ghcr.io/jedwards1230/charts/earmark --version 0.50.19

v0.50.19 Release Notes (Patch)

This patch release hardens the ASR runner with transient-failure retries, resume from the last finished window, and mid-job yielding to GPU arbitration. It also adds a database migration (00018) and read-only status fields for gpu-arbiter's yield request.

Features

ASR runner: transient retry and classified failures

The Python runner now classifies each job failure as transient (DB connection loss, CUDA out of memory, unreachable books share, network I/O errors, ffmpeg/ffprobe timeouts) or permanent.

  • Transient failures are retried in place with exponential backoff and jitter. Each class has its own retry budget.
  • When retries run out, the job returns to pending with its attempt kept and the error recorded. The claim that uses the last attempt fails the job instead, so jobs can't get stuck at the attempt cap.
  • After an I/O release, the runner holds new claims until the books share answers again. Heartbeat, self-update, and the GPU gate keep running during the hold.
  • Permanent failures (for example, a missing file on a reachable share or undecodable audio) mark the job failed, as before.
  • ffmpeg and ffprobe calls now have timeouts, so a hung read no longer stalls a job indefinitely.

ASR runner: resume from the last finished window

Chunked jobs write a local checkpoint after each window. A restart, crash, or retry resumes at the next window instead of 0 s.

  • Checkpoints are stored in RUNNER_CHECKPOINT_DIR (off disables them) and are invalidated when the source file, model, or settings change.
  • Checkpoints are deleted when the job completes or fails permanently, and pruned after 7 days.
  • The transcript is still written once, at the end of the job.

ASR runner: mid-job yield to gpu-arbiter

A new runner_control.arbiter_yield flag lets gpu-arbiter ask the runner to leave the GPU. The runner checks this between windows of a chunked job.

  • An eviction costs at most one window of work.
  • Pausing the runner now also stops a long job at its next window boundary, not after the whole file. The job resumes from its checkpoint when unpaused.
  • Single-pass files, and chunked jobs with checkpointing disabled, still finish before parking.

Status API: arbiter yield visibility

/api/v1/status now includes:

  • pipeline.arbiterYield: whether gpu-arbiter is holding the runner off the GPU
  • pipeline.arbiterYieldAt: the timestamp of the arbiter's last write, omitted until it has written once

These fields are read-only and independent of the batch phase.

Improvements

  • Database migration 00018 adds arbiter_yield and arbiter_yield_at to runner_control. The change is additive and uses a constant default, so it rewrites no rows.
  • Heartbeat reliability: the per-job heartbeat reconnects after a failed connection or stamp, instead of going silent for the rest of the job. A silent heartbeat could let the stale sweep re-pend a running job.
  • Claim-guarded job writes: the heartbeat, _mark_done, and failure writes now check that the runner still holds the claim. A stale write can no longer overwrite a job that another claim has finished or reset. _mark_done also tolerates a replayed write when the job is already done.
  • Fix: the runner's provenance-column check no longer fails when the connection returns rows as dictionaries.

Upgrade notes

  • Apply the Earmark release containing migration 00018 before updating the runner pin (asr_runner_windows_source_ref) in homelab-ansible.
  • The gpu-arbiter gpu_gate change that writes arbiter_yield must ship with or after that pin bump. The gate does not work against a schema without the column.
  • New runner settings are optional. Defaults are listed in runner/README.md and docs/CONTRACT.md.

Commits

  • chore(earmark): bump chart version to 0.50.19 (525e95f)
  • feat(runner): ASR runner hardening — transient retry, resume from last window, mid-job yield (00018) (#216) (66164fe)

Full Changelog: v0.50.18...v0.50.19