Add Oban Lifeline plugin to rescue orphaned jobs

[profesorabbott]

Jobs stuck in "executing" state for more than 30 minutes (e.g. after a container restart) will automatically be rescued and retried.

What's new?

N/A

What's changed?

N/A

What's fixed?

N/A

Any other comments?

Copied from PR Add Oban Lifeline plugin to rescue orphaned jobs from the original branch

[salvageop]

@profesorabbott do you have more details on when this becomes a problem? My primary concern is that some jobs are long running - the slow channel refresh can take many hours.. TBH since I haven't hit this issue, I don't know the context..

[profesorabbott]

@salvageop the main time i see it is after a machine power loss container restart. I pulled this PR from the original repo so I can test my environment further to see if it's really necessary or if it reads false negatives. #48 includes a manual option for this behavior so it might not be something that should be automated.

[salvageop]

TL;DR: based on Claude research, this does not appear to be safe as it could easily damage other long-running jobs, which are definitely common in PF. Tools to deal with this situation should come with #48 . I'll go ahead and close this PR. If the stuck job issue persists, I'll circle back to this.

  How Oban actually runs a job

  1. A producer (one per queue) periodically fetches a batch of available jobs up to the queue's concurrency limit.
  2. It atomically transitions each fetched row to executing, bumping attempt and setting attempted_at = now.
  3. It spawns a supervised in-memory Task process that calls your perform/1.
  4. On return it transitions the row to completed (or retryable/discarded on error).

  So an executing row in the DB corresponds to a live process on the node that's actually running it. That coupling is the whole crux of the issue.

  What "orphaned" really means

  A row only becomes genuinely stuck in executing when the node dies mid-job — a container restart, OOM kill, crash. The process vanishes but the DB row stays executing forever because nothing is left to transition it. For a self-hosted
  Docker app that restarts on updates/reboots, this is the common, legitimate cause — and it's exactly what this diagnostics page is trying to solve.

  Crucially, Pinchflat has no automatic rescue for this today:
  - config/runtime.exs:61 — Oban plugins are only Pruner + Cron. No Oban.Plugins.Lifeline.
  - lib/pinchflat/boot/post_job_startup_tasks.ex is empty (init/1 does nothing) — even though CLAUDE.md describes it as the place that "reschedules stalled jobs." The intent exists; the implementation doesn't.

  So orphans really do pile up here. The feature is solving a real gap.

  The double-execution bug

  The problem is the detection heuristic in reset_stuck_jobs/1 (and reset_job/1):

  where: j.state == "executing",
  where: j.attempted_at < ^threshold   # 30 minutes

  "Executing for > 30 minutes" is not the same as "orphaned." Several Pinchflat workers legitimately run far longer than 30 min:
  - MediaCollectionIndexingWorker — a full yt-dlp metadata fetch of a large channel/playlist (thousands of items). On your 10k-item system this is very plausible.
  - MediaDownloadWorker — a long or 4K video download.

  When the operator hits "Reset All Stuck Jobs" on such a row, Repo.update_all flips it executing → available while the original process is still alive and working. A producer then fetches that same row, transitions it back to executing, and
  spawns a second process running perform/1 for the same job concurrently.

  A few mechanics worth being precise about:

  - It's the same row, not a duplicate insert — so the workers' unique: constraints (e.g. media_download_worker.ex:7) do not protect you. Uniqueness is enforced at insertion time, not execution; resetting state bypasses it entirely.
  - State corruption on completion — when the original process finishes, it acks/transitions a row that a second runner now owns (different attempt). The two completions race and can clobber each other's final state.
  - The reset also forces attempt: 1 / attempted_at: nil, ignoring max_attempts — so a job that should be discarded after exhausting retries gets resurrected instead.

  What it costs in this app specifically

  Two concurrent yt-dlp processes for the same source/media item means: duplicate downloads, file-write races on the same output path, doubled bandwidth, and increased YouTube rate-limit/IP-ban risk — which directly defeats the purpose of
  the YT_DLP_WORKER_CONCURRENCY throttle (runtime.exs:45).

  This isn't a niche worry — Oban says so itself

  Oban's own Lifeline plugin does essentially what this PR does (time-based executing → available), and its moduledoc carries this warning verbatim (deps/oban/lib/oban/plugins/lifeline.ex:14):

  ▎ This plugin may transition jobs that are genuinely executing and cause duplicate execution.

  Lifeline defaults to a 60-minute rescue_after for that reason, and points to Oban Pro's liveness-aware DynamicLifeline for accuracy. The PR's 30-min, on-demand, "Reset All" button is a more aggressive version of a mechanism Oban already
  flags as hazardous.