Skip to content

v1.5.2 — Fishtank Worker Crash Fix

Choose a tag to compare

@ManletPride ManletPride released this 23 Mar 04:46
· 58 commits to main since this release
c06c100

This is a bug fix release targeting a crash loop introduced by Fishtank.live CDN instability. No new features; drop-in replacement for v1.5.1.


Bug Fixes

Worker Crash Loop on Fishtank CDN Timeouts

resolve_best_fishtank_variant() returned a bare string when the master playlist fetch timed out, but its only call site always unpacks two values:

recording_url, stream_info = resolve_best_fishtank_variant(...)

This caused a ValueError: too many values to unpack that crashed and immediately restarted the worker on a ~60-second loop — observed as 55 consecutive crashes over approximately 100 minutes in a single session during a period of CDN instability. The intended fail-open behaviour (fall back to recording from the master URL directly) was never reached. Fixed by returning master_url, "" to match every other return path in the function.

This bug affected both fishtank:director and fishtank:cameraman channels, though cameraman was significantly more exposed due to its already-known connection instability.

Flat Sleep on Worker Crash

The except Exception handler at the bottom of record_worker slept a flat 60 seconds after every crash regardless of how many had occurred consecutively. Combined with the above bug, this produced a tight crash-restart loop with no meaningful backoff.

Worker crashes now use exponential backoff: 60 s → 120 s → 240 s → … → capped at 900 s (15 min). The backoff resets to 60 s after any successful recording remux. A Worker restarting in Xs (backoff) line is emitted at each restart so the behaviour is visible in the console log.

Silent Low-Bitrate Variant Selection

resolve_best_fishtank_variant() logged at INFO level regardless of the selected variant's bitrate. Variants under 500 kbps — which typically indicate a degraded or stub CDN rendition — passed silently and produced recordings too small to survive the remux size check, making degraded sessions invisible in the log until the deleted-file messages appeared downstream.

The function now logs a WARNING when the winning variant is under 500 kbps:

[fishtank] Selected best variant: 227kbps — unusually low, stream may be degraded

Upgrade Notes

Drop-in replacement for v1.5.1. No config changes.