Skip to content

v1.2.3

Choose a tag to compare

@troke12 troke12 released this 10 Apr 01:00
· 3 commits to master since this release

Release notes – v1.2.3

Backend

Fix: SFTP connection pool dies permanently after idle timeout

  • Symptom: After ~15 minutes of inactivity, all docker push operations fail with connection lost on every blob upload. The registry returns HTTP 500 on PUT and keeps retrying indefinitely. The only recovery was a full container restart.
  • Cause: Hetzner Storage Box drops idle SSH connections after ~15 minutes. When getClient() detected a stale connection and reconnection failed, it returned the already-closed client back to the pool via putClient(). Since putClient() never health-checked, the dead client cycled through the pool permanently — poisoning all 4 slots.
  • Fix: Complete rewrite of the SFTP connection pool with 5 resilience mechanisms:

1. Keepalive goroutine

A background goroutine pings all pooled connections every 5 minutes using Getwd(). Dead connections are closed and replaced before they can be checked out by a caller. This prevents Hetzner's idle timeout from ever killing connections silently.

2. Safe checkout with retry

getClient() now retries reconnection up to 3 times with exponential backoff (2s, 4s, 6s). If all retries fail, it returns nil instead of a dead client — callers receive a clear SFTP unavailable error rather than a cryptic connection lost deep in the write path.

3. Health-checked return

putClient() now verifies the connection is alive before returning it to the pool. Dead connections are discarded and a background goroutine (fillPool) attempts to replace them, restoring pool capacity without blocking the current request.

4. Partial pool initialization

NewDriverPool() no longer fails if one of the initial connections can't be established. It starts with whatever connections succeed and fills the rest in background. This prevents a transient SFTP blip during startup from taking down the entire registry.

5. Safe Reader/Writer lifecycle

Reader() and Writer() on the pooled driver previously returned the SFTP client to the pool via defer putClient() while the file handle was still in use by the caller — a use-after-return race. New poolReadCloser and poolFileWriter wrappers hold the client until the caller explicitly closes the file.

Summary

Area Change Type
SFTP Pool Keepalive goroutine (5 min ping interval) Fix
SFTP Pool Retry with backoff on stale connection checkout Fix
SFTP Pool Health-check on client return, auto-replace dead connections Fix
SFTP Pool Partial init + background pool fill Fix
SFTP Pool Reader/Writer hold client until file closed Fix
SFTP Pool Pool.Alive() exposes healthy connection count Enhancement
SFTP Pool Pool.Close() for graceful shutdown Enhancement

Upgrade notes

  1. Pull the latest backend image after CI publishes the tag: docker pull troke12/refity-backend:v1.2.3 (or :latest if you track that).
  2. Restart the backend container. No database migration is required.
  3. Behavior change: None. This is a purely internal fix — the registry API is unchanged. The pool now self-heals instead of requiring manual container restarts after idle periods.