Repository navigation
v1.2.3
Release notes – v1.2.3
Backend
Fix: SFTP connection pool dies permanently after idle timeout
- Symptom: After ~15 minutes of inactivity, all
docker pushoperations fail withconnection loston every blob upload. The registry returns HTTP 500 on PUT and keeps retrying indefinitely. The only recovery was a full container restart. - Cause: Hetzner Storage Box drops idle SSH connections after ~15 minutes. When
getClient()detected a stale connection and reconnection failed, it returned the already-closed client back to the pool viaputClient(). SinceputClient()never health-checked, the dead client cycled through the pool permanently — poisoning all 4 slots. - Fix: Complete rewrite of the SFTP connection pool with 5 resilience mechanisms:
1. Keepalive goroutine
A background goroutine pings all pooled connections every 5 minutes using Getwd(). Dead connections are closed and replaced before they can be checked out by a caller. This prevents Hetzner's idle timeout from ever killing connections silently.
2. Safe checkout with retry
getClient() now retries reconnection up to 3 times with exponential backoff (2s, 4s, 6s). If all retries fail, it returns nil instead of a dead client — callers receive a clear SFTP unavailable error rather than a cryptic connection lost deep in the write path.
3. Health-checked return
putClient() now verifies the connection is alive before returning it to the pool. Dead connections are discarded and a background goroutine (fillPool) attempts to replace them, restoring pool capacity without blocking the current request.
4. Partial pool initialization
NewDriverPool() no longer fails if one of the initial connections can't be established. It starts with whatever connections succeed and fills the rest in background. This prevents a transient SFTP blip during startup from taking down the entire registry.
5. Safe Reader/Writer lifecycle
Reader() and Writer() on the pooled driver previously returned the SFTP client to the pool via defer putClient() while the file handle was still in use by the caller — a use-after-return race. New poolReadCloser and poolFileWriter wrappers hold the client until the caller explicitly closes the file.
Summary
| Area | Change | Type |
|---|---|---|
| SFTP Pool | Keepalive goroutine (5 min ping interval) | Fix |
| SFTP Pool | Retry with backoff on stale connection checkout | Fix |
| SFTP Pool | Health-check on client return, auto-replace dead connections | Fix |
| SFTP Pool | Partial init + background pool fill | Fix |
| SFTP Pool | Reader/Writer hold client until file closed | Fix |
| SFTP Pool | Pool.Alive() exposes healthy connection count |
Enhancement |
| SFTP Pool | Pool.Close() for graceful shutdown |
Enhancement |
Upgrade notes
- Pull the latest backend image after CI publishes the tag:
docker pull troke12/refity-backend:v1.2.3(or:latestif you track that). - Restart the backend container. No database migration is required.
- Behavior change: None. This is a purely internal fix — the registry API is unchanged. The pool now self-heals instead of requiring manual container restarts after idle periods.