fix(api): keep unreachable content hosts out of the rendezvous rotation

[dylanjeffers]

What's broken

updateNodes builds the rendezvous rotation straight from the on-chain registry. Registration says a node is entitled to serve content — it doesn't say the box is up. The only way to exclude a dead node today is for a human to notice and ship a config.DeadNodes entry.

audius.zeogrid.com has been sitting in that gap: refusing connections, still registered as validator Id 99, still in the rotation.

$ curl https://audius.zeogrid.com/health_check
curl: (7) Failed to connect to audius.zeogrid.com port 443 after 35 ms

Measured against prod on 2026-08-11:

• returned as the primary host on 9 of 25 sampled requests (36%) for gotamaX's cover photo
• ~8% of sampled trending artists have it in their image host set

Why this looked like a data bug

The blobs are fine. Every affected image serves 200 from its mirrors — gotamaX changed their cover photo three times and all three writes landed correctly with distinct CIDs. Only the URL was bad, so the browser fell back to its cached copy and the user reported their new profile picture or album art "reverting" on refresh.

This is one of two causes behind the current batch of "my profile photo keeps reverting" support reports. The other is a genuine write-path bug, fixed separately in OpenAudio/go-openaudio#510.

The fix

The existing 1-minute nodesPoller now probes each registered node's /health_check and drops the ones that don't answer.

Guards against the obvious ways active health checking could make things worse:

Guard	Why
3 consecutive failures before ejection (~3 min)	A restart or blip shouldn't pull a node out from under live traffic
One success re-admits and resets the count	Recovery shouldn't need a deploy
Fail open if >50% would be ejected	Most probes failing at once means our network is the problem; emptying the rotation would break every asset URL we serve

This also removes the need to hand-maintain config.DeadNodes.

Tests

Six cases in content_host_health_test.go covering each guard: healthy nodes retained, ejection only after the threshold, re-admission on recovery with the counter reset, fail-open, empty registry, and deregistered endpoints being forgotten.

Full ./api/ suite passes (0 failures). TestSearch fails in my sandbox for want of Elasticsearch — verified identical on clean main, so unrelated.

🤖 Generated with Claude Code

[raymondjacobson]

Don't love this change feels over engineered. We already have the store all hosts - why is it needed?

[dylanjeffers]

You're right, and the premise I opened this on was wrong. Closing.

I checked the client path I should have checked before filing. useProfilePicture and useCoverPhoto both pass preloadImageFn: preload into useImageSize, and fetchWithFallback walks artwork.mirrors swapping the hostname until one loads — with preload rejecting on both onerror and a 5s timeout. There's a second recovery path via onError + getNextMirrorUrl for <img> failures too.

And your point about store-all is the load-bearing one: Select() unconditionally appends StoreAllNodes, so every asset is guaranteed a mirror that holds everything. A dead primary costs a failed request and a retry, not a broken image.

The one way this could still have bitten is that fetchWithFallback calls preloadImageFn?.(url) with optional chaining, so a caller that omits it silently gets no fallback — but both profile hooks pass it, so that's theoretical.

What I actually verified stands: zeogrid is down, still registered as validator Id 99, and came back as primary on 9 of 25 sampled requests. I just drew the wrong user-facing conclusion from it. #1017 is the one-liner adding it to config.DeadNodes instead.

Worth noting for the original support thread: this means the dead node never explained gotamaX. Their cover photo writes all landed and would have rendered via mirrors, and profile_picture_sizes was never submitted in any transaction — so they're the same client-side upload failure as the other reporter, still undiagnosed. OpenAudio/go-openaudio#510 is unaffected; different code path, and it stands on the 1,241 stuck rows and a test that reproduces the prod error.