fix(discovery): never pin a failed upstream OpenAPI probe

[bussyjd]

Problem

upstreamOpenAPICache.refresh keys on offer.Generation and short-circuits once an entry exists for that generation:

if ok && cached.generation == offer.Generation {
    return
}
doc := fetch(offer)
c.entries[offer.UID] = upstreamOpenAPICacheEntry{generation: offer.Generation, doc: doc}

It records the result unconditionally, including nil. So a single failed probe pins the offer to its route-table fallback until someone edits the CR.

The blast radius is not one offer. reconcileStaticSite rebuilds the shared bundle from this cache on every offer's reconcile, so one cached nil overwrites a good document for the whole stack.

A controller restart empties the cache and gives each offer exactly one attempt, which is where this bites hardest: any offer whose single post-restart probe misses — upstream still rolling out, cold start, transient 5xx, or simply the probe's short timeout — silently degrades from one resource per real paid route down to a single root entry in /.well-known/x402 and openapi.json.

The cache's own doc comment already describes it as holding "the last-good upstream fetch". Treating a nil as good is what broke that contract.

Observed live

After a controller image swap on a stack serving nine paid offers, three dropped from 12 / 12 / 16 advertised resources to 1 while two others were unaffected, then recovered as their reconciles landed. Same controller build throughout — no code change can explain it, which is what pointed at cache state.

This is pre-existing, not a regression: git diff across the rc2→rc3 range for upstream_openapi.go is empty.

Fix

refresh now leaves the generation unrecorded when a probe misses on an offer that could serve a document, so the next reconcile retries and any last-good doc survives. Stale beats silently collapsed.

The terminal/transient distinction matters, and is the reason this isn't a one-line change: fetchUpstreamOpenAPI also returns nil by design for agent and inference offers, and for offers with no upstream Service. Those can never serve a document, so their nil is terminal and must still be cached — otherwise they'd be probed on every single reconcile forever.

offerHasProbeableUpstream is extracted so refresh and fetchUpstreamOpenAPI cannot disagree about which case they're in.

Tests

upstream_openapi_cache_test.go covers all four behaviours:

• a failed probe is not pinned — the next refresh retries, and a later success is cached
• a later failure does not evict the last-good document
• a successful probe is still fetched once per generation, not per reconcile (the original contract)
• a terminal nil (agent / inference / no upstream service) is cached, so those offers aren't probed forever

Verified the regression test genuinely catches the bug — with the guard removed it fails with fetch called 1 times after second refresh, want 2 — the failed probe was pinned, and passes with it restored.

go build ./... and go test ./internal/serviceoffercontroller/... pass.

Note on sequencing

v0.14.0-rc3 is already published and does not contain this fix. Since the failure self-recovers as reconciles land, it wasn't judged a release blocker. This targets the integration branch for the next RC.

[bussyjd]

Superseded — the commit 20d3c85c from this branch is already in integration/v0.14.0-rc3 and shipped in the published v0.14.0-rc3. The merge fast-forwarded, so GitHub never marked this PR as merged even though its content landed.

Verified in the release build on a live nine-offer stack. Note the scope, which is narrower than this PR's title suggests: it fixes the permanent form (a failed probe pinned for the whole generation, i.e. until the CR is edited). It does not remove the 30–90s window after a controller restart, when the cache starts empty and the shared bundle is rebuilt before every offer has reconciled. That is tracked separately.

Closing as landed; rolled up to main in #811.