fix(worker): provider readiness check + circuit breaker for unreachable Ollama

[mrviduus]

First: a correction to the premise

This PR was requested as a fix for a production incident — Sentry issue 7656871760, 30 events, Name or service not known (ollama:11434), tagged environment: Production, "38 books starved".

It was not production. Four independent checks:

Signal	The Sentry events	Production (checked over ssh)
Runtime	linux-arm64	x86_64
Release	ee15354c	never deployed (71c6b1da → df551f15 → bb7cfc1c → 58fec417)
ollama container	not resolvable	Up 4 weeks (healthy), worker resolves it to 172.18.0.8
Worker log, 24 h	38 transport failures	0 occurrences of "ollama"
Backfill candidates (status = Ready AND genre IS NULL)	38	0 — all 50 prod books have a genre

The events came from a developer machine. The "38 starved books" were a local database.

Why the tag lied — and this is the actual bug the episode exposed: the local .env sets ASPNETCORE_ENVIRONMENT=Production, so a dev box pointed at the production DSN wrote into the production Sentry project under a production tag. Reading that UI, "prod" is the only available conclusion.

Changes

1. Stop dev events masquerading as prod. SentryBootstrap.ResolveEnvironmentName downgrades a Production claim to production-unverified when SENTRY_RELEASE is absent — the release comes from the GIT_SHA build arg in both Dockerfiles, so every CI image has one and no dotnet run ever does. Real production unaffected. A rename, not a drop: an unverified event is still worth having, it just must not masquerade. Plus a .env.example warning.

2. Startup readiness check. AiProviderReadinessCheck replaces EnrichmentKeyCheck (strict superset; the old one's hardcoded provider set had already drifted past openai-pdf, the most expensive route in the system — a prefix rule now makes that impossible). Probes every provider the route table actually references, once, before any worker loop starts. Ollama via GET /api/tags on a 2 s timeout — same shape as the API's existing /health/ready probe. OpenAI-family by key presence only: it never spends a token to prove a paid provider is up. One LogError naming the affected tags, one Sentry warning, circuit seeded open. Cannot crash or block the host.

3. Circuit breaker. ProviderCircuitPolicy — pure statics with the clock as a parameter (this repo has no clock abstraction; mirrors AlarmThrottle). OpenedAt survives escalation so JustTripped is true exactly once per outage — that single predicate is the entire "one event per condition, not 30" requirement. Backoff 1 → 5 → 30 min. Detection lives inside OllamaLlmClient because it is the only place that sees the wire: the client swallows transport failures and returns an empty response, so a decorator would see success-with-empty and could not tell a dead Ollama from a model that said nothing. Non-2xx deliberately does not trip — that means Ollama is alive and answering (usually a wrong Ollama:Model → 404), fails in milliseconds, and burying it under 30 minutes of silence would turn a cheap visible error into an invisible one.

4. Batch gates that prevent data loss. UserBookEnrichmentService.EnrichAsync stamps Completed, not Failed, when the generator returns null — exactly what an unreachable provider produces. Claiming rows during an outage therefore drains the queue into a terminal "done, nothing filled" state nothing revisits. The gates leave rows Pending instead. This one is provider-independent and was worth the trip on its own.

No silent fallback in either direction. ModelGateway.cs is not modified at all. Rerouting an outage onto a paid provider is the 2026-07-14 390%-CPU incident with the arrow reversed.

Before / after

	Before	After
Dead provider, per worker start	up to 50 × 90 s ≈ 77 min of dead wall-clock	1 probe request, then abort
Sentry events per episode	30 duplicates	1 warning + automatic recovery
Pending rows during an outage	drained to Completed, nothing filled	left Pending, picked up on recovery
Provider down at startup	discovered later, by absence of data	named in the first log line, with the affected tasks

Verified against the real failure, not a mock

Local Ollama stopped, Worker started with SENTRY_DSN unset:

Metadata backfill: enriching 38 user books
Metadata backfill: aborting after 0 enriched / 1 failed — provider 'ollama'
  is unavailable; the remaining candidates stay queued

Exactly one HTTP request reached the wire; the 38 candidate rows were still queued afterwards.

That per-book re-check exists because the first live run exposed a gap in my own design: the startup probe opens the circuit on a 1-minute rung while the backfill only wakes at its 2-minute start delay, so the circuit is legitimately half-open by then and a single up-front gate let the whole batch begin.

1363 unit tests green (79 new), build + dotnet format --verify-no-changes clean. No migration.

Rollback plan

Ai:ProviderHealth:Enabled=false restores byte-for-byte the previous behaviour without a deploy. Full revert is a clean git revert — no migration, no schema, no data.

Notes

• Sentry issue 7656871760 should be resolved as "not production" rather than fixed-in-release; both new events carry explicit fingerprints keyed on (event kind, provider) so future ones group stably and "resolved in release" sticks.
• Not built, deliberately: no Polly, no new DB column, no background prober (the half-open call is the probe), no breaker for OpenAI (its failures are quota/5xx, not reachability), no readiness check in the API (it already has /health/ready, and it inherits the breaker via AddApplication()).

🤖 Generated with Claude Code