Skip to content

feat(selfhost): add a per-analyzer circuit breaker to review-enrichment #2541

Description

@JSONbored

Parent: #1936

Problem

The review-enrichment service's analyzer pipeline is correctly fail-safe with respect to the main review verdict — an enrichment timeout degrades review quality (missing supply-chain/dependency context) but never blocks or fails the review itself. However, within the enrichment service, individual analyzers that depend on a third-party HTTP API have no memory of recent failures: every incoming request re-attempts a currently-unhealthy external dependency from a cold state, even seconds after an identical call just timed out or errored. Live telemetry showed a sustained, multi-times-per-hour pattern of exactly this from one specific third-party-dependent analyzer.

Separately, analyzer cost classes execute strictly sequentially (each class must fully drain before the next starts), so a slow or degraded class can eat into the time budget available to later classes rather than being isolated from them.

Requirements

  • Add a per-analyzer (or per-external-dependency) circuit breaker: after a small number of consecutive failures within a short window, skip that analyzer for a cooldown period rather than re-attempting it on every request.
  • The breaker only needs to be in-process state (the service is a long-lived process already) — no new persistence layer required.
  • Evaluate whether cost-class execution can be parallelized (or budget-isolated) rather than strictly sequential, so a degraded class doesn't starve the time budget of unrelated later classes; implement if it's a low-risk change, otherwise file as a follow-up with the reasoning documented.

Deliverables

  • A circuit-breaker wrapper around the identified third-party-dependent analyzer call(s), generalized enough to apply to any future analyzer with the same shape.
  • Tests covering: a healthy analyzer runs normally, a repeatedly-failing analyzer trips the breaker and is skipped (returning its existing fail-safe degraded result) until cooldown, and the breaker recovers after cooldown.
  • A short note on the cost-class parallelization question (implemented, or filed as a scoped follow-up with rationale).

Acceptance criteria

  • A known-unhealthy analyzer stops generating repeated failed external calls once its breaker trips, without changing the fail-safe behavior callers already rely on.
  • No change to the non-blocking, fail-safe contract of the enrichment service as a whole.

Expected outcome

Review-enrichment stops wasting request budget and wall-clock on a third-party dependency already known to be unhealthy in the current window, improving both enrichment coverage and the main review's wall-clock latency during a dependency outage.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions