Skip to content

Feat/circuit breaker on gateful#1812

Merged
pikonha merged 14 commits intodevfrom
feat/circuit_breaker_on_gateful
Apr 24, 2026
Merged

Feat/circuit breaker on gateful#1812
pikonha merged 14 commits intodevfrom
feat/circuit_breaker_on_gateful

Conversation

@LeonardoVieira1630
Copy link
Copy Markdown
Member

@LeonardoVieira1630 LeonardoVieira1630 commented Apr 13, 2026

Summary

Add a circuit breaker with exponential backoff to the Gateful API gateway, protecting the system from cascading failures when upstream DAO APIs become unavailable. When a DAO backend exceeds a configurable failure threshold, its circuit opens and requests are fast-failed with a 503 response until a cooldown-based probe succeeds.

Changes

  • New CircuitBreaker class (shared/circuit-breaker.ts): implements the CLOSED -> OPEN -> HALF_OPEN state machine with configurable failure threshold, cooldown, exponential backoff, and max cooldown cap
  • New CircuitBreakerRegistry (shared/circuit-breaker-registry.ts): lazily creates and manages per-DAO circuit breaker instances
  • Config: added CIRCUIT_BREAKER_FAILURE_THRESHOLD, CIRCUIT_BREAKER_COOLDOWN_MS, and CIRCUIT_BREAKER_MAX_COOLDOWN_MS env vars with sensible defaults
  • Health endpoint: now returns per-DAO circuit breaker states (state + next retry time) in the /health response
  • Proxy route: wraps upstream calls in the circuit breaker; 5xx responses from upstreams count as failures
  • Fan-out utility: routes each DAO fetch through its circuit breaker; downgrades CircuitOpenError logs from error to warn
  • Services (DaosService, DelegationService): updated constructors to accept and pass through the registry
  • Global error handler: catches CircuitOpenError and returns a 503 JSON response
  • Tests: added unit tests for the circuit breaker state transitions and updated existing tests to supply the registry

Uploading Screen Recording 2026-04-13 at 15.39.47.mov…

LeonardoVieira1630 and others added 2 commits April 13, 2026 15:38
Introduces a CircuitBreaker class implementing the CLOSED → OPEN → HALF_OPEN
state machine with configurable failure threshold and exponential backoff
cooldown. Includes a CircuitBreakerRegistry for lazy per-DAO breaker creation
and unit tests covering threshold, reset, and probe behavior.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wires the CircuitBreakerRegistry into config (env vars), the proxy
catch-all, the fan-out helper, and both service classes. Adds a global
CircuitOpenError handler returning 503. Exposes per-DAO circuit states
on the /health endpoint. Updates all affected tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 13, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
anticapture Ready Ready Preview, Comment Apr 23, 2026 2:40pm
anticapture-storybook Ready Ready Preview, Comment Apr 23, 2026 2:40pm

Request Review

Search endpoints were missing the cache-control middleware, causing
search results to bypass the gateful Redis cache-aside layer entirely.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@vercel vercel Bot temporarily deployed to Preview – anticapture-storybook April 13, 2026 20:56 Inactive
@github-actions github-actions Bot added the API label Apr 13, 2026
@LeonardoVieira1630 LeonardoVieira1630 marked this pull request as ready for review April 13, 2026 20:57
@claude
Copy link
Copy Markdown

claude Bot commented Apr 13, 2026

Claude encountered an error —— View job


I'll analyze this and get back to you.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f11b8b136a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread apps/gateful/src/index.ts Outdated
@pikonha

This comment was marked as resolved.

4xx client errors should not count as upstream failures. Mirrors the
existing proxy route behavior (res.status >= 500).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolve conflicts:
- apps/api proposals: keep both skip-pagination tag (dev) and setCacheControl (HEAD)
- apps/gateful/src/index.ts: keep CircuitBreakerRegistry import (HEAD) and storeOpenApiSpec (dev)
- apps/gateful/src/shared/fan-out.ts: use dev's structured logger with HEAD's CircuitOpenError vs error severity split
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c126b5539c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread apps/gateful/src/index.ts
Comment thread apps/gateful/src/shared/fan-out.ts
@vercel vercel Bot temporarily deployed to Preview – anticapture-storybook April 22, 2026 21:04 Inactive
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 54f5716018

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread apps/gateful/openapi/gateful.json Outdated
Comment thread apps/gateful/src/proxy/route.ts
@pikonha pikonha merged commit 226668d into dev Apr 24, 2026
9 checks passed
@pikonha pikonha deleted the feat/circuit_breaker_on_gateful branch April 24, 2026 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants