FailoverAI v0.1.0: observable image job failover | 可观测的图片任务故障切换

FailoverAI v0.1.0 | 可观测的图片任务故障切换

English | 简体中文

English

The request timed out. The bill might not have.

Retries sound simple until a paid AI request disappears between "sent" and "confirmed."

Did the Provider reject it? Did it finish and lose the response? If the system sends the request again, will the user get one image or two? Will the job be billed once or twice?

FailoverAI is an open-source reference implementation built around those questions. The first release focuses on image generation and makes the routing decision visible from the first API call to the final saved asset.

Watch the handoff

Inject HTTP 503 into the local primary Provider. The Worker records the failed attempt, calls the local backup once and saves the only confirmed result.

Submit the same request again with the same idempotency key. The API returns the original job, and the Provider call count does not increase.

What this release proves

1. The API stores one job and its idempotency record in PostgreSQL.
2. A Worker claims the job with FOR UPDATE SKIP LOCKED and a renewable lease.
3. A confirmed availability failure such as 429 or 503 can move to the backup once.
4. The first confirmed image becomes the only final result.
5. An expired job can be reclaimed, while the old Worker is blocked from overwriting the new result by a fencing token.

The interesting behavior is what the gateway refuses to do. A user request error does not trigger fallback. An adapter TypeError does not pretend to be a Provider outage. A timeout or interrupted connection leaves the result unconfirmed, so the job stops in manual_review instead of creating another request that may also be billed.

Run the 30-second demo

cp .env.example .env
docker compose up --build

Open http://localhost:3001, select HTTP 503, click Generate image, then click Idempotent replay.

The console shows job events, Provider attempts, HTTP status, latency, queue timing, Worker heartbeat and the final asset. English is the default interface, and Chinese is available from the header.

Current boundary

Both Providers are local fakes, and the generated image is a deterministic local SVG. The project does not call real or paid services. Authentication, multi-tenant isolation, real Provider adapters and PostgreSQL high availability are outside this release.

FailoverAI is a reproducible reference implementation for reliable AI job routing and failure handling. It is not presented as production ready.

简体中文

请求超时了，但账单未必停了。

AI 任务最难处理的故障，不一定是明确返回一个 503。

更麻烦的是没有结果：Provider 到底没收到请求，还是已经接单，只是连接在返回途中断了？这时自动重试，可能多生成一张图，也可能多产生一次费用；直接停止，用户又拿不到结果。

FailoverAI 把这个灰区做成了一个可以亲手验证的开源参考实现。首版只处理图片生成，但从 API 接收任务到最终结果写入，每一步都有记录。

看见故障切换

给本地 primary Provider 注入 HTTP 503。Worker 会记录失败调用，只切换一次 backup，并保存唯一经过确认的结果。

使用相同幂等键再次提交同一请求，API 会返回原任务，Provider 调用次数不会增加。

这个版本证明了什么

1. API 只创建一个任务，并把幂等记录保存在 PostgreSQL。
2. Worker 使用 FOR UPDATE SKIP LOCKED 和可续期租约领取任务。
3. 429、503 等明确的可用性故障最多切换 backup 一次。
4. 系统只保存第一个经过确认的图片结果。
5. 租约过期后新 Worker 可以接手，fencing token 会阻止旧 Worker 覆盖新结果。

更重要的是系统明确拒绝做什么。用户请求错误不会触发切换；适配器 TypeError 不会伪装成 Provider 故障；超时或连接中断会让结果处于未确认状态，任务停在 manual_review，不会自动创建第二次可能计费的请求。

30 秒启动演示

cp .env.example .env
docker compose up --build

打开 http://localhost:3001，选择 HTTP 503，点击“生成图片”，再点击“幂等重提”。

控制台会展示任务事件、Provider 调用、HTTP 状态、延迟、队列等待、Worker 心跳和最终结果。界面默认使用英文，可以在页头切换中文。

当前边界

两个 Provider 都是本地 fake Provider，图片也是本地生成的确定性 SVG。项目不会调用真实或付费服务。认证、多租户隔离、真实 Provider 适配器和 PostgreSQL 高可用方案不在本次发布范围内。

FailoverAI 是一个可复现的可靠 AI 任务路由与故障处理参考实现，不是生产级网关。