v0.3.18 — 网关侧真修：reset 时刻加 45 天上限，哨兵日期不再钉死熔断器

v0.3.18

网关侧真修：reset 时刻加 45 天上限

2026-11-11 的那起「opencode_go_1 离池 73 年」事故，此前只有投影层拦截 + 看门狗兜底
（knowledge/docs/maxllm-ha.md §10），网关二进制本身仍会把上游胡写的重置时刻当真。
0.3.18 把真修落进网关。

事故机理：一条 streamed 200 的失败事件里同时带着配额措辞和哨兵日期
2099-01-01；find_absolute_times 把它读成绝对时刻
（reset_at_ms = 4070908800000），熔断器于是 force-open 到那一刻 ——
而厂商自己的用量接口报的是 rolling 0%，根本没有限额。于是该成员从
auto 池里静默消失，池子悄悄缩了一圈。

为什么 45 天是上限：网关路由的每一种配额窗口都是「小时到月」量级
（5 小时 / 周度 / 月度）。比 45 天还远的重置时刻不可能是重置时刻 ——
它要么是哨兵值，要么是坏头，要么是烂掉的旧期限（SQLite 里存的，或从
还没带上限的同伴那里传染来的）。

上限加在信任发生的地方：

层	0.3.17 及以前	0.3.18
头部解析	x-ratelimit-reset: 2100 直接采信	超出上限 ⇒ 丢弃该头，回落去读 body
body 候选	全部原样交给路由	超限候选在变成期限前被丢掉（识别层仍照旧返回全部候选，时间戳形状的测试因此不必跟着日历腐烂）
laya 的判断	采信它选的时刻	同样受上限约束；远期选择降级成「有配额、重置时刻未知」
流内失败判定（200 里的配额墙）	直接采信	同一上限
熔断器（最后一道防线）	record_quota_until 原样 park；extend_quota_until 照单全收同伴的期限；restore 把哨兵行原样恢复	park 改走冷却阶梯（分钟 → 小时）；拒绝采纳同伴的越限期限；restore 丢弃越限的持久化行 ⇒ 一次重启不再把成员钉死

故意不设上限的地方：运维/配置级挂起（[quota_parking] 与 admin 的
force-open）保持无界 —— 人手写 until = 2099-01-01 就是那个意思，
那是另一个信任域。

回归测试：事故样本（一条 200 流，其失败里写着 2099-01-01）现在是测试 ——
成员必须落在冷却阶梯上，且 /admin/breakers 里活着的期限不得是那个哨兵。
另加：头部 x-ratelimit-reset: 4102444800（2100）被拒、运行时算出来的
+3 天 body 时刻照常采信、breaker 的 record/extend/restore 三处守卫、
以及一条 wiremock 上游回 4070908800000 ⇒ QuotaUnknown。

部署状态（2026-11-12）

节点	版本	sha256[:16]
oleNas01	0.3.18	425ff0759ef85c29
oleFedora01	0.3.18	425ff0759ef85c29
vm_tecent_cloud_01	0.3.17	3de86d58bbb7011a（未升级：SSH 凭据本会话不可用，见下）
oleMacBookAir	0.3.17	37b6586fcbc87153（arm64 原生构建；未升级：Mac 不可达）

腾讯云节点待 SSH 恢复后补一次（推的是同一份产物，sha 保持 425ff075…；
产物已经在 NAS 的 ~/opt/maxllm/bin/maxllm-server 上等着，deploy.sh 从 NAS 取）：

# 两种写法等价：默认 REMOTE 就是 root@10.126.126.199（mesh IP）；
# 用别名需要那台机器的 ~/.ssh/config 里有 vm_tecent_cloud_01 条目。
REMOTE=root@10.126.126.199 SOURCE=oliveagle@10.126.126.6 \
  ~/.agents/ops/machines/vm_tecent_cloud_01/maxllm/deploy.sh

（已用 DRY_RUN=1 验证过这条命令：取二进制、取配置、路径改写、cmd_cli 守卫、
dotvault 渲染 env 全部通过，只差最后那一步 SSH。）

Mac 本机待可达后走 native release 构建 → install 覆盖 →
launchctl kickstart -k gui/$(id -u)/com.oliveagle.maxllm。

附：本仓源码与 CI

本版本源码在 ~/repos/github.com/oliveagle/maxllm-src（origin
git@github.com:oliveagle/maxllm.git），两个提交分开：

• 4b02bd4 — fix(gateway): a reset instant no quota window can express is not a reset
• edc728c — chore(release): 0.3.17 -> 0.3.18

CI（check / fmt / clippy -D warnings / policy / scripts / test / build）
在 main 与 v0.3.18 两个 ref 上全绿。

v0.3.10 — drop the Command Code CLI bridge (vendor TOS)

v0.3.10

撤下 Command Code CLI 桥接

厂商的 /alpha/* 面是他们自家 cmd 客户端的私有协议，直连会回
Proxy use detected. This endpoint only serves CLI. Use Command Code provider API instead.，
并写明「Continued proxying of your subscription violates the TOS and will result in account ban.」
把包月订阅经自建网关转出去属于厂商明令禁止的用法，因此本版本把桥接整体撤掉：

• 删除 crate crates/maxllm-commandcode/（独立 sidecar，工作区内无其它 crate 依赖它）；
• maxllm.toml 移除 [providers.cmd_cli] 与两条路由 /v1/responses/cli_cmd、/v1/channels/cli_cmd；
• 线上（oleNas01）停用 maxllm-commandcode.service，二进制移出 bin/；
• 保留 kind = "commandcode"（厂商授权的 Provider API 路径）：Go 计划下它恒 403
  upgrade_required，升级 Provider 计划后 /v1/responses/cmd 原生透传即可用，网关侧无需改动。

附带：逆向 cmd 的识别机制（写入 docs/ops_review_20261108.md 附录 C）

cmd 并没有一个「魔法 token」证明自己是 CLI，识别是分层的：

1. 路径面 — /alpha/* = CLI 私有协议，/provider/v1/* = 公开 API；
2. 自称头 — User-Agent: cli（常量，每个请求都带）、x-cli-surface
   （仅 acp/rpc 面才带，普通 CLI 不带）、x-cli-environment: production、x-command-code-version；
3. 设备指纹 — 首次运行把 thumbmark = sha256(salt + machineId|MACs) 上报
   /alpha/fingerprint/record，components 里 machineId / MAC / 用户名 / 主机名 / git 邮箱全部 hash，
   并标 runtime: "cli"；
4. 凭证 — Authorization: Bearer <apiKey>，来自 ~/.commandcode/auth.json
   （或 COMMAND_CODE_API_KEY 环境变量），登录走 https://commandcode.ai/studio/auth/cli；
5. 服务端判定 — 路径族决定「这是 CLI 面」，套餐决定「能不能用 API」，/alpha/* 上另有反代理护栏。

实测补充：User-Agent: cli 并不是所有 /alpha/* 端点的准入条件（/alpha/namespaces、
/alpha/billing/credits、/alpha/sandbox/sessions 用普通 curl UA 同样 200）——它只是客户端自称；
真正的闸门在服务端。另外 POST /alpha/generate 的 403 响应体会把 CLI 上传的内容寻址文件池索引
（syspool/<user>/main/<sha256>）原样回显，属上游的一个信息泄漏。

验证

• cargo fmt --all -- --check、cargo clippy --workspace --all-targets -- -D warnings、
  cargo test --workspace 全绿；
• python3 scripts/check_responses_policy.py --config maxllm.toml → OK（每个能上 Responses 的上游仍是 native）。

v0.3.9 — Command Code CLI bridge (native Responses), cache-hit billing, client-abort accounting

v0.3.9

新增

• Command Code CLI 桥接（新 crate crates/maxllm-commandcode）：把厂商自有的
  cmd -p <query> --output-format json 客户端包成 OpenAI 服务，原生实现 Responses 契约
  （POST /v1/responses，流式 SSE 事件按 response.created → in_progress →
  output_item.added → content_part.added → output_text.delta → output_text.done →
  content_part.done → output_item.done → response.completed → data: [DONE] 顺序，
  每个事件带单调 sequence_number），并同时提供 Chat Completions。
  reasoning.effort 映射到 CLI 的 --effort。19 个单测，clippy -D warnings 通过。
• commandcode.ai provider（native Responses 优先）与 open-design.ai provider
  （原生 Rust 实现），均已进 /v1/models 与独立路由。
• prompt-cache 计费：/v1/responses 读取上游返回的缓存命中数，统一两套 usage 解析器；
  admin 按模型的 cache rate 给缓存命中计费（线上实测 MiniMax 命中率 98.5%，批量成本降到 1/4）。
• 可观测性：stream 截断、用量与成本可见且可自纠正；client_aborted_total 与
  provider 故障分开统计；/health 不再把 half-open 熔断器算作 open；看板按 provider
  分别报 client abort 与截断。

修复

• 未知模型自动改写：上游 404（vLLM 风格）或 failover 时，把 body 里的 model 改写成
  该 provider 的 default_model 后重试一次，避免整条 fallback 链集体 400。
• 客户端中断不再记成 provider 故障：被下游中断的流按 client abort 记账，
  不再污染熔断器与截断率；fast-path 的中断抓取也不再产生幽灵 request_logs 行。
• 熔断器不再每 60 秒抖动（stream demotion hold 升级）；上游已消失的 provider 其
  resilience 状态会被清理；native Responses 路径的 tool_choice: "any" 修复。
• 落盘的录制 body 在写盘前脱敏凭据与 PII；/health 半开探测不再误开熔断器。

运维 / 工具

• scripts/：新增 openobserve_alerts.py、stream_outcomes.py（按“谁关的流”区分 200
  的归属）、redact_recordings.py、rotate_logs.py；pool_health.py 改为探测每个具名
  auto 池；check_responses_policy.py 校验「凡是能上 Responses 的上游都必须 native」。
• docs/ops_review_20261108.md：生产网关运维评审（截断、用量核算、可观测性缺口）。

验证

• CI（workflow_dispatch，ref v0.3.9）：Check / Test / Clippy / Format /
  Responses policy / Build Release 全部通过。
• 本地门禁：cargo fmt --all -- --check、cargo clippy --workspace --all-targets -- -D warnings、
  cargo test --workspace 全绿。

注：本仓库的 push 事件不触发 Actions，CI 需手动 gh workflow run CI --ref v0.3.9；
因此 v0.3.9 未发布 Docker 镜像（docker job 需要 push 事件）。

v0.3.8 — named auto pools (auto = cloud, auto_home = home)

Named auto pools

[auto_models.<key>] declares extra rotation pools next to the default [auto_model], each answering to its own sentinel model name. A deployment can now keep the rotation sets disjoint — auto over cloud vendors, auto_home over the home GPUs — instead of one pool that mixes machines which are off half the day with cloud APIs.

• [auto_model] stays the default pool (sentinel auto), so an existing single-pool config is unchanged. A named pool's sentinel is its table key unless the block sets name.
• The route's provider names the pool an auto_model = true route rotates over when it matches one; otherwise the first declared pool serves it.
• RequestCtx.auto_pool pins the pool for the whole request: the sentinel that entered the request also bounds every later failover hop, so a request can no longer drift from one rotation set into another (previously one auto pool was searched for any hop's provider name).
• GET /v1/models lists every enabled pool sentinel (owned_by: maxllm-auto).
• Pool names must be non-empty and unique; an auto route naming a missing or disabled pool fails validation at startup.

Verification

• cargo test --workspace: all green (211 translate tests etc.).
• cargo test -p maxllm-gateway --test integration: 107 passed, 4 ignored (2 new: pools keep their sets apart; a body-only auto_home sentinel is adopted into its named pool).
• Binary: musl static, sha256 f80d23577e150935cf3a788149ac5afbd508a220d68ee5919157b6b5a37e3be5.

v0.3.7 — per-provider utc_offset for zone-less reset times

What changed

429 error messages often state when the quota resets, and many upstreams
write that instant in their own local time with no zone marker. Zhipu's
production shape:

{"code":"1310","message":"您已达到每周/每月使用上限，您的限额将在 2026-11-10 22:58:54 重置。"}

22:58:54 is Beijing time, but the body scan read it as UTC and force-opened
the provider until 2026-11-11 06:58:54Z — eight hours after the real reset,
so a healthy channel stayed walled off all morning.

Providers now take an optional offset that is applied only to zone-less
timestamps in that provider's own error text:

[providers.one_api_zhipu_1]
utc_offset = "+08:00"   # also +0800 / +08 / -05:30 / Z

Anything that carries its own offset (Retry-After, x-ratelimit-reset-*,
+0800, Z, RFC 3339) is unaffected — the config is an assumption for the
ambiguous case, never an override. Unset keeps the previous UTC reading; an
invalid value is rejected at config load:

provider 'mock' has invalid utc_offset 'Asia/Shanghai' (expected e.g. "+08:00", "+0800" or "Z")

Verification

Check	Result
cargo fmt --all -- --check	clean
cargo clippy --workspace --all-targets -D warnings	clean
cargo test --workspace	668 passed, 0 failed (4 ignored)
cargo test --test integration ×5	105/105 each run
Release binary --version	maxllm 0.3.7
Release binary starts with utc_offset = "+08:00", /health	{"status":"ok"}

End-to-end proof in the integration suite: the same zone-less message leaves
the breaker closed (maxllm_circuit_breaker_state = 0) when
utc_offset = "+08:00" makes the instant already past, and force-opens it
(= 1) without the offset because it reads as 7 h ahead.

Asset

maxllm-server-linux-x86_64-musl — static-pie, sha256
e6d41f3033babd8c0e31e6e102c764977c7a507ac488cfcab4171cfc378b214f

v0.3.6 — replay a large request body on failover

What's in v0.3.6

Patch release on top of v0.3.5. No wire-format or configuration changes —
one gateway bug fix, one routing refinement, and a test-harness fix.

A large request body can now fail over (the multi-minute silent turn)

Pingora keeps at most 64 KiB of the client body for a replay and silently drops
the rest. A real agent request is much larger — codex sends ~100 KB of
instructions and tool schemas — so any failover after the body had been
consumed handed the next upstream a truncated body. The upstream waited for the
rest, the client waited for a response, and neither side ever moved:

1794062827116  status 0  dur=391337ms  coding_home_olefedora_gpu0  /v1/responses

Pingora only hands the body to request_body_filter when it has a retry buffer
or the body is empty (if buffer.is_some() || session.is_body_empty()), and
get_retry_buffer() returns None exactly when the buffer truncated — so for a
large body the filter was never called at all.

pingora-proxy 0.8.0 (and 0.9.1) still has that condition, so it is vendored in
vendor/pingora-proxy with one added term: when the downstream body is already
complete, the filter is driven once with an empty final chunk. The gateway keeps
the client body itself and re-feeds it whole, re-translated for the provider
that now answers (whose model name differs). See vendor/README.md and
vendor/pingora-proxy-maxllm.patch.

Measured after the fix:

94 KB body, plain route          200 in 0.82 s   (was: hang, 391 s)
94 KB body, forced failover      200 in 2.94 s   (403 primary → 200 fallback)

A body-level auto sentinel is rewritten in place

On a route without auto_model the sentinel only exists in the body
({"model": "auto"}), and the provider has to be picked before the body is
read. When the picked provider is itself a pool member — a pinned session, or a
route whose own provider is listed in the pool — the body is now rewritten to
that entry's model instead of forwarding the literal sentinel. Previously the
upstream rejected it and the request failed over to a different member on
every turn, throwing away the upstream prompt cache for the conversation; and a
member that does not validate model names (an Ollama / llama.cpp style server)
would silently serve whatever model it happened to host.

Tests

• the integration harness now claims gateway ports inside the test binary and
  fails loudly when a gateway dies during startup — portpicker handed the
  same port to two parallel tests, so a test could read a 200 from another
  test's gateway (random failures in whichever test lost the race);
• the unclassifiable-429 test pins an unreachable laya endpoint instead of
  silently using the developer's own decision server on port 18405.

cargo test --workspace green (5 consecutive integration runs),
cargo clippy --workspace --all-targets -- -D warnings clean.

Upgrading

No config change. Replace the binary and restart:

sha256sum maxllm-server-linux-x86_64-musl
# bc09f0c52fc59805c6a6cef5df33638b5f396f95ede3aec43e10893ccfb775a8
systemctl --user restart maxllm

MaxLLM v0.3.5 — auto: replay when an upstream hangs up, sentinel never leaks upstream

What's in v0.3.5

Patch release on top of v0.3.2, folding in the three auto fixes made
while chasing a five-minute silent turn. No wire-format or configuration
changes.

auto sentinel works on any route (v0.3.3)

A client that selects auto in its model picker only changes the model
string — its base URL stays on the summary route (/v1/responses). The
sentinel was only recognised through route.auto_model, so the request went
upstream verbatim: every provider answered "unknown model" (404/400), the
client got a 502, and codex-style clients retried with backoff — 337 s of
silence in the reported incident.

The sentinel now means "rotate" regardless of the route: the failover walk
enters the [auto_model] pool instead of the route's own chain, the first hop
pins auto_target/auto_requested so later hops stay inside the pool, and
the rewritten body travels non-verbatim. Cost: one rejected attempt
(100–300 ms). Alternative: a hard 502 for a model the client never meant to
name.

Rotation actually rotates (v0.3.4)

On a route without auto_model, the pool was entered mid-request, so no
select_provider slot ever rotated for it: the walk in pool order pinned every
such request to the first working entry and left the rest idle. The sentinel
path now tries the session binding first and then round-robins over the healthy
entries.

A member that accepts the connection and hangs up (v0.3.5)

A model server that crashed — or whose supervisor still holds the listen socket
while the worker is gone — accepts the TCP connection and closes it before
sending a byte. Pingora reports ConnectionClosed while reading response
headers and by default only replays that on a reused connection, so a dead
pool member handed the client a hard 502 while the rest of the pool sat idle.

error_while_proxy now treats "nothing came back and the connection died" like
a connect failure: it counts against the member's circuit breaker, unpins the
session and replays the request on the next pool entry or fallback, bounded by
the route's failover budget. Errors that arrive after the upstream produced
headers are still never replayed.

The sentinel never reaches an upstream (v0.3.5)

A session pinned to a pool member that also sits in the route's fallback chain
was sent "model": "auto" verbatim: strict providers answered 400, and
llama.cpp-style servers accepted it and served whatever model they host. The
pinned entry's model now replaces the sentinel.

Diagnosis and operations

• Every failed attempt is recorded (provider=status, provider=connect,
  provider=dns, provider=closed) and appended to the 502, so a trailing DNS
  failure no longer hides the real answer of the other providers.
• scripts/pool_health.py probes each pool entry through its own
  /v1/responses/<provider> route and prints a pass/fail table; it exits
  non-zero when any member fails, so it can gate a deploy.

Verification on the deployed gateway

• pool probe: 25 members, 12 healthy, 13 failing (10 GPUs down, one 403, one
  quota 503) — all failures are endpoint availability, none is gateway logic;
• 20 fresh sessions on /v1/responses with model=auto: 20/20 200,
  6 distinct models served;
• 10 sessions on /v1/responses/auto: 10/10 200;
• 5 requests on one session: 5/5 200, all on the same member (sticky);
• zero upstream requests carrying "model": "auto".

Tests

Gateway suite: 129 unit + 100 integration + 4 ignored (integration includes
the new test_upstream_hanging_up_before_answering_fails_over,
test_auto_rotation_survives_a_pool_member_that_hangs_up and
test_sentinel_rewritten_when_the_session_is_pinned_to_a_pool_member).
cargo clippy --all-targets -D warnings clean. One pre-existing unit test
(rate_limit_resolver::tests::unclassifiable_error_falls_back_to_default_cooldown)
fails on a clean checkout as well and is unrelated.

Artifact

maxllm-server-linux-x86_64-musl — x86_64 static-pie, glibc-independent
(maxllm 0.3.5, sha256 227fe7c4802e87e50847ec1d8801d8a619b8c6fa40601dd0d78e5fa88ebf3292).

MaxLLM v0.3.2 — regression-test coverage (admin, gateway, translate, SigV4)

What's in v0.3.2

Patch release on top of v0.3.1. No feature, configuration or wire-format
changes — this release locks the current behaviour in with more regression
tests and ships the static gateway binary used on oleNas01.

Test coverage (behaviour locked in)

• translate: responses translation argument semantics; opencode
  normalization branches; AWS SigV4 date arithmetic and the clock helper.
• gateway: the rate-limit header / body / learned-pattern resolvers;
  provider label mapping, circuit-breaker registry, model-from-path parsing,
  Retry-After, per-provider error normalisation, streamed usage extraction
  (including the bounded SSE tail), and the [auth] section wiring the
  key_auth plugin.
• admin: circuit-breaker / session-binding / rate-limit-pattern / request-log
  store round-trips; team-member removal; unmatched-route 404s.

Other

• chore(git): ignore every cargo-mutants output directory.

Artifacts

• maxllm-server-linux-x86_64-musl — static-pie build of the gateway
  (--version → maxllm 0.3.2), same artifact deployed to oleNas01.

Compatibility

No public API, configuration or wire-format changes: upgrading from v0.3.1 is
drop-in.

MaxLLM v0.3.1 — regression-test suite, Rust 1.99 clippy fix, docker semver tag + manual CI trigger

What's in v0.3.1

Patch release on top of v0.3.0. The feature set is unchanged — this release hardens the
CI/release path and locks the current behaviour in with a large regression-test suite.

CI / release engineering

• ci(docker): the Docker image tag is now derived from the git tag instead of a hardcoded
  0.1.0, so a v0.3.1 tag yields the 0.3.1 image (plus latest and the commit-SHA tag).
• ci: added a workflow_dispatch trigger, so CI can be started manually on any ref
  (this fork's CI is dispatched manually on the release tag).
• fix(plugin): allow clippy::double_must_use on the #[async_trait] traits. Rust 1.99
  added that lint and it fires on the boxed futures generated by async_trait; CI clippy
  (-D warnings) is green again.

Test coverage (behaviour locked in)

• config: serde defaults and the validate() branches (unknown plugin / unknown provider
  references) — config mutation score 21.8% → 96.4%.
• plugin: auto-compaction end to end (truncate / sliding-window / LLM strategies, token
  estimation, apply_llm_compaction against a local HTTP mock).
• plugin: cache TTL / streaming / cacheable-status rules and the rate-limit key
  extraction (header, client IP, session).
• plugin: contract test over every built-in plugin category — name plus the configuration
  keys each category actually reads.
• plugin: guardrail engine (pass / block / log / modify, pre-call and post-call paths) and
  the built-in guardrail names.

Compatibility

No public API, configuration or wire-format changes: upgrading from v0.3.0 is drop-in.

v0.3.0 — routing strategies, Bedrock SigV4, virtual keys, per-key rate limits, provider retries

Highlights

Routing

• Latency-, cost- and least-connections strategies — latency_based, cost_based and least_connections are now implemented, joining the existing round-robin / random / weighted strategies. The per-provider in-flight count is exposed as maxllm_provider_active_requests_total.

Providers

• AWS Bedrock SigV4 — requests to Bedrock are now signed with AWS SigV4 and the model name is routed into the invocation URL. Adds the aws_sigv4 signing module in maxllm-translate.
• Same-provider retry — num_retries and retry_backoff_ms are now wired: a transient 5xx is retried on the same provider (with exponential backoff) before the gateway fails over. New metric: maxllm_provider_retries_total{provider,status}.

Multi-tenant gateway

• Virtual keys — API keys minted through the admin API are now enforced at the gateway: 401 for unknown/disabled keys, 402 when the spend budget is exhausted, and a model allow-list per key.
• Per-key rate limits — each virtual key can carry its own requests_per_minute / tokens_per_minute budget, enforced before the request is proxied.
• Top-level [auth] and [rate_limit] config sections are now honoured (previously silently ignored) and synthesised into the hot-state plugin chain.

Server

• [server] socket knobs — tcp_reuseport and tcp_fastopen are applied to the listening socket (SO_REUSEPORT / TCP_FASTOPEN).

CLI

• make run / make run-debug and the README now use the real argument syntax (maxllm start --config …, maxllm config check|providers).
• maxllm init writes a starter config with global_plugins at the top level (previously nested under [server], which silently disabled authentication) and the correct requests_per_minute field name.

Quality gates

• cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all -- --check are both clean — the CI workflow's clippy and format gates now pass.
• Full test suite green: 45 admin / 9 config / 96 gateway / 93 integration (+4 live, ignored by default) / 139 plugin / 177 translate.

Breaking / migration notes

• None — existing configs keep working; the previously ignored top-level [auth] / [rate_limit] sections now take effect, so re-check them if you had leftover defaults.

Full changelog: v0.2.0...v0.3.0