v0.3.10 — drop the Command Code CLI bridge (vendor TOS)

v0.3.10

撤下 Command Code CLI 桥接

厂商的 /alpha/* 面是他们自家 cmd 客户端的私有协议，直连会回
Proxy use detected. This endpoint only serves CLI. Use Command Code provider API instead.，
并写明「Continued proxying of your subscription violates the TOS and will result in account ban.」
把包月订阅经自建网关转出去属于厂商明令禁止的用法，因此本版本把桥接整体撤掉：

• 删除 crate crates/maxllm-commandcode/（独立 sidecar，工作区内无其它 crate 依赖它）；
• maxllm.toml 移除 [providers.cmd_cli] 与两条路由 /v1/responses/cli_cmd、/v1/channels/cli_cmd；
• 线上（oleNas01）停用 maxllm-commandcode.service，二进制移出 bin/；
• 保留 kind = "commandcode"（厂商授权的 Provider API 路径）：Go 计划下它恒 403
  upgrade_required，升级 Provider 计划后 /v1/responses/cmd 原生透传即可用，网关侧无需改动。

附带：逆向 cmd 的识别机制（写入 docs/ops_review_20261108.md 附录 C）

cmd 并没有一个「魔法 token」证明自己是 CLI，识别是分层的：

1. 路径面 — /alpha/* = CLI 私有协议，/provider/v1/* = 公开 API；
2. 自称头 — User-Agent: cli（常量，每个请求都带）、x-cli-surface
   （仅 acp/rpc 面才带，普通 CLI 不带）、x-cli-environment: production、x-command-code-version；
3. 设备指纹 — 首次运行把 thumbmark = sha256(salt + machineId|MACs) 上报
   /alpha/fingerprint/record，components 里 machineId / MAC / 用户名 / 主机名 / git 邮箱全部 hash，
   并标 runtime: "cli"；
4. 凭证 — Authorization: Bearer <apiKey>，来自 ~/.commandcode/auth.json
   （或 COMMAND_CODE_API_KEY 环境变量），登录走 https://commandcode.ai/studio/auth/cli；
5. 服务端判定 — 路径族决定「这是 CLI 面」，套餐决定「能不能用 API」，/alpha/* 上另有反代理护栏。

实测补充：User-Agent: cli 并不是所有 /alpha/* 端点的准入条件（/alpha/namespaces、
/alpha/billing/credits、/alpha/sandbox/sessions 用普通 curl UA 同样 200）——它只是客户端自称；
真正的闸门在服务端。另外 POST /alpha/generate 的 403 响应体会把 CLI 上传的内容寻址文件池索引
（syspool/<user>/main/<sha256>）原样回显，属上游的一个信息泄漏。

验证

• cargo fmt --all -- --check、cargo clippy --workspace --all-targets -- -D warnings、
  cargo test --workspace 全绿；
• python3 scripts/check_responses_policy.py --config maxllm.toml → OK（每个能上 Responses 的上游仍是 native）。

v0.3.9 — Command Code CLI bridge (native Responses), cache-hit billing, client-abort accounting

v0.3.9

新增

• Command Code CLI 桥接（新 crate crates/maxllm-commandcode）：把厂商自有的
  cmd -p <query> --output-format json 客户端包成 OpenAI 服务，原生实现 Responses 契约
  （POST /v1/responses，流式 SSE 事件按 response.created → in_progress →
  output_item.added → content_part.added → output_text.delta → output_text.done →
  content_part.done → output_item.done → response.completed → data: [DONE] 顺序，
  每个事件带单调 sequence_number），并同时提供 Chat Completions。
  reasoning.effort 映射到 CLI 的 --effort。19 个单测，clippy -D warnings 通过。
• commandcode.ai provider（native Responses 优先）与 open-design.ai provider
  （原生 Rust 实现），均已进 /v1/models 与独立路由。
• prompt-cache 计费：/v1/responses 读取上游返回的缓存命中数，统一两套 usage 解析器；
  admin 按模型的 cache rate 给缓存命中计费（线上实测 MiniMax 命中率 98.5%，批量成本降到 1/4）。
• 可观测性：stream 截断、用量与成本可见且可自纠正；client_aborted_total 与
  provider 故障分开统计；/health 不再把 half-open 熔断器算作 open；看板按 provider
  分别报 client abort 与截断。

修复

• 未知模型自动改写：上游 404（vLLM 风格）或 failover 时，把 body 里的 model 改写成
  该 provider 的 default_model 后重试一次，避免整条 fallback 链集体 400。
• 客户端中断不再记成 provider 故障：被下游中断的流按 client abort 记账，
  不再污染熔断器与截断率；fast-path 的中断抓取也不再产生幽灵 request_logs 行。
• 熔断器不再每 60 秒抖动（stream demotion hold 升级）；上游已消失的 provider 其
  resilience 状态会被清理；native Responses 路径的 tool_choice: "any" 修复。
• 落盘的录制 body 在写盘前脱敏凭据与 PII；/health 半开探测不再误开熔断器。

运维 / 工具

• scripts/：新增 openobserve_alerts.py、stream_outcomes.py（按“谁关的流”区分 200
  的归属）、redact_recordings.py、rotate_logs.py；pool_health.py 改为探测每个具名
  auto 池；check_responses_policy.py 校验「凡是能上 Responses 的上游都必须 native」。
• docs/ops_review_20261108.md：生产网关运维评审（截断、用量核算、可观测性缺口）。

验证

• CI（workflow_dispatch，ref v0.3.9）：Check / Test / Clippy / Format /
  Responses policy / Build Release 全部通过。
• 本地门禁：cargo fmt --all -- --check、cargo clippy --workspace --all-targets -- -D warnings、
  cargo test --workspace 全绿。

注：本仓库的 push 事件不触发 Actions，CI 需手动 gh workflow run CI --ref v0.3.9；
因此 v0.3.9 未发布 Docker 镜像（docker job 需要 push 事件）。

v0.3.8 — named auto pools (auto = cloud, auto_home = home)

Named auto pools

[auto_models.<key>] declares extra rotation pools next to the default [auto_model], each answering to its own sentinel model name. A deployment can now keep the rotation sets disjoint — auto over cloud vendors, auto_home over the home GPUs — instead of one pool that mixes machines which are off half the day with cloud APIs.

• [auto_model] stays the default pool (sentinel auto), so an existing single-pool config is unchanged. A named pool's sentinel is its table key unless the block sets name.
• The route's provider names the pool an auto_model = true route rotates over when it matches one; otherwise the first declared pool serves it.
• RequestCtx.auto_pool pins the pool for the whole request: the sentinel that entered the request also bounds every later failover hop, so a request can no longer drift from one rotation set into another (previously one auto pool was searched for any hop's provider name).
• GET /v1/models lists every enabled pool sentinel (owned_by: maxllm-auto).
• Pool names must be non-empty and unique; an auto route naming a missing or disabled pool fails validation at startup.

Verification

• cargo test --workspace: all green (211 translate tests etc.).
• cargo test -p maxllm-gateway --test integration: 107 passed, 4 ignored (2 new: pools keep their sets apart; a body-only auto_home sentinel is adopted into its named pool).
• Binary: musl static, sha256 f80d23577e150935cf3a788149ac5afbd508a220d68ee5919157b6b5a37e3be5.

v0.3.7 — per-provider utc_offset for zone-less reset times

What changed

429 error messages often state when the quota resets, and many upstreams
write that instant in their own local time with no zone marker. Zhipu's
production shape:

{"code":"1310","message":"您已达到每周/每月使用上限，您的限额将在 2026-11-10 22:58:54 重置。"}

22:58:54 is Beijing time, but the body scan read it as UTC and force-opened
the provider until 2026-11-11 06:58:54Z — eight hours after the real reset,
so a healthy channel stayed walled off all morning.

Providers now take an optional offset that is applied only to zone-less
timestamps in that provider's own error text:

[providers.one_api_zhipu_1]
utc_offset = "+08:00"   # also +0800 / +08 / -05:30 / Z

Anything that carries its own offset (Retry-After, x-ratelimit-reset-*,
+0800, Z, RFC 3339) is unaffected — the config is an assumption for the
ambiguous case, never an override. Unset keeps the previous UTC reading; an
invalid value is rejected at config load:

provider 'mock' has invalid utc_offset 'Asia/Shanghai' (expected e.g. "+08:00", "+0800" or "Z")

Verification

Check	Result
cargo fmt --all -- --check	clean
cargo clippy --workspace --all-targets -D warnings	clean
cargo test --workspace	668 passed, 0 failed (4 ignored)
cargo test --test integration ×5	105/105 each run
Release binary --version	maxllm 0.3.7
Release binary starts with utc_offset = "+08:00", /health	{"status":"ok"}

End-to-end proof in the integration suite: the same zone-less message leaves
the breaker closed (maxllm_circuit_breaker_state = 0) when
utc_offset = "+08:00" makes the instant already past, and force-opens it
(= 1) without the offset because it reads as 7 h ahead.

Asset

maxllm-server-linux-x86_64-musl — static-pie, sha256
e6d41f3033babd8c0e31e6e102c764977c7a507ac488cfcab4171cfc378b214f

v0.3.6 — replay a large request body on failover

What's in v0.3.6

Patch release on top of v0.3.5. No wire-format or configuration changes —
one gateway bug fix, one routing refinement, and a test-harness fix.

A large request body can now fail over (the multi-minute silent turn)

Pingora keeps at most 64 KiB of the client body for a replay and silently drops
the rest. A real agent request is much larger — codex sends ~100 KB of
instructions and tool schemas — so any failover after the body had been
consumed handed the next upstream a truncated body. The upstream waited for the
rest, the client waited for a response, and neither side ever moved:

1794062827116  status 0  dur=391337ms  coding_home_olefedora_gpu0  /v1/responses

Pingora only hands the body to request_body_filter when it has a retry buffer
or the body is empty (if buffer.is_some() || session.is_body_empty()), and
get_retry_buffer() returns None exactly when the buffer truncated — so for a
large body the filter was never called at all.

pingora-proxy 0.8.0 (and 0.9.1) still has that condition, so it is vendored in
vendor/pingora-proxy with one added term: when the downstream body is already
complete, the filter is driven once with an empty final chunk. The gateway keeps
the client body itself and re-feeds it whole, re-translated for the provider
that now answers (whose model name differs). See vendor/README.md and
vendor/pingora-proxy-maxllm.patch.

Measured after the fix:

94 KB body, plain route          200 in 0.82 s   (was: hang, 391 s)
94 KB body, forced failover      200 in 2.94 s   (403 primary → 200 fallback)

A body-level auto sentinel is rewritten in place

On a route without auto_model the sentinel only exists in the body
({"model": "auto"}), and the provider has to be picked before the body is
read. When the picked provider is itself a pool member — a pinned session, or a
route whose own provider is listed in the pool — the body is now rewritten to
that entry's model instead of forwarding the literal sentinel. Previously the
upstream rejected it and the request failed over to a different member on
every turn, throwing away the upstream prompt cache for the conversation; and a
member that does not validate model names (an Ollama / llama.cpp style server)
would silently serve whatever model it happened to host.

Tests

• the integration harness now claims gateway ports inside the test binary and
  fails loudly when a gateway dies during startup — portpicker handed the
  same port to two parallel tests, so a test could read a 200 from another
  test's gateway (random failures in whichever test lost the race);
• the unclassifiable-429 test pins an unreachable laya endpoint instead of
  silently using the developer's own decision server on port 18405.

cargo test --workspace green (5 consecutive integration runs),
cargo clippy --workspace --all-targets -- -D warnings clean.

Upgrading

No config change. Replace the binary and restart:

sha256sum maxllm-server-linux-x86_64-musl
# bc09f0c52fc59805c6a6cef5df33638b5f396f95ede3aec43e10893ccfb775a8
systemctl --user restart maxllm

MaxLLM v0.3.5 — auto: replay when an upstream hangs up, sentinel never leaks upstream

What's in v0.3.5

Patch release on top of v0.3.2, folding in the three auto fixes made
while chasing a five-minute silent turn. No wire-format or configuration
changes.

auto sentinel works on any route (v0.3.3)

A client that selects auto in its model picker only changes the model
string — its base URL stays on the summary route (/v1/responses). The
sentinel was only recognised through route.auto_model, so the request went
upstream verbatim: every provider answered "unknown model" (404/400), the
client got a 502, and codex-style clients retried with backoff — 337 s of
silence in the reported incident.

The sentinel now means "rotate" regardless of the route: the failover walk
enters the [auto_model] pool instead of the route's own chain, the first hop
pins auto_target/auto_requested so later hops stay inside the pool, and
the rewritten body travels non-verbatim. Cost: one rejected attempt
(100–300 ms). Alternative: a hard 502 for a model the client never meant to
name.

Rotation actually rotates (v0.3.4)

On a route without auto_model, the pool was entered mid-request, so no
select_provider slot ever rotated for it: the walk in pool order pinned every
such request to the first working entry and left the rest idle. The sentinel
path now tries the session binding first and then round-robins over the healthy
entries.

A member that accepts the connection and hangs up (v0.3.5)

A model server that crashed — or whose supervisor still holds the listen socket
while the worker is gone — accepts the TCP connection and closes it before
sending a byte. Pingora reports ConnectionClosed while reading response
headers and by default only replays that on a reused connection, so a dead
pool member handed the client a hard 502 while the rest of the pool sat idle.

error_while_proxy now treats "nothing came back and the connection died" like
a connect failure: it counts against the member's circuit breaker, unpins the
session and replays the request on the next pool entry or fallback, bounded by
the route's failover budget. Errors that arrive after the upstream produced
headers are still never replayed.

The sentinel never reaches an upstream (v0.3.5)

A session pinned to a pool member that also sits in the route's fallback chain
was sent "model": "auto" verbatim: strict providers answered 400, and
llama.cpp-style servers accepted it and served whatever model they host. The
pinned entry's model now replaces the sentinel.

Diagnosis and operations

• Every failed attempt is recorded (provider=status, provider=connect,
  provider=dns, provider=closed) and appended to the 502, so a trailing DNS
  failure no longer hides the real answer of the other providers.
• scripts/pool_health.py probes each pool entry through its own
  /v1/responses/<provider> route and prints a pass/fail table; it exits
  non-zero when any member fails, so it can gate a deploy.

Verification on the deployed gateway

• pool probe: 25 members, 12 healthy, 13 failing (10 GPUs down, one 403, one
  quota 503) — all failures are endpoint availability, none is gateway logic;
• 20 fresh sessions on /v1/responses with model=auto: 20/20 200,
  6 distinct models served;
• 10 sessions on /v1/responses/auto: 10/10 200;
• 5 requests on one session: 5/5 200, all on the same member (sticky);
• zero upstream requests carrying "model": "auto".

Tests

Gateway suite: 129 unit + 100 integration + 4 ignored (integration includes
the new test_upstream_hanging_up_before_answering_fails_over,
test_auto_rotation_survives_a_pool_member_that_hangs_up and
test_sentinel_rewritten_when_the_session_is_pinned_to_a_pool_member).
cargo clippy --all-targets -D warnings clean. One pre-existing unit test
(rate_limit_resolver::tests::unclassifiable_error_falls_back_to_default_cooldown)
fails on a clean checkout as well and is unrelated.

Artifact

maxllm-server-linux-x86_64-musl — x86_64 static-pie, glibc-independent
(maxllm 0.3.5, sha256 227fe7c4802e87e50847ec1d8801d8a619b8c6fa40601dd0d78e5fa88ebf3292).

MaxLLM v0.3.2 — regression-test coverage (admin, gateway, translate, SigV4)

What's in v0.3.2

Patch release on top of v0.3.1. No feature, configuration or wire-format
changes — this release locks the current behaviour in with more regression
tests and ships the static gateway binary used on oleNas01.

Test coverage (behaviour locked in)

• translate: responses translation argument semantics; opencode
  normalization branches; AWS SigV4 date arithmetic and the clock helper.
• gateway: the rate-limit header / body / learned-pattern resolvers;
  provider label mapping, circuit-breaker registry, model-from-path parsing,
  Retry-After, per-provider error normalisation, streamed usage extraction
  (including the bounded SSE tail), and the [auth] section wiring the
  key_auth plugin.
• admin: circuit-breaker / session-binding / rate-limit-pattern / request-log
  store round-trips; team-member removal; unmatched-route 404s.

Other

• chore(git): ignore every cargo-mutants output directory.

Artifacts

• maxllm-server-linux-x86_64-musl — static-pie build of the gateway
  (--version → maxllm 0.3.2), same artifact deployed to oleNas01.

Compatibility

No public API, configuration or wire-format changes: upgrading from v0.3.1 is
drop-in.

MaxLLM v0.3.1 — regression-test suite, Rust 1.99 clippy fix, docker semver tag + manual CI trigger

What's in v0.3.1

Patch release on top of v0.3.0. The feature set is unchanged — this release hardens the
CI/release path and locks the current behaviour in with a large regression-test suite.

CI / release engineering

• ci(docker): the Docker image tag is now derived from the git tag instead of a hardcoded
  0.1.0, so a v0.3.1 tag yields the 0.3.1 image (plus latest and the commit-SHA tag).
• ci: added a workflow_dispatch trigger, so CI can be started manually on any ref
  (this fork's CI is dispatched manually on the release tag).
• fix(plugin): allow clippy::double_must_use on the #[async_trait] traits. Rust 1.99
  added that lint and it fires on the boxed futures generated by async_trait; CI clippy
  (-D warnings) is green again.

Test coverage (behaviour locked in)

• config: serde defaults and the validate() branches (unknown plugin / unknown provider
  references) — config mutation score 21.8% → 96.4%.
• plugin: auto-compaction end to end (truncate / sliding-window / LLM strategies, token
  estimation, apply_llm_compaction against a local HTTP mock).
• plugin: cache TTL / streaming / cacheable-status rules and the rate-limit key
  extraction (header, client IP, session).
• plugin: contract test over every built-in plugin category — name plus the configuration
  keys each category actually reads.
• plugin: guardrail engine (pass / block / log / modify, pre-call and post-call paths) and
  the built-in guardrail names.

Compatibility

No public API, configuration or wire-format changes: upgrading from v0.3.0 is drop-in.

v0.3.0 — routing strategies, Bedrock SigV4, virtual keys, per-key rate limits, provider retries

Highlights

Routing

• Latency-, cost- and least-connections strategies — latency_based, cost_based and least_connections are now implemented, joining the existing round-robin / random / weighted strategies. The per-provider in-flight count is exposed as maxllm_provider_active_requests_total.

Providers

• AWS Bedrock SigV4 — requests to Bedrock are now signed with AWS SigV4 and the model name is routed into the invocation URL. Adds the aws_sigv4 signing module in maxllm-translate.
• Same-provider retry — num_retries and retry_backoff_ms are now wired: a transient 5xx is retried on the same provider (with exponential backoff) before the gateway fails over. New metric: maxllm_provider_retries_total{provider,status}.

Multi-tenant gateway

• Virtual keys — API keys minted through the admin API are now enforced at the gateway: 401 for unknown/disabled keys, 402 when the spend budget is exhausted, and a model allow-list per key.
• Per-key rate limits — each virtual key can carry its own requests_per_minute / tokens_per_minute budget, enforced before the request is proxied.
• Top-level [auth] and [rate_limit] config sections are now honoured (previously silently ignored) and synthesised into the hot-state plugin chain.

Server

• [server] socket knobs — tcp_reuseport and tcp_fastopen are applied to the listening socket (SO_REUSEPORT / TCP_FASTOPEN).

CLI

• make run / make run-debug and the README now use the real argument syntax (maxllm start --config …, maxllm config check|providers).
• maxllm init writes a starter config with global_plugins at the top level (previously nested under [server], which silently disabled authentication) and the correct requests_per_minute field name.

Quality gates

• cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all -- --check are both clean — the CI workflow's clippy and format gates now pass.
• Full test suite green: 45 admin / 9 config / 96 gateway / 93 integration (+4 live, ignored by default) / 139 plugin / 177 translate.

Breaking / migration notes

• None — existing configs keep working; the previously ignored top-level [auth] / [rate_limit] sections now take effect, so re-check them if you had leftover defaults.

Full changelog: v0.2.0...v0.3.0

MaxLLM v0.2.0

MaxLLM v0.2.0 — oliveagle fork 发布

以 one-api 渠道池为供给、OpenAI Responses 契约为出口的 Pingora 网关。本版本在
upstream v0.1.0（2026-04）之上再叠加 57 个提交，把「多渠道统一入口 + 7x24 稳定供给」
所需的完整链路一次交付（NAS 线上版本同源）。

新增能力

OpenAI Responses API 契约

• endpoint_type = "responses"：Responses ↔ chat completions 双向翻译，任何 chat 型上游都能吃
• 原生 Responses 中继（native_responses），opencode 类上游按字节透传
• SSE 桥接：非流式客户端把上游 SSE 折叠成单个 JSON 对象；流式客户端补齐
  sequence_number、正确结束事件、response.failed 错误语义
• 工具调用、推理(reasoning)、图片附件、并行 tool_calls、tool_choice 在两种契约间无损互转

opencode.ai ("zen") 线格式归一化

• 会话头 x-opencode-session 缺失/空白时注入；tool_choice: "any" → "required"
• tool_call arguments 对象强制字符串化、孤儿/前向引用 tool 消息清理、缺失 tool 结果补齐
• reasoning / reasoning_text 方言镜像到 reasoning_content；UA 安全注入与指纹窄化

韧性（Resilience）

• 3 态熔断器（Closed / Open / HalfOpen）+ 配额冷却期，状态持久化到 SQLite
• session-sticky 路由：会话钉住 provider，保住上游上下文
• 请求级 HTTP failover：按路由预算跨 fallback 链切换；DNS 解析失败同样计入熔断并换家
• 每 provider 可配连接超时；优雅退出排空上限 30s，SIGTERM 立即退出

限流时间精确解析（分层）

1. 正则/结构化层：直接读上游的 reset 时间、Retry-After、配额字段
2. 学习层：累积上游限流文本的表达方式，越用越准
3. laya 决策层：仍不确定时交给 laya 决策模型判定

据此精确计算恢复时间点，而不是盲目重试浪费时间。

auto 哨兵模型（自动轮替）

• 客户端发 model=auto → 网关在 [auto_model] 池内轮替所有模型与 provider
• session-sticky 优先，无绑定才 round-robin；请求体 model 改写为命中渠道自己的模型
• 池子写在网关自己的配置里，不依赖 one-api 的渠道表
• failover 在池内重选（跳过已尝试/熔断条目）

可观测

• Prometheus 指标（熔断状态、亲和命中、延迟分位、token 吞吐、failover 事件）
• JSONL 审计录制（请求/响应体）、SQLite 管理存储与虚拟密钥

文档

• README.md：配置、auto 池、opencode 转换、韧性策略
• RESPONSES_CONTRACT.md：Responses 契约与事件语义

产物

maxllm-server-linux-x86_64-musl —— musl 静态构建（static-pie，glibc 无关），可直接丢到
Debian/Alpine 等任意 Linux 上运行（NAS 部署即用此产物）。

scp maxllm-server-linux-x86_64-musl <host>:/path/maxllm-server
./maxllm-server --config maxllm.toml