Compatibility Clinic #1: seeking 10 authorized endpoint runs

[whyiug]

We are looking for the first 10 reviewed field reports from people who run an LLM API endpoint they own or are explicitly authorized to test.

The useful question is narrow: where does an endpoint return 200 OK, but still fail at the response-envelope, SSE-lifecycle, or client boundary? PASS, FAIL, WARN, and INCONCLUSIVE results are all welcome. This is not a provider scorecard or compatibility certification.

Try v0.1.1

On Linux or macOS, install the exact release and run the no-key loopback demo:

curl --proto '=https' --tlsv1.2 -fsSL \
  https://raw.githubusercontent.com/whyiug/agentapi-doctor/v0.1.1/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
doctor demo

Then check one endpoint you own, operate, or have explicit permission to assess:

export DOCTOR_TOKEN='replace-with-an-authorized-test-token'

doctor test \
  --base-url 'https://your-authorized-endpoint.example/v1' \
  --protocol openai-chat \
  --model 'your-model-id' \
  --auth-env DOCTOR_TOKEN

Protocol IDs are openai-chat, openai-responses, and anthropic-messages. The Quick Start covers custom headers and trusted local HTTP.

Each endpoint run sends at most four requests under one 60-second deadline. Structural checks request 64 output tokens; the OpenAI Chat and Responses terminal-status check requests 512. An endpoint can reject or ignore those request fields, so they are not client-enforced cost ceilings.

Reply with five lines

- Doctor version:
- Protocol:
- Result:
- Failing or inconclusive check: (write `none` if all passed)
- One sentence: what helped or confused you?

A summary is enough. A report, endpoint name, provider name, run ID, or raw response is not required.

Optional context

• OS / architecture
• Self-hosted runtime, private gateway, or provider API
• Runtime or gateway version
• Model family or a redacted label
• Sanitized observed behavior or real-client symptom
• Reproduction frequency

Before sharing

Only test a system you own or are explicitly authorized to assess. Do not post credentials, private endpoint URLs, customer data, production prompts, complete raw responses, or an artifact you have not reviewed. Doctor keeps evidence in the local .agentapi/ directory and does not upload it. Built-in redaction helps with known secret forms, but redaction is not anonymization.

For a possible vulnerability in AgentAPI Doctor, use private vulnerability reporting instead of this thread.

For usable reports, the maintainer will suggest the smallest next diagnostic step and, where practical, turn the boundary into a synthetic fixture, regression test, documentation change, or upstream-ready report. Permission will be requested before quoting someone or publishing submitted material.

The clinic closes after 10 usable reports or on 2026-08-13, whichever comes first.

v0.1.1 boundary

One Quick Check runs four selected raw HTTP checks for one exact endpoint, model, protocol, and versioned artifact set. A result does not prove arbitrary SDK, Agent, provider, deployment, or future-version compatibility. Registry, Matrix, hosted verification, and vendor certification remain outside the supported v0.1.x product. See Known Limitations.

---

中文简述

本帖征集首批 10 份经过人工检查的授权 endpoint 运行反馈。请只测试你自己运营或已获明确授权的 endpoint。安装 v0.1.1 后，先运行无需 API key、不会访问外部服务的 doctor demo，再运行一次有界 endpoint 检查。

回复只需 5 行：Doctor version、protocol、result、失败或 INCONCLUSIVE 的 check，以及一句“什么有帮助或哪里不清楚”。不需要公开 endpoint、Provider、run ID 或完整报告。请勿提交 token、私有 URL、生产数据或未经审阅的 payload。

[Yigtwxx]

Context: Forrest reached out by email about openclaw/openclaw#101608 (which makes response.incomplete a usage-bearing terminal event) and asked which response.incomplete assertion would be most useful as Doctor's first executable fixture. Answering here so it's public. This isn't a field report, so I'm skipping the run-summary template.

The assertion I'd start with: a stream that terminates with response.incomplete must carry a populated usage object on that terminal event, and must never emit response.completed.

Reasoning, from the bug class we just fixed in openclaw (openclaw/openclaw#100954): clients commonly attach their usage/cost accounting only to the response.completed branch. When the server truncates at max_output_tokens (or a content filter fires), the stream ends with response.incomplete instead — and those clients silently record zero tokens and zero cost while the provider still bills the request. Nothing visibly breaks; accounting just drifts. That is exactly the "200 OK but broken at the SSE-lifecycle boundary" category your clinic post describes, and a fixture would have caught it.

Concretely, the fixture I'd build first:

1. Emit a few ordinary output-delta events, then a single terminal event:
   • type: "response.incomplete",
   • response.status: "incomplete",
   • response.incomplete_details.reason: "max_output_tokens",
   • response.usage populated with nonzero input_tokens / output_tokens, and ideally nonzero input_tokens_details.cached_tokens (the cache buckets are where client-side accounting mistakes hide most often).
2. Assert:
   • exactly one terminal event, and its type is response.incomplete — no response.completed anywhere in the stream;
   • usage is present on the terminal event and internally consistent (total_tokens = input + output);
   • (for a client-under-test) whatever usage the client reports afterward is nonzero.

On the live-check side this is cheap to force within your four-request budget: requesting the API-minimum max_output_tokens (16 on the Responses API) with any non-trivial prompt reliably terminates the stream with response.incomplete / max_output_tokens in a single call.

On "validate cost accounting": I'd keep Doctor out of price math entirely. Cost is computed client-side from rate tables, so the wire-level invariant worth asserting is only that the terminal event delivers the usage envelope the client needs. Whether a client then prices it correctly is that client's bug — as it was ours.

If you want a second fixture after that: same shape with incomplete_details.reason: "content_filter", since clients that string-match on "max_output_tokens" tend to get that variant wrong.

[whyiug]

@Yigtwxx, thank you for turning the OpenClaw bug class into a concrete fixture proposal. I shipped the first product-boundary coverage in v0.1.1.

What landed:

• an offline four-request Responses run where the terminal status is response.incomplete, the reason is max_output_tokens, and no response.completed is emitted;
• a coherent usage envelope including nonzero cached_tokens, plus a missing-usage variant;
• the normal product result remains 3 PASS + 1 INCONCLUSIVE with zero findings, and Doctor still does no price math.

One deliberate boundary: I did not make usage presence a normative pass/fail requirement. Under Doctor's current catalog, provider usage is optional diagnostic evidence, so missing usage stays unknown/INCONCLUSIVE rather than becoming zero or a target failure. I also did not add a fifth live request or force the live Quick Check to truncate; the four-request budget and current live semantics are unchanged.

The content_filter variant is deferred until its terminal-outcome taxonomy receives separate review.

Implementation: PR #15

This materially improved the fixture while keeping the claim bounded to wire evidence. Thank you.

[Yigtwxx]

That's a faithful implementation of the proposal — and shipping the missing-usage variant alongside it was a good call. Keeping usage presence non-normative (INCONCLUSIVE rather than pass/fail) seems right for a diagnostics tool: on OpenAI-compatible gateways, "terminal event arrived without a usage envelope" is a real-world state, not necessarily a spec violation, and the signal a client integrator actually needs is exactly the one you now surface — whether the envelope was delivered at all. Staying out of price math and inside the four-request budget both match what I'd hoped for.

Deferring content_filter until its terminal-outcome taxonomy is reviewed makes sense; that variant is only worth adding once the semantics are pinned down.

Thanks for the quick turnaround — good luck with the clinic.