Known limit: self-hosted OpenAI-compatible servers (token usage absent on the client surface, recoverable at the server surface)

[qaisermehdi3-coder]

Proposal for standard/known-limits-unprovable-by-surface.md §2: a row for self-hosted OpenAI-compatible servers, which the topology table does not cover yet.

Where it comes from. pydantic/pydantic-ai#8704 (I opened it): a provider response with no usage object is recorded as zero tokens. Their triage reproduced it end to end: priced at exactly $0.00, and token and cost limits never trip.

Why self-hosted is where this happens most. Checked in vLLM's source (main at c3f5270): a streamed response carries usage only if the client sends stream_options.include_usage, or the server runs with --enable-force-include-usage, which is off by default (should_include_usage in vllm/entrypoints/serve/utils/api_utils.py; default in vllm/entrypoints/launchers/cli_args.py). A client that does not ask gets no usage. On the client surface, per-request tokens for those streams are UNPROVABLE, and recording them as 0 is a FAIL.

Expanding the surface. vLLM's /metrics exposes server-side counters vllm:prompt_tokens, vllm:generation_tokens and vllm:prompt_tokens_cached, plus per-request histograms such as vllm:request_inference_time_seconds (vllm/v1/metrics/loggers.py). That makes totals over a period provable, by reconciling client sums against the server counters. Per-request attribution stays UNPROVABLE from counters alone.

On self-hosted, cost is GPU time, not a token price. And GPU time for the same config is not a constant. My measurements on one L40S, same container, minutes apart, three repeats of 32 configs: CUDA-graph configs spread 0.7% mean (2.6% worst), eager configs 37.3% mean (78.5% worst). Data: https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/l40s-own. So a GPU-time cost from one run describes that machine and should carry its spread.

Proposed row. Topology: self-hosted OpenAI-compatible server (I only checked vLLM). Not visible: stream usage unless the client requests it. UNPROVABLE: per-request tokens on those streams. Surface expansion: server counters for period totals; GPU time × hourly rate for cost, with the spread disclosed.

Happy to write this up as a PR or a conformance vector if useful.

— Qaiser (Qvunex: measuring GPU and self-hosted inference cost)

[qaisermehdi3-coder]

Tested it live. Colab T4, vLLM 0.27.1, Qwen2.5-0.5B-Instruct. Three identical requests at temperature 0: one non-streaming, one stream without stream_options, one stream with include_usage.

Client side: the plain request reported 36 prompt / 16 output tokens; the stream without stream_options reported nothing. Server side, /metrics before and after (vllm:request_success_total, vllm:request_prompt_tokens_sum, vllm:request_generation_tokens_sum): 3 requests, 108 / 48 tokens. Client recorded 72 / 32. Gap: 36 / 16, exactly the unreported request.

So the period total moves from UNPROVABLE to provable at the server surface; which request it belonged to still does not. The comparison is qvunex_single.py --reconcile before.txt after.txt in my repo: stdlib only, reads saved snapshots, no network code.