GitHub Copilot's /models carries the capacities the catalog short-circuit assumes are missing

[allenliang2022]

Summary

discovery.ts answers a catalog-backed route from the installed pi-ai catalog and deliberately makes no network call, because "the installed entries carry context windows and output caps no listing endpoint reports". For GitHub Copilot that premise does not hold: its GET /models reports everything a catalog entry needs. So a Copilot route could discover models live instead of waiting for a pi-ai release.

This is a report rather than a patch, since CONTRIBUTING.md says external PRs cannot be accepted.

What Copilot's listing actually carries

Per model, GET {endpoint}/models returns:

catalog field	listing source
wire protocol	supported_endpoints
context window	capabilities.limits.max_context_window_tokens
output cap	capabilities.limits.max_output_tokens
reasoning levels	capabilities.supports.reasoning_effort[]
image input	capabilities.supports.vision
selectability	model_picker_enabled, policy.state

A real entry (2026-08-19):

{
  "id": "gemini-3.7-flash",
  "name": "Gemini 3.7 Flash",
  "vendor": "Google",
  "model_picker_enabled": true,
  "policy": { "state": "enabled" },
  "supported_endpoints": ["/chat/completions"],
  "capabilities": {
    "type": "chat",
    "limits": { "max_context_window_tokens": 1000000, "max_output_tokens": 64000 },
    "supports": { "reasoning_effort": ["low","medium","high"], "vision": true, "tool_calls": true }
  }
}

Why it matters today

reuseCatalogProvider drops catalog-owned dynamic refresh, and filterModels only intersects, so a model GitHub ships after pi-ai's last release cannot be selected at all — not for want of entitlement, but because the id is absent from a compiled-in JSON file. Adopting it means raising a dependency range, rebuilding, and restarting the host.

Measured on my install: @earendil-works/pi-ai 0.82.1's Copilot catalog had neither gemini-3.6-flash nor gemini-3.7-flash; 0.84.0 (2026-08-06) added 3.6, and 0.84.2 (2026-08-14) added 3.7. Upstream tracks Copilot within about a week — the lag is the dependency bump, not the data.

Also worth noting: the caret range on a 0.x dependency (^0.82.1) pins the minor, so pnpm up cannot cross it; and 0.84 requires two drift-gate updates in llm-pi-ai (baseten added to OpenAICompletionsCompat['thinkingFormat'], and StopReason gaining pending/deferred, which breaks the exhaustive switch in stream.ts).

Two rules a live path has to respect

I built a local plugin to check whether this is actually feasible, and both of these came out of running it rather than reading code.

1. Protocol choice is derivable, and the ordering matters. Many Copilot models accept several endpoints — every Anthropic model also answers /chat/completions, and some GPT models answer both /responses and /chat/completions. This order reproduces pi-ai's own curated api for every model that both the listing and the catalog describe (23 models on my account, asserted in a unit test against getBuiltinModels('github-copilot')):

/v1/messages        -> anthropic-messages
else /responses     -> openai-responses
else /chat/completions -> openai-completions

Picking the first match in listing order instead would silently drop thinking blocks and cache control on Anthropic models.

2. off must not be offered on openai-responses. Copilot rejects such a turn outright:

HTTP 400 {"code":"invalid-argument",
          "error":"This model does not support `reasoning_effort` value `none`."}

observed on grok-4.6. This agrees with pi-ai's curated catalog, where all 14 openai-responses Copilot entries pin thinkingLevelMap.off to null, while the Anthropic and Completions entries leave it open. The listing itself never mentions off among reasoning_effort, so this cannot be derived from the payload — it has to be encoded as a rule.

What a working version looks like

The local plugin reuses PiAiAdapter and pi-ai's own github-copilot provider (OAuth exchange, all three protocols, and filterModels entitlement all unchanged), and replaces exactly one member: getModels. Precedence is curated-first — the pi-ai entry wins for every id both describe, because it carries compat quirks and thinking-level spellings the listing cannot express (claude-opus-4.6 carries forceAdaptiveThinking and a max effort the listing does not report). Discovery only contributes ids the catalog has never described, so the route is only ever widened; an unreachable endpoint or a missing credential costs the new ids and nothing else.

Result on my account: 22 models from the catalog alone, 23 with live discovery (grok-4.6 is the addition), and a real streaming turn through that discovered model completes normally.

Two implementation notes that may be useful regardless of what upstream decides:

• The plugin must declare credentials in inject. Reading it through ctx.get('credentials') at mount time races plugin registration; the read falls back to the launch environment, discovery fails, and — because discovery is designed to degrade rather than throw — the route silently keeps serving the static catalog. The symptom is "one model missing", with no error anywhere.
• PiAiAdapterOptions.profiles being a callback memoized by map identity is what makes hot refresh possible at all: swapping in a new map is enough for the next call to see new models, with no restart. That seam is doing more work than its docs suggest.

Suggestion

Either relax the catalog short-circuit for providers whose listing is rich enough (Copilot is the clear case, and it could be a provider-level capability flag rather than a special case), or document the merge rules — curated-first precedence, protocol derivation, the off-on-Responses restriction — so that plugins can do this safely without rediscovering the 400 the hard way.

Environment: Windows 11, source checkout at 0.1.0-rc.5, @earendil-works/pi-ai 0.84.2.

[allenliang2022]

Follow-up: as a third-party plugin this does not build today, for two packaging reasons — neither of them in the plugin.

I tried repackaging the above as a standalone plugin (the shape dsh plugin add installs) rather than a workspace package inside a checkout, so it could be reused without one. Measured 2026-08-19 against @deepseek-ai/dsh-llm-pi-ai@0.1.0-rc.7:

1. Two copies of pi-ai. The published dsh-llm-pi-ai depends on @earendil-works/pi-ai@^0.82.1, and a caret on a 0.x range pins the minor. A plugin that needs 0.84 — which is where the newer Copilot ids live — therefore installs 0.82.1 and 0.84.2 side by side. tsc refuses the mismatch:

Property 'publish' is missing in type 'RefreshModelsContext' (pi-ai 0.82.1)
  but required in type 'RefreshModelsContext' (pi-ai 0.84.2)

That is the compile-time face of a runtime hazard: a Provider built against one copy handed to an adapter that expects the other.

2. PiAiAdapterOptions has no resolveCredentials in the published build. It exposes profiles, resolveApiKey, and resolveAttachments. An auth: oauth route needs the credential seam to reach pi-ai's OAuth store, so the published adapter cannot serve a Copilot route at all — the plugin would have to reimplement message conversion, streaming, cancellation, and error mapping instead of reusing them.

Minor, but it costs an install to discover: the latest dist-tag on these framework packages points at 0.0.1-rc.1, an early build that depends on the unpublished @deepseek-ai/dsh-environment. A dependency range of * resolves there and fails with ERR_PNPM_FETCH_404. Pinning ^0.1.0-rc.7 installs a working set.

I am not asking for anything specific — recording it because the first two are the difference between "a plugin reuses the adapter" and "a plugin reimplements streaming", which seems worth knowing when deciding how much of this belongs upstream. The logic itself is framework-independent and its tests (protocol derivation against the installed catalog, the Responses off restriction, catalog precedence) run with no network and no credentials.

[zoahdev]

对照当前 main HEAD 99f6f02（rc.7）核验，你的两个核心前提在 rc.7 上仍然成立（报告基于 rc.5，未变）：

1. 目录短路（packages/llm/llm-pi-ai/src/discovery.ts）

模块头注释明写："A route the installed pi-ai catalog ships is answered from that catalog, with no network call at all: pi-ai's registry is the authoritative list for its own providers, and it carries the capacities a listing endpoint would not disclose."——目录条目零网络调用，与你的描述一致。同时 LISTABLE_PROTOCOLS 只有 openai-completions/openai-responses：anthropic-messages 协议的 listing 根本不会被询问（注释说"every other protocol reports that it cannot be interrogated"）——所以你说的"协议推导顺序很重要"（anthropic 模型必须走 /v1/messages 才能保住 thinking/cache control）不是实现细节，而是当前代码根本不覆盖的路径。

2. profiles 记忆化接缝（packages/llm/llm-pi-ai/src/adapter.ts）

PiAiAdapter.current()：const profiles = this.config.profiles(); if (this.snapshot?.profiles === profiles) return this.snapshot——确实按 map 身份记忆化（=== 引用比较），换一个新 map 就能让下一次调用看到新模型，无需重启。你的 note 2 准确，而且这条注释（"Resolution memoizes its result, so an unchanged configuration is recognized by identity"）说明这是有意的热刷新接缝，比文档暗示的更强。

3. 对两条建议的倾向

• provider 级能力旗标（"listing-rich provider"）是对的形态，但注意 discovery.ts 现在的协议白名单是硬编码集合，做成旗标需要先解决"listing 规范化"（Copilot 的 capabilities.limits 结构和 OpenAI 兼容网关的 context_window 是两套字段，ListingEntry 现在只读 OpenAI 形态）——这是比旗标更重的改动；
• 最小可行是文档化合并规则（curated-first、协议推导顺序、off-on-Responses 限制）——零代码、立刻能让第三方插件安全实现。你的本地插件已经证明了可行性，把三条规则写进 llm-pi-ai 的 README/discovery 注释就能闭环。

4. 一个可落地的加固（把 400 变成配置期错误）

off-on-Responses 规则除了文档，还可以在 pi-ai 侧编码为发送前校验：openai-responses 路由上若请求含 reasoning_effort: off，直接抛配置错误而不是把请求发出去吃 400。你的实测（grok-4.6 400）说明这能省掉用户一次"为什么失败"的排查。规则来源就是你发现的目录事实（14 个 responses 条目全部 pin null）——这正是"listing 无法表达、必须编码"的典型。

5. 关于 follow-up 的打包问题

两副本 pi-ai 与 #2660/#3033 的 dsh-tools 双实例同族（Symbol/单例分裂）。建议：插件声明 @earendil-works/pi-ai 为 peerDependency（host 0.82.1 共享），或明确"需要 0.84 的 Copilot ids 时单独装且接受双实例风险"——别让 plugin add 的 peer-guard 以为装好了其实跑的是两份。这条值得在插件 README 里写清楚。

[allenliang2022]

Thanks — this is a much better-grounded reply than my post deserved, and I owe you an apology for the ordering: my "the code is public now" comment went up a day and a half after yours without answering it. I had not seen it.

Checked every point against the tree. Three confirmations, one refinement, and one correction to something I wrote above.

1. LISTABLE_PROTOCOLS — confirmed, and it makes the case narrower than I stated

discovery.ts does hold exactly {openai-completions, openai-responses}, with the reason stated in the comment: guessing at the others would "report an authentication failure as a provider with no models".

One thing worth adding, because it changes what "lift the short-circuit" would have to mean. That whitelist is a route-level gate, and api is a route-level setting — but Copilot's listing describes models across three protocols under one route. So even if the catalog short-circuit were lifted for Copilot, a route declared openai-completions would pass the gate and then stamp every discovered model with that one protocol, silently downgrading the Anthropic ones. The per-model derivation is not an optimization on top of the existing path; it is the thing that makes a Copilot listing usable at all. Your framing ("a path the current code does not cover") is right, and the reason is per-model protocol rather than the whitelist itself.

2. profiles memoization — confirmed

current() compares by identity and returns the cached snapshot, and the class comment calls it out deliberately. Nothing to add.

3. Capability flag vs. listing normalization — you are right, and the evidence is sharper than I had it

ListingEntry reads flat fields: context_window / context_length for capacity and max_output_tokens / max_tokens for the cap. Copilot reports neither: it nests them as capabilities.limits.max_context_window_tokens and capabilities.limits.max_output_tokens, and puts reasoning and vision under capabilities.supports. So a provider-level flag would flip a switch onto a reader that cannot parse the payload. The normalization is the work; the flag is the cheap part. Agreed that documenting the merge rules is the minimal viable step.

4. Pre-send validation for off-on-Responses — mostly already expressible, with one real gap

Worth knowing before designing this: the declarative form already exists and is safe by default. In resolveModelReasoning, a declared reasoningEfforts dict pins every undeclared level to null, precisely because pi-ai's own defaulting is asymmetric (absent means supported for the five base levels but unsupported for xhigh/max). So a settings-declared Responses model that simply omits off already refuses it. A model with no reasoningEfforts and no installed entry lands on reasoning: false and sends no effort at all. Both are safe.

The gap is narrower than "add a pre-send check", and I think it sits one layer earlier than you proposed:

• A profile author who explicitly declares off on an openai-responses model gets a map where off is absent, which the comment documents as "supported, send nothing" — correct for most providers, a guaranteed 400 on Copilot's Responses endpoint. This is self-contained misconfiguration, and this repo's own convention is that such a thing "fails loud at load". resolveModelReasoning already has the provider, the entry, and the protocol, so it can reject it there with a message naming the model — strictly better than a 400 at turn time, and it needs no request path.
• The case that actually bit me has no configuration to validate: a discovered model, where code decides the map. There the rule cannot be a validator at all; it has to be encoded wherever the model is built. That is the half that only documentation can transfer to a plugin author, which is your point 3 again.

5. Correction to my own follow-up above — I understated the packaging blocker

I wrote that the published PiAiAdapterOptions "has no resolveCredentials". That is true but it implies a missing option. Checking the published artifact instead of the type signature:

@deepseek-ai/dsh-llm-pi-ai@0.1.0-rc.7 ships no OAuth store at all — no such module in the package. Its catalog.d.ts says as much: "pi-ai's other method, OAuth, resolves from a stored OAuth credential alone: resolveProviderAuth has no ambient path for it, nothing here runs a login flow."

In the tree I have been working from, that seam (oauth-store.ts, the profile's auth field, the exchange-and-persist path) is local fork work, not upstream — git log --diff-filter=A attributes the file to a commit that is not on the master I have. So the accurate statement is: a third-party plugin cannot reuse the adapter for a Copilot route because the capability it would reuse is not published, not because one option is absent. My branches are stacked on that local OAuth work, which readers of the earlier comment could not have known — my omission.

Caveat on my own evidence: my clone is shallow and its master predates the 99f6f02 you checked, so I can only assert this about the published npm artifact (which is what a plugin author actually installs) and about my stale master. If rc.7's tree does carry an OAuth path that simply is not in the published build, I would want to know.

6. On declaring pi-ai as a peerDependency

Measured: a plugin-side peer declaration does not dedupe here. The published dsh-llm-pi-ai declares @earendil-works/pi-ai as a regular dependency at ^0.82.1, so pnpm nests its own copy no matter what the plugin asks for — my install had 0.82.1 and 0.84.2 side by side while the plugin declared the peer. Your warning about the peer-guard reporting a satisfied install is exactly the failure mode.

Following your suggestion through, though, I think it dissolves the problem rather than trading against it — and it corrects an assumption of mine. The plugin does not actually need pi-ai 0.84. Its models come from the listing, so an older curated catalog simply means discovery contributes more ids, not fewer; 0.84 matters to the host's catalog freshness, not to the plugin's logic. If that holds, the honest declaration is a peer of >=0.82.1, the plugin shares the host's single copy, and the dual-instance hazard disappears. I have not tested it against 0.82.1 — the imports it relies on (builtinProviders, getBuiltinModels, the Model/Provider types) look present in that line, but "looks present" is not a test result, and the OAuth blocker in §5 means I cannot run the third-party shape end to end yet regardless.

[allenliang2022]

The code is public now in case it is easier to read than prose. Two stacked branches on my fork — not a PR, since CONTRIBUTING.md says external ones cannot be accepted, just a reference:

• fix/pi-ai-0.84-upgrade — the dependency bump plus the three things 0.84 changes: the baseten thinking-format gate, the two new StopReason members, and the abort classification. The last one is the interesting commit: both pre-existing abort tests pass again unchanged, which is what convinced me the fix was aimed at the right layer.
• feat/llm-copilot-live — the plugin, stacked on the above. 15 unit tests, no network and no credentials required.

The test worth pointing at is tests/copilot.spec.ts: the protocol-derivation case asserts against getBuiltinModels('github-copilot') rather than a hand-written fixture, so if that ordering rule ever stops reproducing the curated catalog's api choice, it fails loudly instead of drifting.

[weijiafu14]

Consumer-side datapoint for this proposal, from the one place that reads these capacities downstream of the seam you're changing:

We maintain pi2dsh (runs Pi plugins on DSH; plugins read the model directory through our projection), and our examples regression caught exactly the split your proposal touches: DSH separates "directory membership" (listModels) from "per-route capacity" (resolveModelInfo), and a consumer that only reads the first silently loses contextWindow — Pi plugins then fall back to guessed defaults (Pi's own compaction literally does model.contextWindow || 128000, which for a 1M-window model compacts an order of magnitude early). We had shipped that bug ourselves: the resolve seam existed, was tested, and was wired into nothing until a regression run exposed it.

Why that matters for this thread: the short-circuit premise ("no listing endpoint reports capacities") being false for Copilot means the resolve half is exactly where live /models capacities belong — membership can stay catalog-backed while resolveModelInfo consults the live listing when the provider provably reports capacities. That shape keeps the no-network-call default for providers where the premise holds, and matches how downstream consumers actually read capacities (per-route resolution, not the listing).

No opinion on the stacked branches' internals — the above is evidence that the capacity-resolution seam has real downstream consumers whose failure mode is silent-wrong, so whichever fix lands, a regression asserting "declared/live capacity actually reaches a route consumer" is worth including.