llm-pi-ai: per-model default thinking level, for catalogs where only some models think

[daveycodez]

Summary

llm-pi-ai can't express "think by default, but only where thinking exists". A route's default thinking level is route-wide (profile.reasoning) and there's no per-model equivalent, so a catalog whose models disagree about thinking has no correct configuration.

This bites any provider serving a mixed set of local models — some with a thinking branch in their chat template, some without. Concretely: an oMLX server with 12 models, 6 of which think.

The trap

Two functions close it:

// adapter.ts
const reasoning = resolveReasoningLevel(model, options.reasoningEffort ?? profile.reasoning)

function resolveReasoningLevel(model, effort) {
  if (effort === undefined) return undefined            // -> thinking disabled
  if (getSupportedThinkingLevels(model).some(l => l === effort)) return effort
  throw new LlmError(..., 'UNSUPPORTED_REASONING_EFFORT')
}

// pi-ai
function getSupportedThinkingLevels(model) {
  if (!model.reasoning) return ['off']    // a model declaring no efforts offers only "off"
  ...
}

An unsupported level is refused, not ignored. So on one route:

profile.reasoning	thinking models	non-thinking models
'high'	✅ think	❌ every turn throws UNSUPPORTED_REASONING_EFFORT
'off'	❌ never think	✅ fine
unset	❌ Default means thinking off	✅ fine

The third row is the surprising one: with no default, resolveReasoningLevel returns undefined, and the dispatch turns that into thinking: { type: 'disabled' } (anthropic) or enable_thinking: false (qwen-chat-template). A reasoning model picked from the composer silently does not reason until the user opens the effort menu and chooses a level.

It's protocol-independent — resolveReasoningLevel runs in streamWithSnapshot before dispatch, so openai-completions and anthropic-messages behave identically here.

There's no narrower scope to retreat to: modelFields is name, contextWindow, maxTokens, input, reasoningEfforts, compat, and modelOverrides is z.object(modelFields).

Workaround, and why it isn't good

Declare reasoningEfforts on every model so a route-level high becomes legal everywhere. It works, and it's what I'm shipping today — but it puts an effort menu on six models that ignore it entirely. A control that does nothing is its own bug.

Proposal

A per-model defaultReasoning, consulted only when a request names no level:

options.reasoningEffort ?? profile.configuredReasoning.get(options.model) ?? profile.reasoning

It follows the existing configuredMaxTokens idiom exactly — a deployment's choice about requests, kept beside the materialized model rather than on pi-ai's Model, for the same reason that one is:

• catalog.ts — defaultReasoning?: ModelThinkingLevel on PiAiModelProfile; a configuredReasoning map built alongside configuredMaxTokens; validated at load so a level a model doesn't offer names the model instead of failing the user's next turn
• config.ts — schema field, carried onto the profile
• adapter.ts — the one-line resolution change above

Deliberately not surfaced through describableReasoningLevel. Leaving the route's advertised default undefined keeps the picker's "Default" row, and selecting it resolves through the new field — so "Default" comes to mean this model's own default rather than no thinking, which is what a user reading the word expects.

Purely additive: a config that names no defaultReasoning behaves exactly as it does now.

Result

Same route, same picker entry:

• thinking models → Default thinks, full effort menu
• models with no thinking branch → no effort control, no level sent, nothing thrown

Verification

Applied against master and run locally:

• packages/llm/llm-pi-ai — 266 tests pass (11 files), including two added to catalog.spec.ts's "per-model reasoning efforts" block covering the recorded default and the load-time refusal
• Also verified end-to-end by patching the shipped dsh-llm-pi-ai in an installed DSH Desktop and pointing it at a live oMLX server: MiniMax and Qwen2.5-Coder answer with no effort control, Qwen3.8 thinks on Default, Off disables

Diff (+80 / -3) is below. CONTRIBUTING.md says external PRs aren't being accepted right now, so I'm posting it here rather than opening one — happy to move it wherever is most useful, or to just leave the analysis if the shape isn't what you'd want.

Diff

diff --git a/packages/llm/llm-pi-ai/src/adapter.ts b/packages/llm/llm-pi-ai/src/adapter.ts
index f06b27a..24708cf 100644
--- a/packages/llm/llm-pi-ai/src/adapter.ts
+++ b/packages/llm/llm-pi-ai/src/adapter.ts
@@ -335,7 +335,11 @@ export class PiAiAdapter extends LlmAdapter {
     const model = this.modelOf(snapshot, options.provider, options.model)
     const reasoning = resolveReasoningLevel(
       model,
-      options.reasoningEffort ?? profile.reasoning,
+      // The model's own default outranks the route's: a route-wide level
+      // cannot describe a catalog whose models disagree about thinking.
+      options.reasoningEffort
+        ?? profile.configuredReasoning.get(options.model)
+        ?? profile.reasoning,
     )
     const apiKey = await this.config.resolveApiKey(options.provider, profile)
 
diff --git a/packages/llm/llm-pi-ai/src/catalog.ts b/packages/llm/llm-pi-ai/src/catalog.ts
index 1e9b126..a06be73 100644
--- a/packages/llm/llm-pi-ai/src/catalog.ts
+++ b/packages/llm/llm-pi-ai/src/catalog.ts
@@ -12,6 +12,7 @@
  * @module dsh-llm-pi-ai/catalog
  */
 
+import { getSupportedThinkingLevels } from '@earendil-works/pi-ai'
 import { builtinProviders, getBuiltinModels, getBuiltinProviders } from '@earendil-works/pi-ai/providers/all'
 import type { BuiltinProvider } from '@earendil-works/pi-ai/providers/all'
 import type {
@@ -545,6 +546,18 @@ export interface PiAiModelProfile {
    * default on its own.
    */
   maxTokens?: number
+  /**
+   * The thinking level a request that names none resolves to for this model.
+   *
+   * The route's `reasoning` answers the same question, but route-wide, and
+   * `resolveReasoningLevel` refuses a level a model does not offer rather than
+   * ignoring it. A catalog whose models disagree — some with a thinking
+   * branch, some without — therefore has no route-wide answer: naming a
+   * thinking level fails every model that cannot think, and naming none leaves
+   * the models that can defaulting to not thinking. This is the per-model
+   * answer, and it is only consulted when the request names no level.
+   */
+  defaultReasoning?: ModelThinkingLevel
   /**
    * Request modalities this model accepts. Absent — or empty, which describes
    * a model that accepts nothing and so states no answer either — keeps the
@@ -769,6 +782,14 @@ export interface RouteCatalog {
    * picked, so only an explicit configuration lands here.
    */
   configuredMaxTokens: ReadonlyMap<string, number>
+  /**
+   * Per-model default thinking levels this route configured, by model id.
+   *
+   * Separate from the materialized model for the same reason as
+   * {@link configuredMaxTokens}: it is a deployment's choice about requests,
+   * not a capability of the model, and pi-ai's `Model` has no field for it.
+   */
+  configuredReasoning: ReadonlyMap<string, ModelThinkingLevel>
 }
 
 /**
@@ -830,6 +851,7 @@ export function resolveRouteModels(request: RouteCatalogRequest): RouteCatalog {
   }
   const seen = new Set<string>()
   const configuredMaxTokens = new Map<string, number>()
+  const configuredReasoning = new Map<string, ModelThinkingLevel>()
   const models = entries.map((entry) => {
     if (entry.id.length === 0) invalid(provider, 'has a model with an empty id')
     if (seen.has(entry.id)) invalid(provider, `lists model "${entry.id}" more than once`)
@@ -859,6 +881,18 @@ export function resolveRouteModels(request: RouteCatalogRequest): RouteCatalog {
     // Only a value the profile named is a deployment choice; the catalog's is
     // the model's capability and stays out of request defaults.
     if (entry.maxTokens !== undefined) configuredMaxTokens.set(entry.id, entry.maxTokens)
+    const modelReasoning = resolveModelReasoning(provider, entry, base)
+    if (entry.defaultReasoning !== undefined) {
+      // Refused here rather than at request time: a level this model does not
+      // offer is a configuration mistake, and it should name the model while
+      // the route is being read, not fail the user's next turn.
+      const offered = getSupportedThinkingLevels({ ...base, ...modelReasoning } as Model<Api>)
+      if (!offered.some(level => level === entry.defaultReasoning)) {
+        invalid(provider, `model "${entry.id}" defaultReasoning "${entry.defaultReasoning}" is not a level it offers;`
+          + ` declare it in reasoningEfforts, or drop the default`)
+      }
+      configuredReasoning.set(entry.id, entry.defaultReasoning)
+    }
     return {
       // The installed entry lays the floor, and the fields below override it.
       // Enumerating instead would silently drop every `Model` field this
@@ -875,7 +909,7 @@ export function resolveRouteModels(request: RouteCatalogRequest): RouteCatalog {
       cost: base?.cost ?? NO_COST,
       contextWindow,
       maxTokens,
-      ...resolveModelReasoning(provider, entry, base),
+      ...modelReasoning,
       ...resolveModelCompat(provider, entry, request.compat, base, api),
     }
   })
@@ -889,5 +923,5 @@ export function resolveRouteModels(request: RouteCatalogRequest): RouteCatalog {
     invalid(provider, `sets compat "${field}", but no model on the route speaks a protocol that takes it;`
       + ` it exists on ${takers.join(', ')}`)
   }
-  return { models, configuredMaxTokens }
+  return { models, configuredMaxTokens, configuredReasoning }
 }
diff --git a/packages/llm/llm-pi-ai/src/config.ts b/packages/llm/llm-pi-ai/src/config.ts
index 5fdcc2c..ca8cfeb 100644
--- a/packages/llm/llm-pi-ai/src/config.ts
+++ b/packages/llm/llm-pi-ai/src/config.ts
@@ -207,6 +207,11 @@ export interface ResolvedPiAiProviderProfile
    * own, so a catalog capability must not appear here.
    */
   configuredMaxTokens: ReadonlyMap<string, number>
+  /**
+   * Per-model default thinking levels, by model id. Consulted only when a
+   * request names no level of its own, ahead of the route's `reasoning`.
+   */
+  configuredReasoning: ReadonlyMap<string, ModelThinkingLevel>
 }
 
 /** Plugin configuration: the provider routes this instance owns. */
@@ -293,6 +298,10 @@ const modelFields = {
   // `{}`, and absent must stay distinguishable — it means "inherit the
   // installed catalog's capability", while `false` disables reasoning.
   reasoningEfforts: z.union([z.const(false), reasoningEfforts]),
+  // Per-model, because the route's `reasoning` cannot serve a mixed catalog:
+  // it applies to every model, and a level a model does not offer is refused
+  // rather than ignored.
+  defaultReasoning: z.union(THINKING_LEVELS),
   compat: compatProfile,
 }
 
@@ -451,6 +460,7 @@ export function resolveProfiles(
       ...rest.headers === undefined ? {} : { headers: { ...rest.headers } },
       ...rest.thinkingBudgets === undefined ? {} : { thinkingBudgets: { ...rest.thinkingBudgets } },
       configuredMaxTokens: catalog.configuredMaxTokens,
+      configuredReasoning: catalog.configuredReasoning,
       piProvider: buildProvider({
         provider,
         displayName,
diff --git a/packages/llm/llm-pi-ai/tests/catalog.spec.ts b/packages/llm/llm-pi-ai/tests/catalog.spec.ts
index 99886cd..9d5fb0a 100644
--- a/packages/llm/llm-pi-ai/tests/catalog.spec.ts
+++ b/packages/llm/llm-pi-ai/tests/catalog.spec.ts
@@ -600,6 +600,35 @@ describe('per-model reasoning efforts', () => {
     return model
   }
 
+  it('records a per-model default level, and only for the model that names one', () => {
+    // A route-wide `reasoning` cannot describe a catalog whose models disagree
+    // about thinking: it applies to every model, and `resolveReasoningLevel`
+    // refuses a level a model does not offer rather than ignoring it. So the
+    // default is per model, and a model that declares none carries none.
+    const profile = resolveProfiles(declared([
+      { id: 'thinker', reasoningEfforts: { off: null, high: 'high' }, defaultReasoning: 'high' },
+      { id: 'plain' },
+    ])).get('acme-gateway')
+
+    expect(profile?.configuredReasoning.get('thinker')).toBe('high')
+    expect(profile?.configuredReasoning.get('plain')).toBeUndefined()
+  })
+
+  it('refuses a per-model default the model does not offer, naming the model', () => {
+    // At load rather than at request time: this is a configuration mistake,
+    // and it should surface while the route is being read instead of failing
+    // the user's next turn.
+    expect(() => resolveProfiles(declared([
+      { id: 'thinker', reasoningEfforts: { off: null, low: 'low' }, defaultReasoning: 'high' },
+    ]))).toThrow(/model "thinker" defaultReasoning "high" is not a level it offers/)
+
+    // A model with no thinking branch offers only `off`, so any thinking level
+    // is refused for it too.
+    expect(() => resolveProfiles(declared([
+      { id: 'plain', defaultReasoning: 'high' },
+    ]))).toThrow(/defaultReasoning "high" is not a level it offers/)
+  })
+
   it('declares selectable levels with their wire spellings on a hand-declared model', () => {
     const model = modelOf(declared([{
       id: 'acme-think',

[wyouwd1]

Confirmed on a real commercial endpoint too (Volcengine Ark, api/coding/v3): same throw site, same failure. Three additions to the analysis above.

1. The trap is wider than the table — an explicit non-reasoning declaration doesn't escape either

reasoningEfforts: false (the deployment explicitly declaring the model cannot think) still fails every request on a route with a reasoning default. Runnable repro — drop into packages/llm/llm-pi-ai/tests/:

import { afterEach, describe, expect, it, vi } from 'vitest'
import { Context } from '@deepseek-ai/cordis'
import LlmRuntime, { createUserMessage } from '@deepseek-ai/dsh-llm'
import * as LlmPiAi from '@deepseek-ai/dsh-llm-pi-ai'
import { assemble } from './assemble.ts'
import { closeMockServers, mockServer, textEvents } from './mock-server.ts'

afterEach(async () => { vi.unstubAllEnvs(); await closeMockServers() })

const providers = (models: LlmPiAi.PiAiModelProfile[]): Record<string, LlmPiAi.PiAiProviderProfile> => ({
  doubao: {
    apiKeyEnv: 'PI_TEST_KEY', api: 'openai-completions', reasoning: 'max',
    models,
  },
})

describe('route reasoning default vs non-reasoning models', () => {
  const thinking = { id: 'thinker', contextWindow: 1000, maxTokens: 1000,
    reasoningEfforts: { off: null, low: 'low', high: 'high', max: 'max' } }

  it('fails every request on a model with no reasoningEfforts', async () => {
    const server = await mockServer([{ events: textEvents }])
    vi.stubEnv('PI_TEST_KEY', 'test-key')
    const ctx = new Context()
    await ctx.plugin(LlmRuntime)
    await ctx.plugin(LlmPiAi, { providers: providers([thinking, { id: 'flash', contextWindow: 1000, maxTokens: 1000 }]) })
    const result = await assemble(ctx, { provider: 'doubao', model: 'flash',
      messages: [createUserMessage({ content: [{ type: 'text', text: 'hi' }], source: { kind: 'plugin', plugin: 'test' } })] })
    expect(result.finish).toMatchObject({ kind: 'error', failure: { code: 'UNSUPPORTED_REASONING_EFFORT' } })
    expect(server.requests).toHaveLength(0)
  })

  it('fails identically even with an explicit reasoningEfforts: false', async () => {
    const server = await mockServer([{ events: textEvents }])
    vi.stubEnv('PI_TEST_KEY', 'test-key')
    const ctx = new Context()
    await ctx.plugin(LlmRuntime)
    await ctx.plugin(LlmPiAi, { providers: providers([{ id: 'flash', contextWindow: 1000, maxTokens: 1000, reasoningEfforts: false }]) })
    const result = await assemble(ctx, { provider: 'doubao', model: 'flash',
      messages: [createUserMessage({ content: [{ type: 'text', text: 'hi' }], source: { kind: 'plugin', plugin: 'test' } })] })
    expect(result.finish).toMatchObject({ kind: 'error', failure: { code: 'UNSUPPORTED_REASONING_EFFORT' } })
  })
})

2. A smaller complementary fix: capability-gate the route default itself

defaultReasoning fixes deployments that configure it; it leaves every existing mixed-route deployment broken until the operator touches each model. A zero-config guard in the same spot also closes the trap:

// adapter.ts, streamWithSnapshot
const reasoning = options.reasoningEffort === undefined && !model.reasoning
  ? undefined
  : resolveReasoningLevel(model, options.reasoningEffort ?? profile.reasoning)

Semantics: the route default binds only models that can reason (a non-reasoning model omits the reasoning option — the wire request is exactly what describing the model already advertises); an explicit reasoningEffort is still refused strictly before I/O; a reasoning model with an unsupported route default is still refused (the existing adapter.spec.ts contract is untouched).

Verified locally: regression spec 4/4, adapter.spec.ts 50/50, llm-pi-ai + llm suites 516/516. This composes with the defaultReasoning proposal — the guard fixes the zero-config default, defaultReasoning remains the knob for per-model defaults.

3. Provider intel: Volcengine Ark serves glm-5.3-flash thinking-always, with a working effort dial

Measured against https://ark.cn-beijing.volces.com/api/coding/v3/chat/completions (same short request, stream: false):

request	completion_tokens	notes
no reasoning_effort (default)	290–321	defaults to the max tier (most expensive), reasoning_content ~1000 chars
reasoning_effort: "low"	51–67 (stable ×5)	~83% cheaper
"medium" / "high" / "max" / "xhigh"	accepted	all 200, no silent fallback
"off"	400	value 'off' is invalid
thinking: {"type":"disabled"}	400	thinking.type 'disabled' is not supported by this model

So on Ark the model is thinking-always with a live top-level reasoning_effort dial (the public parameter table only covers Managed Agents). A mixed route with reasoning: max is a legitimate shape there too — and the request-path refusal plus the missing picker (no reasoning metadata → no effort menu) is what makes the non-reasoning member of that route unusable in practice.

[jianzhangg]

Confirmed from a third deployment, plus one undocumented escape hatch that may be worth folding into the proposal.

The deployment

A single local multi-vendor gateway (openai-completions, one baseURL, one key, ~10 models). The models' native effort vocabularies genuinely disagree:

models	native levels	top level
deepseek-v4.1-flash, glm-5.3-flash, glm-5.3, claude-opus-4-6-thinking	off … max	max
qwen3.8-max, qwen-flash	off, xhigh	xhigh
gemini-3.8-flash-high, step-5-preview	off, high	high

Same trap, same throw site. Two notes from living with it:

The OP's third row is the one that bites silently. With profile.reasoning unset, the composer reads Default after every model switch, and for a model that does reason, Default omits the effort — so it quietly does not think. Nothing errors; the answer is just worse. That symptom is what sent me looking, and it is much harder to notice than the UNSUPPORTED_REASONING_EFFORT row.

Splitting the route works but is the wrong shape. One gateway becomes N routes differing only in reasoning. displayName is the only field that can tell them apart in the picker, so you end up with gw, gw (claude), gw (gemini) — three entries for one endpoint. It also changes provider in the session log for models that did not change, which is noise for anything reading those logs.

An undocumented escape hatch

reasoningEfforts keys are DSH level names; the values are wire spellings (the README's max: ultra example). So a model can accept a route-wide default while still sending its own vocabulary:

gw:
  reasoning: max          # route default
  models:
    - id: qwen3.8-max
      reasoningEfforts:
        max: xhigh        # DSH "Max" -> wire "xhigh"
    - id: deepseek-v4.1-flash
      reasoningEfforts:
        max: max

Now reasoning: max is legal on every model, the route-wide default resolves, nothing throws, and each model still receives its correct wire value. I verified the mechanism in resolveModelReasoning / getSupportedThinkingLevels, and that the gateway accepts both reasoning_effort: xhigh and reasoning_effort: max (200, reasoning content returned).

The cost is the label: every model's top level reads Max in the picker, so xhigh and high lose their real names. That is the same "a control that lies" objection the OP raised against the other workaround — just relocated from does nothing to is misnamed. So it is a workaround, not a fix, and it does not remove the need for defaultReasoning. Still worth documenting, since today it is the only way to keep a mixed gateway as a single route.

+1 on the proposal

configuredMaxTokens is the right precedent, and validating at load — so a bad level names the model instead of failing the next turn — is what makes it usable. Two suggestions for the sketch:

• modelOverrides is z.object(modelFields), so if defaultReasoning lands on modelFields it comes free for catalog-shipped models too. Worth stating, since that is the cheapest path for anyone who does not want to re-declare a whole models list.
• An unset per-model value should inherit the route's default rather than mean "no thinking", so the field stays a pure narrowing override and cannot regress existing configs.