Intelligent auto purpose

[Edgarus79]

Hi
I'm wondering, what is the real purpose of combo type: Intelligent Auto. Since I have access to a couple of openai/claude accounts, I've created combos that aggregate models that I need to use (for example Sonnet 5 from 3-4 providers). But when combo is in use, other models that are configured are also utilized (deepseek, kiro/mistral) even they're not assigned to combo.

Is that for puprose or I'm using it not the way it should be used?

Best regards
PG

[diegosouzapw]

Hey @Edgarus79! Good question -- there's a distinction between two things that share the "auto" name:

1. The special auto/* model (used outside any named combo) is by design documented to draw from all connected providers -- see docs/routing/AUTO-COMBO.md.
2. A named combo with strategy auto and an explicit model list (the 3-4 Sonnet targets you picked) should stay restricted to that list -- the code only broadens the candidate pool to every active provider's models when the combo has no explicit models[] and no candidatePool set (open-sse/services/combo/autoStrategy.ts).

If your combo shows deepseek/kiro/mistral even though you explicitly assigned only the Sonnet targets, that suggests the combo's models[] list isn't actually populated the way the editor shows -- a possible persistence gap, not intended behavior. Could you paste the combo's config JSON (from the combo editor's export, or GET /api/combos/{id})? That'll tell us for sure whether models is empty server-side -- if so, I'll open a tracking issue.

[Edgarus79]

Simple stuff, a combo which aggregates two other combos:

{
  "name": "fast-v1",
  "models": [
    {
      "id": "fast-v1-ref-1-edek-gpt-5-6-luna",
      "kind": "combo-ref",
      "comboName": "edek/gpt-5.6-luna",
      "weight": 0
    },
    {
      "id": "fast-v1-ref-2-edek-claude-haiku-4-5",
      "kind": "combo-ref",
      "comboName": "edek/claude-haiku-4.5",
      "weight": 0
    }
  ],
  "strategy": "auto",
  "config": {
    "maxRetries": 1,
    "retryDelayMs": 500,
    "concurrencyPerModel": 3,
    "handoffThreshold": 0.85,
    "handoffModel": "",
    "maxMessagesForSummary": 30,
    "nestedComboMode": "execute",
    "trackMetrics": true,
    "reasoningTokenBufferEnabled": true,
    "compressionMode": "standard",
    "zeroLatencyOptimizationsEnabled": false,
    "candidatePool": [],
    "weights": {
      "quota": 0.16,
      "health": 0.2,
      "costInv": 0.16,
      "latencyInv": 0.12,
      "taskFit": 0.08,
      "stability": 0.05,
      "tierPriority": 0.05,
      "tierAffinity": 0.05,
      "specificityMatch": 0.05,
      "contextAffinity": 0.08,
      "resetWindowAffinity": 0
    },
    "modePack": "quality-first",
    "explorationRate": 0.03,
    "routerStrategy": "cost",
    "slaHardConstraints": false,
    "judgeModel": "ds/deepseek-v4-pro"
  },
  "id": "07eb8aa7-d5c3-4b14-9678-301862224938",
  "isHidden": false,
  "sortOrder": 1,
  "createdAt": "2026-07-22T14:06:23.733Z",
  "updatedAt": "2026-07-29T04:27:29.663Z",
  "version": 2,
  "repairNote": "[db-health:2026-07-23T17:30:53.492Z] 1 missing connection pin(s) cleared."
}

Aggregated combos:

[
  {
    "name": "edek/gpt-5.6-luna",
    "models": [
      {
        "id": "edek-gpt-5-6-luna-model-1-codex-gpt-5-6-luna",
        "kind": "model",
        "model": "codex/gpt-5.6-luna",
        "providerId": "codex",
        "weight": 0
      },
      {
        "id": "edek-gpt-5-6-luna-model-2-kiro-gpt-5-6-luna",
        "kind": "model",
        "model": "kiro/gpt-5.6-luna",
        "providerId": "kiro",
        "weight": 0
      }
    ],
    "strategy": "auto",
    "config": {
      "maxRetries": 1,
      "retryDelayMs": 2000,
      "handoffThreshold": 0.85,
      "handoffModel": "",
      "maxMessagesForSummary": 30,
      "trackMetrics": true,
      "reasoningTokenBufferEnabled": true,
      "compressionMode": "standard",
      "zeroLatencyOptimizationsEnabled": false,
      "resetAwareQuotaCacheTtlMs": 0,
      "resetAwareQuotaCacheMaxStaleMs": 0
    },
    "id": "d33c974b-2467-457b-9ca8-931ee2df439a",
    "isHidden": false,
    "sortOrder": 7,
    "createdAt": "2026-07-28T13:07:22.487Z",
    "updatedAt": "2026-07-29T04:44:00.685Z",
    "version": 2
  },
  {
    "name": "edek/claude-haiku-4.5",
    "models": [
      {
        "id": "edek-claude-haiku-4-5-model-1-claude-claude-haiku-4-5-20251001",
        "kind": "model",
        "model": "claude/claude-haiku-4-5-20251001",
        "providerId": "claude",
        "weight": 0
      },
      {
        "id": "edek-claude-haiku-4-5-model-2-kiro-claude-haiku-4-5",
        "kind": "model",
        "model": "kiro/claude-haiku-4.5",
        "providerId": "kiro",
        "weight": 0
      }
    ],
    "strategy": "auto",
    "config": {
      "maxRetries": 1,
      "retryDelayMs": 2000,
      "handoffThreshold": 0.85,
      "handoffModel": "",
      "maxMessagesForSummary": 30,
      "trackMetrics": true,
      "reasoningTokenBufferEnabled": true,
      "compressionMode": "standard",
      "failoverBeforeRetry": true,
      "zeroLatencyOptimizationsEnabled": false,
      "resetAwareQuotaCacheTtlMs": 0,
      "resetAwareQuotaCacheMaxStaleMs": 0
    },
    "id": "20768bb0-a427-4851-a350-976f654aac9a",
    "isHidden": false,
    "sortOrder": 11,
    "createdAt": "2026-07-28T13:12:31.958Z",
    "updatedAt": "2026-07-29T04:44:23.660Z",
    "version": 2
  }
]

So as you can see, there is no other model attached than specified. What bothers me is a "judgeModel": "ds/deepseek-v4-pro" in intelligent auto, but I thinkt that this is leftover after type switching and reconfiguration.

Today I switched fast-v1 and edek/luna and edek/haiku models to intelligent auto, I thought that I screwed sth but when i get into logs I found this:

Every combo that is named like real model, it aggregates the same model from different accounts, so there is no way that it has spark configures. Since fast-v1 uses only luna and haiku, there should be no terra or codex calls.
It seems like this mode aggregates all models from all accounts available during run.

Edit:
As you can see, intelligent auto used gpt-5.6-sol in edek/claude-sonnet-5 :)

[diegosouzapw]

@Edgarus79 -- that JSON is exactly what I needed, and it confirms a real bug, not a misconfiguration on your end. Thank you for the thorough detail.

You're right: fast-v1 references only two sub-combos (edek/gpt-5.6-luna, edek/claude-haiku-4.5), and those sub-combos each list only two concrete models. There is no path in that tree to gpt-5.6-sol, terra, or the stray codex calls in your logs -- so an auto combo with a populated models[] is pulling in models from outside its declared pool.

The intended behavior is documented: a named combo with strategy: "auto" and an explicit model list (whether direct model entries or combo-ref entries) should stay restricted to that list. The guard that enforces it lives in open-sse/services/combo/autoStrategy.ts (expandAutoComboCandidatePool) -- it's meant to short-circuit full-catalog expansion when models[] is populated. The fact that your combo still expands means that guard isn't firing, most likely because the combo object reaching that function doesn't carry the resolved models field at the call site. I'm tracing it now.

I've opened #9032 to track the fix. The plan there: confirm the models field is present when the guard evaluates, thread it through if it's been stripped, and add a regression test so a combo-ref/models[] combo never expands beyond its declared targets.

One side note you already spotted: "judgeModel": "ds/deepseek-v4-pro" in fast-v1's config is a leftover from a previous strategy type -- it's only used by the fusion strategy (the judge that synthesizes the panel's answers), not by auto. Harmless here, but worth clearing so it doesn't mislead a future reader of the config.

Will update here when the patch lands.

[diegosouzapw]

One more update, @Edgarus79. PR #8598, merged on July 27, changed the auto candidate expansion so a non-empty models[] is returned as the explicit pool instead of expanding to the full catalog. The current regression suite covers explicit model entries, combo-ref, empty versus populated pools, and duplicate suppression.

That appears to cover the JSON you posted even though #9032 was opened afterward. Please retry the exact combo on the current v3.8.50 branch or the next release. If it stays inside the Sonnet-only pool, I will close #9032 as already fixed by #8598. If it still escapes, paste one fresh routing trace and we will have a narrower repro.