Skip to content

[Feature]: Tech debt enhance robustness in model family detection #3206

@scholarsmate

Description

@scholarsmate

1. Why Do You NEED This Feature?

Tech debt: replace substring model-family detection with catalog metadata

Problem

crates/forge_app/src/transformers/model_specific_reasoning.rs::family() classifies Anthropic model families by substring-matching the model id:

  if id.contains("opus-4-7") {
      AnthropicModelFamily::AdaptiveOnly
  } else if id.contains("opus-4-6") || id.contains("sonnet-4-6") {
      AnthropicModelFamily::AdaptiveFriendly
  } ...

This breaks for every Anthropic-compatible proxy that re-skins the canonical id.

Concrete example: AskSage exposes Opus 4.7 as google-claude-47-opus (no dashes between the digits, vendor token in front). The substring opus-4-7 is not present, so the model falls through to LegacyNoEffort, gets set_default_legacy_budget injected, and forge ships thinking.type=enabled + budget_tokens=10000, which Anthropic 4.7 rejects with:

| thinking.type.enabled is not supported for this model. Use thinking.type.adaptive and output_config.effort to control thinking behavior.

The same brittleness appears in crates/forge_repo/src/provider/anthropic.rs::interleaved_thinking_required(), which uses the same substring approach and would mis-tag the same ids.

PRs #3193 / #3194 add the missing variants (47-opus, 46-opus, etc.) as a tactical fix, but every future re-skinned id will need another patch in the same place.

2. What Is NOT Possible Right Now?

Because provider.json does not support this schema yet and the code is not in place to prefer that over the substring matching method.

3. What WILL Be Possible With This Feature?

More robust support for family handling.

Proposed Solution (User Experience)

crates/forge_repo/src/provider/provider.json already enumerates every model forge supports per provider. Add a family field (or a capabilities: {...} object) to each entry, like this:

  {
    "id": "google-claude-47-opus",
    "name": "Claude Opus 4.7",
    "family": "opus-4-7",
    ...
  }

Then family() becomes a straightforward catalog lookup:

  fn family(&self) -> AnthropicModelFamily {
      self.catalog
          .find(&self.model_id)
          .and_then(|m| m.family)
          .map(AnthropicModelFamily::from)
          .unwrap_or_else(|| Self::infer_from_id(&self.model_id))  // existing substring fallback
  }

The substring matcher stays as a fallback for unknown ids, so behavior is unchanged for anything not in the catalog.

Why this approach over the alternatives

  • Regex / id normalization. Tightens the substring check but still encodes provider-specific naming in code. Every new naming convention is another regex.
  • Structured id parser. Solves naming variance but assumes ids carry parseable version info. Many gov / enterprise re-skins are opaque tokens.
  • Capabilities flags directly on the model entry (accepts_legacy_thinking, accepts_adaptive_thinking, etc.). Strictly cleaner but requires defining a capabilities schema and back-filling every entry. Worth doing eventually, but a bigger lift than adding one family field.
  • Runtime /models discovery. Most Anthropic-compatible proxies don't return capability metadata.

The catalog-field approach reuses an existing source of truth, ships in a single PR, and removes the recurring patch-the-detector tax.

Scope:

  • Add optional family field to the model schema in crates/forge_repo/src/provider/provider.json.
  • Backfill canonical Anthropic / Bedrock / Vertex entries.
  • Have ModelSpecificReasoning::family() and interleaved_thinking_required() consult the catalog first, then fall back to substring matching for unknown ids.
  • Update tests in model_specific_reasoning.rs to cover the catalog path and the fallback.

No breaking change for existing users — the substring detector remains as a safety net.

Alternatives Considered

No response

Feature Category

AI/LLM Integration

Priority/Impact

High - Would significantly improve my workflow

Examples from Other Tools

No response

Additional Context

No response

Pre-submission Checklist

  • I have searched existing issues and confirmed this is not a duplicate
  • I am willing to submit a PR to implement this feature

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: featureBrand new functionality, features, pages, workflows, endpoints, etc.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions