Custom model objects with multiple keys are silently skipped in resolveModelCollections

[frantj]

Description

When defining a custom model in a blueprint using object syntax with multiple fields (e.g. id, url, modelName, inherit), the model is silently skipped during resolution and a warning is logged instead.

Expected behavior

Custom model objects with an id field should be passed through directly and included in the evaluation.

Root cause

In src/cli/commands/run-config.ts, resolveModelCollections checks for a generic single-key provider object before checking for a custom model object with an id property. Since a custom model has multiple keys, it fails the single-key check and is skipped.

The fix is to check for modelEntry.id first:

} else if (typeof modelEntry === 'object' && modelEntry !== null && !Array.isArray(modelEntry) && modelEntry.id) {
    // Custom model definition — pass through directly
    normalizedConfigModels.push(modelEntry);
} else if (typeof modelEntry === 'object' && modelEntry !== null && !Array.isArray(modelEntry)) {
    // Provider shorthand — expect exactly one key
    ...

Additional context

No error is thrown — the silent skip makes this hard to diagnose without reading the warning carefully. Fix is available in frantj/wevalapp@339c52a.

Steps to Reproduce

Define a custom model in a blueprint:

models:
  - id: my-custom-agent
    url: http://localhost:3001/v1
    modelName: gpt-4o

Run an evaluation. The model is not included — instead you see:

Invalid object entry in models array: Must have exactly one key (the provider). Found 3 keys. Skipping: {"id":"my-custom-agent","url":"...","modelName":"gpt-4o"}

Blueprint ID

No response

Page / URL where issue occurred

No response

Browser

Chrome

Operating System

macOS

Browser Console Logs / Stack Traces

Screenshots

No response