feat(ts-sdk): add modelOverride option to all evaluators#34
feat(ts-sdk): add modelOverride option to all evaluators#34adnanrhussain merged 7 commits intomainfrom
Conversation
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Pull request overview
This PR extends the TypeScript SDK evaluators to support a modelOverride (provider + model) so callers can swap the underlying LLM without changing evaluator logic, while also adding Anthropic as a supported provider and improving provider/model labeling in metadata and telemetry.
Changes:
- Add
Providerenum +ModelOverridetoBaseEvaluatorConfig, including key-validation rules that prioritize the override provider. - Refactor provider/model tracking to use
LLMProvider.label(computed at provider construction) and remove per-evaluator label helpers. - Update telemetry schema/docs to include
model_override, and update unit tests/docs to cover override behavior and new metadata fields.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| sdks/typescript/src/evaluators/base.ts | Adds Provider, ModelOverride, override-aware key validation, telemetry flagging, and createConfiguredProvider helper |
| sdks/typescript/src/providers/base.ts | Extends LLMProvider interface with a canonical label |
| sdks/typescript/src/providers/ai-sdk-provider.ts | Implements label for VercelAIProvider at construction time |
| sdks/typescript/src/evaluators/vocabulary.ts | Uses createConfiguredProvider, switches metadata to defaultProviders, and derives model/provider labels from provider instances |
| sdks/typescript/src/evaluators/sentence-structure.ts | Uses a single shared provider instance + label for telemetry/metadata |
| sdks/typescript/src/evaluators/smk.ts | Uses createConfiguredProvider and label for telemetry/metadata |
| sdks/typescript/src/evaluators/conventionality.ts | Uses createConfiguredProvider and label for telemetry/metadata |
| sdks/typescript/src/evaluators/grade-level-appropriateness.ts | Uses createConfiguredProvider and label for telemetry/metadata |
| sdks/typescript/src/evaluators/text-complexity.ts | Updates metadata to defaultProviders and ensures telemetry provider reflects override when active |
| sdks/typescript/src/evaluators/index.ts | Re-exports Provider and ModelOverride |
| sdks/typescript/src/index.ts | Exposes Provider and ModelOverride in the public SDK entrypoint |
| sdks/typescript/src/telemetry/types.ts | Adds model_override?: boolean to telemetry event type |
| sdks/typescript/docs/telemetry.md | Documents the new model_override telemetry field |
| sdks/typescript/README.md | Documents modelOverride, adds example usage, and updates config/options docs |
| sdks/typescript/tests/unit/evaluators/validation.test.ts | Adds coverage for override key validation and provider factory params |
| sdks/typescript/tests/unit/evaluators/vocabulary.test.ts | Adds override metadata.model assertions; updates provider mocks to include label |
| sdks/typescript/tests/unit/evaluators/sentence-structure.test.ts | Updates tests for single shared provider and new mock shape |
| sdks/typescript/tests/unit/evaluators/smk.test.ts | Adds override metadata.model assertion; updates provider mocks |
| sdks/typescript/tests/unit/evaluators/text-complexity.test.ts | Updates metadata assertions to defaultProviders and provider mocks |
| sdks/typescript/tests/unit/evaluators/conventionality.test.ts | Updates metadata assertions and provider mocks |
| sdks/typescript/tests/unit/evaluators/grade-level-appropriateness.test.ts | Updates provider mocks to include label |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
czi-fsisenda
left a comment
czi-fsisenda
left a comment
There was a problem hiding this comment.
Looks good!
One non-blocking general comment.
| 'VercelAIProvider does not support custom type. Use config.customProvider directly.' | ||
| ); | ||
| } | ||
| this.label = `${config.type}:${config.model ?? DEFAULT_MODELS[config.type]}`; |
There was a problem hiding this comment.
non-blocking: This change is only for the label, but I'm wondering, in general, if we should have a default model. Given that pricing could be very different for different models, I wonder if we should force users to pick a model themselves
There was a problem hiding this comment.
This is a good callout. Ideally if the user is overriding the model, they need to specify both the provider & model (both valid). We should not silently fallback to a default model since that would hide that from the user.
Currently this was also a fallback in the getModel function
const modelId = requestModel || this.config.model || this.getDefaultModel();
I am removing this to ensure we dont silently fallback and the provider + model configuration (internal or override) are valid
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (config.modelOverride) { | ||
| if (!keyFor[config.modelOverride.provider]) { | ||
| throw new ConfigurationError( | ||
| `${humanName[config.modelOverride.provider]} is required when using modelOverride with provider "${config.modelOverride.provider}". Pass ${configKey[config.modelOverride.provider]} in config.` | ||
| ); |
adnanrhussain
force-pushed
the
ahussain/ts_sdk_override_model
branch
from
1c0b917 to
c190ff4
Compare
adnanrhussain
force-pushed
the
ahussain/ts_sdk_override_model
branch
from
c190ff4 to
857ea7c
Compare
Summary
modelOverride: { provider, model }toBaseEvaluatorConfig, letting callers swap the LLM provider and model for any evaluator without touching evaluation logicProvider.Anthropic,anthropicApiKey)modelOverrideis set, only the API key for the override provider is required; default key validation is bypassedLLMProviderinterface now has areadonly labelproperty computed at construction time, eliminating the per-evaluatormodelLabelfield andgetModelLabelhelpermodel_override: booleanflag when override is active;providerfield reflects actual model used in all evaluators includingTextComplexityEvaluatorTest plan
modelOverrideset — confirm only the override provider's key is required, default keys are notresult.metadata.modelreflects the override provider and model (e.g."anthropic:claude-sonnet-4-6")modelOverrideand a missing override key — confirmConfigurationErroris thrownmodel_override: trueappears in the event andproviderreflects the override model