Skip to content

Add modelCapabilities override to all SDK languages#1029

Merged
SteveSandersonMS merged 26 commits intomainfrom
stevesa/override-model-capabilities
Apr 7, 2026
Merged

Add modelCapabilities override to all SDK languages#1029
SteveSandersonMS merged 26 commits intomainfrom
stevesa/override-model-capabilities

Conversation

@SteveSandersonMS
Copy link
Copy Markdown
Contributor

@SteveSandersonMS SteveSandersonMS commented Apr 7, 2026

Summary

Same as #1022 but new PR needed as that one was prematurely merged in error.

Adds modelCapabilities parameter to the Node SDK's createSession and setModel APIs, allowing SDK consumers to override individual model capabilities (e.g., vision support) without specifying the full capabilities object.

Closes #994

Changes

  • Codegen: Regenerated RPC types from runtime schema (adds modelCapabilities to model.switchTo)
  • Types: Added ModelCapabilitiesOverride type, exported from index
  • Session: createSession and setModel accept optional modelCapabilities
  • E2E tests: Two tests verify vision toggle via setModel changes image processing behavior

Usage

` ypescript
// Override vision at session creation
const session = await client.createSession({
model: "my-byok-model",
provider: { type: "openai", baseUrl: "..." },
modelCapabilities: { supports: { vision: true } },
});

// Toggle vision mid-session
await session.setModel("my-byok-model", {
modelCapabilities: { supports: { vision: false } },
});
`

Testing

E2E tests run against a local runtime build and verify that:

  • Starting with vision disabled then enabling via setModel causes image_url content to appear
  • Starting with vision enabled then disabling via setModel causes image_url content to stop appearing

SteveSandersonMS and others added 7 commits April 7, 2026 10:27
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Regenerated RPC types and added modelCapabilities support to Python, Go, and C#
SDKs to match the Node.js implementation. All languages now support:
- modelCapabilities on createSession/create_session/CreateSession
- modelCapabilities on setModel/set_model/SetModel/SetModelAsync
- E2E tests verifying vision toggle via setModel changes image processing

Also includes:
- Regenerated codegen output with proper type names (no Purple/Fluffy)
- C# codegen fix to respect JSON Schema 'title' for class naming
- Fix Go E2E tests for DataContent type change in session events
- Fix Python generated Union import in session events

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace foreach loops with LINQ .Where/.Any in HasImageUrlContent
- Fix Python import ordering (ruff)
- Apply formatter output across Node/Python

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses Cross-SDK Consistency Review:
- Add modelCapabilities to ResumeSessionConfig Pick list
- Forward modelCapabilities in session.resume RPC call

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The session events codegen produces a breaking DataContent type change
in Go due to quicktype flattening discriminated unions into a single
struct. This is a pre-existing codegen architecture issue that needs
a proper fix (per-variant types). Only RPC types need regeneration
for the modelCapabilities feature.

Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
@SteveSandersonMS SteveSandersonMS marked this pull request as ready for review April 7, 2026 10:04
@SteveSandersonMS SteveSandersonMS requested a review from a team as a code owner April 7, 2026 10:04
Copilot AI review requested due to automatic review settings April 7, 2026 10:04
The codegen check requires all generated files to match. Regenerate
session events alongside RPC types. The Go session events now have
DataContent (union type) for Content field — update all Go consumers
to use .Content.String instead of .Content directly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds cross-SDK support for overriding model capabilities (notably vision) at session creation and via setModel, aligning SDK APIs with updated runtime schema and validating behavior through E2E replay snapshots.

Changes:

  • Regenerates RPC/session-event types to include modelCapabilities on model.switchTo (+ related schema-driven type updates).
  • Exposes modelCapabilities overrides in Node/Python/Go/.NET session create/resume and model switching APIs.
  • Adds E2E coverage (snapshots + tests) to verify that toggling vision via setModel changes whether image_url content is sent.
Show a summary per file
File Description
test/snapshots/session_config/vision_enabled_then_disabled_via_setmodel.yaml Adds replay snapshot for toggling vision off mid-session.
test/snapshots/session_config/vision_disabled_then_enabled_via_setmodel.yaml Adds replay snapshot for toggling vision on mid-session.
test/harness/package.json Bumps @github/copilot used by the shared E2E harness.
test/harness/package-lock.json Lockfile update for harness dependency bump.
scripts/codegen/python.ts Improves Python type modernization for nested Union[...] parsing.
scripts/codegen/csharp.ts Adjusts C# RPC class naming to prefer schema title when present.
python/e2e/test_session_config.py Adds Python E2E tests for vision capability overrides.
python/copilot/session.py Adds model_capabilities support to CopilotSession.set_model().
python/copilot/generated/session_events.py Regenerated session events (schema updates, renames, new fields).
python/copilot/generated/rpc.py Regenerated RPC types including ModelCapabilitiesOverride and vision limits.
python/copilot/client.py Adds Python-side override dataclasses + wiring into create/resume payloads.
python/copilot/init.py Exports new Python override types from the package root.
nodejs/test/e2e/session_config.test.ts Adds Node E2E tests validating vision toggling via setModel.
nodejs/src/types.ts Adds modelCapabilities to SessionConfig and exports override type.
nodejs/src/session.ts Extends setModel() options to accept modelCapabilities.
nodejs/src/index.ts Re-exports ModelCapabilitiesOverride.
nodejs/src/generated/session-events.ts Regenerated session event typings (idle/background tasks + UI completion payloads).
nodejs/src/generated/rpc.ts Regenerated RPC types incl. ModelCapabilities(Override) + vision limits.
nodejs/src/client.ts Sends modelCapabilities in session.create and session.resume RPC payloads.
nodejs/package.json Bumps Node SDK dependency on @github/copilot.
nodejs/package-lock.json Lockfile update for Node SDK dependency bump.
go/types.go Adds ModelCapabilities override to Go session config structs.
go/session.go Adds ModelCapabilities override support to Session.SetModel.
go/rpc/generated_rpc.go Regenerated Go RPC types incl. model capabilities overrides + vision limits.
go/internal/e2e/testharness/proxy.go Updates exchange parsing to handle multimodal content (string vs array).
go/internal/e2e/session_config_test.go Adds Go E2E tests for toggling vision via SetModel.
go/client.go Wires ModelCapabilities into Go create/resume session requests.
dotnet/test/ToolsTests.cs Updates tests to use StringContent after harness content type change.
dotnet/test/ToolResultsTests.cs Updates tests to use StringContent after harness content type change.
dotnet/test/SessionConfigTests.cs Adds .NET E2E tests for toggling vision via SetModelAsync.
dotnet/test/Harness/E2ETestBase.cs Uses StringContent for extracting system message from captured requests.
dotnet/test/Harness/CapiProxy.cs Updates harness message Content to JsonElement? + adds StringContent helper.
dotnet/src/Types.cs Adds ModelCapabilities to session config types and JSON context.
dotnet/src/Session.cs Extends SetModelAsync API to accept modelCapabilities.
dotnet/src/Generated/SessionEvents.cs Regenerated events with updated payloads and removed idle backgroundTasks shape.
dotnet/src/Generated/Rpc.cs Regenerated RPC types incl. ModelCapabilitiesOverride and vision limits.
dotnet/src/Client.cs Wires ModelCapabilities into create/resume session RPC requests and JSON context.

Copilot's findings

Files not reviewed (2)
  • nodejs/package-lock.json: Language not supported
  • test/harness/package-lock.json: Language not supported
  • Files reviewed: 29/37 changed files
  • Comments generated: 2

@github-actions

This comment has been minimized.

SteveSandersonMS and others added 3 commits April 7, 2026 11:56
- Replace 'fake-test-model' with 'claude-sonnet-4.5' in session.test.ts
  (runtime 1.0.20-1 validates model names)
- Regenerate vision and session snapshots against published CLI

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
C# harness converts test names to snake_case via regex. PascalCase names
like VisionDisabledThenEnabled produce no underscores between words,
mismatching the shared snapshots. Use underscore-separated names
(Vision_Disabled_Then_Enabled) to match Node/Go/Python conventions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The runtime 1.0.20-1 validates model names and rejects 'fake-test-model'.
Replace with 'claude-sonnet-4.5' in Python, Go, C# session tests
(Node was already fixed). Regenerate all affected snapshots.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

When the replaying proxy can't find a matching conversation on CI,
log the request message count, each stored conversation's message count,
and the exact mismatch point (message index + content diff).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

Copy link
Copy Markdown
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generated by SDK Consistency Review Agent for issue #1029

Log the pre-normalization raw message alongside the normalized one
to identify what content is being dropped during normalization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

SteveSandersonMS and others added 6 commits April 7, 2026 12:47
The replaying proxy's message normalization only handled string content
for user messages. Array content (multimodal messages with image_url parts)
was silently dropped, producing blank user entries that couldn't match
incoming requests containing image content.

Extract text parts and represent image_url parts as [image] markers
so both capture and replay sides normalize consistently.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The blob and file attachment tests used session.send() (fire-and-forget)
then immediately disconnected. The pending CAPI request leaked into
subsequent tests via the shared CLI process, causing the vision test's
proxy to receive mismatched requests. Use sendAndWait to ensure the
CLI finishes processing before the test completes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The blob attachment test was sending an inline image but the model
would try to view it on disk, resulting in 'file not found' errors
in the snapshot. Write the PNG to workDir so the view tool succeeds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract diagnoseMatchFailure() as a pure function that mirrors the
matching logic in findAssistantIndexAfterPrefix, producing clear
per-conversation explanations of why matching failed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Node.js: Make ModelCapabilitiesOverride truly deep-partial so nested
  properties like limits.vision.max_prompt_images can be overridden
  individually without supplying the full object
- C#: Use Path.Join instead of Path.Combine to avoid code quality
  warning about silently dropped path arguments
- Go: Add type aliases for ModelCapabilitiesOverride types in the main
  copilot package so consumers don't need a separate rpc import

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DeepPartial<string[]> was producing (string | undefined)[] instead
of string[], breaking type compatibility with generated RPC types.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

SteveSandersonMS and others added 4 commits April 7, 2026 13:31
Same fix as Node — fire-and-forget Send leaks pending CAPI requests
into subsequent tests via the shared CLI process. SendAndWait ensures
the conversation completes before the test ends.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Write the test image to workDir so the model can view it successfully.
Without the file on disk, the model runs platform-specific shell
commands (Get-ChildItem vs ls) making the snapshot non-portable.
Also use SendAndWait to prevent request leaking into subsequent tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These Content→Content.String changes came from session events codegen
regeneration and are unrelated to the model capabilities PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The session events codegen regeneration introduced a breaking
DataContent union type change that is unrelated to this PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SteveSandersonMS and others added 2 commits April 7, 2026 13:49
These tests were updated for the DataContent breaking change that
we've now reverted. Restore them to match main.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The DataContent union type from session events codegen was reverted,
so these tests need to use *string Content again.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SteveSandersonMS SteveSandersonMS force-pushed the stevesa/override-model-capabilities branch from 2bd6129 to 4f9f91b Compare April 7, 2026 12:58
The Go session events codegen was intentionally reverted to avoid
a breaking DataContent change (see #1031). This means generated
files will be out of sync until that issue is resolved.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SteveSandersonMS SteveSandersonMS force-pushed the stevesa/override-model-capabilities branch from 4f9f91b to 09f7cee Compare April 7, 2026 12:59
@SteveSandersonMS SteveSandersonMS merged commit dfdc6a0 into main Apr 7, 2026
34 checks passed
@SteveSandersonMS SteveSandersonMS deleted the stevesa/override-model-capabilities branch April 7, 2026 13:06
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2026

Cross-SDK Consistency Review ✅

This PR maintains full cross-language parity for the modelCapabilities / ModelCapabilitiesOverride feature.

SDK SessionConfig ResumeSessionConfig setModel Type exported
Node.js ✅ (this PR) ✅ (this PR) ✅ (this PR)
Python ✅ (from #1022) ✅ (from #1022) ✅ (from #1022)
Go ✅ (this PR) ✅ (this PR) ✅ (SetModelOptions)
.NET ✅ (this PR) ✅ (this PR) ✅ (SetModelAsync)

The ModelCapabilitiesOverride type structure (with supports.vision, supports.reasoningEffort, limits.*) is consistent across all four SDKs. Each SDK also exports the type from its public API surface using idiomatic naming (ModelCapabilitiesOverride in TS/.NET/Go, ModelCapabilitiesOverride in Python).

No consistency gaps found.

Generated by SDK Consistency Review Agent for issue #1029 ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

view tool doesn't work with BYOM

2 participants