Add modelCapabilities override to all SDK languages#1029
Add modelCapabilities override to all SDK languages#1029SteveSandersonMS merged 26 commits intomainfrom
Conversation
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Regenerated RPC types and added modelCapabilities support to Python, Go, and C# SDKs to match the Node.js implementation. All languages now support: - modelCapabilities on createSession/create_session/CreateSession - modelCapabilities on setModel/set_model/SetModel/SetModelAsync - E2E tests verifying vision toggle via setModel changes image processing Also includes: - Regenerated codegen output with proper type names (no Purple/Fluffy) - C# codegen fix to respect JSON Schema 'title' for class naming - Fix Go E2E tests for DataContent type change in session events - Fix Python generated Union import in session events Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace foreach loops with LINQ .Where/.Any in HasImageUrlContent - Fix Python import ordering (ruff) - Apply formatter output across Node/Python Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses Cross-SDK Consistency Review: - Add modelCapabilities to ResumeSessionConfig Pick list - Forward modelCapabilities in session.resume RPC call Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The session events codegen produces a breaking DataContent type change in Go due to quicktype flattening discriminated unions into a single struct. This is a pre-existing codegen architecture issue that needs a proper fix (per-variant types). Only RPC types need regeneration for the modelCapabilities feature. Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
The codegen check requires all generated files to match. Regenerate session events alongside RPC types. The Go session events now have DataContent (union type) for Content field — update all Go consumers to use .Content.String instead of .Content directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds cross-SDK support for overriding model capabilities (notably vision) at session creation and via setModel, aligning SDK APIs with updated runtime schema and validating behavior through E2E replay snapshots.
Changes:
- Regenerates RPC/session-event types to include
modelCapabilitiesonmodel.switchTo(+ related schema-driven type updates). - Exposes
modelCapabilitiesoverrides in Node/Python/Go/.NET session create/resume and model switching APIs. - Adds E2E coverage (snapshots + tests) to verify that toggling vision via
setModelchanges whetherimage_urlcontent is sent.
Show a summary per file
| File | Description |
|---|---|
| test/snapshots/session_config/vision_enabled_then_disabled_via_setmodel.yaml | Adds replay snapshot for toggling vision off mid-session. |
| test/snapshots/session_config/vision_disabled_then_enabled_via_setmodel.yaml | Adds replay snapshot for toggling vision on mid-session. |
| test/harness/package.json | Bumps @github/copilot used by the shared E2E harness. |
| test/harness/package-lock.json | Lockfile update for harness dependency bump. |
| scripts/codegen/python.ts | Improves Python type modernization for nested Union[...] parsing. |
| scripts/codegen/csharp.ts | Adjusts C# RPC class naming to prefer schema title when present. |
| python/e2e/test_session_config.py | Adds Python E2E tests for vision capability overrides. |
| python/copilot/session.py | Adds model_capabilities support to CopilotSession.set_model(). |
| python/copilot/generated/session_events.py | Regenerated session events (schema updates, renames, new fields). |
| python/copilot/generated/rpc.py | Regenerated RPC types including ModelCapabilitiesOverride and vision limits. |
| python/copilot/client.py | Adds Python-side override dataclasses + wiring into create/resume payloads. |
| python/copilot/init.py | Exports new Python override types from the package root. |
| nodejs/test/e2e/session_config.test.ts | Adds Node E2E tests validating vision toggling via setModel. |
| nodejs/src/types.ts | Adds modelCapabilities to SessionConfig and exports override type. |
| nodejs/src/session.ts | Extends setModel() options to accept modelCapabilities. |
| nodejs/src/index.ts | Re-exports ModelCapabilitiesOverride. |
| nodejs/src/generated/session-events.ts | Regenerated session event typings (idle/background tasks + UI completion payloads). |
| nodejs/src/generated/rpc.ts | Regenerated RPC types incl. ModelCapabilities(Override) + vision limits. |
| nodejs/src/client.ts | Sends modelCapabilities in session.create and session.resume RPC payloads. |
| nodejs/package.json | Bumps Node SDK dependency on @github/copilot. |
| nodejs/package-lock.json | Lockfile update for Node SDK dependency bump. |
| go/types.go | Adds ModelCapabilities override to Go session config structs. |
| go/session.go | Adds ModelCapabilities override support to Session.SetModel. |
| go/rpc/generated_rpc.go | Regenerated Go RPC types incl. model capabilities overrides + vision limits. |
| go/internal/e2e/testharness/proxy.go | Updates exchange parsing to handle multimodal content (string vs array). |
| go/internal/e2e/session_config_test.go | Adds Go E2E tests for toggling vision via SetModel. |
| go/client.go | Wires ModelCapabilities into Go create/resume session requests. |
| dotnet/test/ToolsTests.cs | Updates tests to use StringContent after harness content type change. |
| dotnet/test/ToolResultsTests.cs | Updates tests to use StringContent after harness content type change. |
| dotnet/test/SessionConfigTests.cs | Adds .NET E2E tests for toggling vision via SetModelAsync. |
| dotnet/test/Harness/E2ETestBase.cs | Uses StringContent for extracting system message from captured requests. |
| dotnet/test/Harness/CapiProxy.cs | Updates harness message Content to JsonElement? + adds StringContent helper. |
| dotnet/src/Types.cs | Adds ModelCapabilities to session config types and JSON context. |
| dotnet/src/Session.cs | Extends SetModelAsync API to accept modelCapabilities. |
| dotnet/src/Generated/SessionEvents.cs | Regenerated events with updated payloads and removed idle backgroundTasks shape. |
| dotnet/src/Generated/Rpc.cs | Regenerated RPC types incl. ModelCapabilitiesOverride and vision limits. |
| dotnet/src/Client.cs | Wires ModelCapabilities into create/resume session RPC requests and JSON context. |
Copilot's findings
Files not reviewed (2)
- nodejs/package-lock.json: Language not supported
- test/harness/package-lock.json: Language not supported
- Files reviewed: 29/37 changed files
- Comments generated: 2
This comment has been minimized.
This comment has been minimized.
- Replace 'fake-test-model' with 'claude-sonnet-4.5' in session.test.ts (runtime 1.0.20-1 validates model names) - Regenerate vision and session snapshots against published CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
C# harness converts test names to snake_case via regex. PascalCase names like VisionDisabledThenEnabled produce no underscores between words, mismatching the shared snapshots. Use underscore-separated names (Vision_Disabled_Then_Enabled) to match Node/Go/Python conventions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The runtime 1.0.20-1 validates model names and rejects 'fake-test-model'. Replace with 'claude-sonnet-4.5' in Python, Go, C# session tests (Node was already fixed). Regenerate all affected snapshots. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
When the replaying proxy can't find a matching conversation on CI, log the request message count, each stored conversation's message count, and the exact mismatch point (message index + content diff). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Generated by SDK Consistency Review Agent for issue #1029
Log the pre-normalization raw message alongside the normalized one to identify what content is being dropped during normalization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
The replaying proxy's message normalization only handled string content for user messages. Array content (multimodal messages with image_url parts) was silently dropped, producing blank user entries that couldn't match incoming requests containing image content. Extract text parts and represent image_url parts as [image] markers so both capture and replay sides normalize consistently. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The blob and file attachment tests used session.send() (fire-and-forget) then immediately disconnected. The pending CAPI request leaked into subsequent tests via the shared CLI process, causing the vision test's proxy to receive mismatched requests. Use sendAndWait to ensure the CLI finishes processing before the test completes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The blob attachment test was sending an inline image but the model would try to view it on disk, resulting in 'file not found' errors in the snapshot. Write the PNG to workDir so the view tool succeeds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract diagnoseMatchFailure() as a pure function that mirrors the matching logic in findAssistantIndexAfterPrefix, producing clear per-conversation explanations of why matching failed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Node.js: Make ModelCapabilitiesOverride truly deep-partial so nested properties like limits.vision.max_prompt_images can be overridden individually without supplying the full object - C#: Use Path.Join instead of Path.Combine to avoid code quality warning about silently dropped path arguments - Go: Add type aliases for ModelCapabilitiesOverride types in the main copilot package so consumers don't need a separate rpc import Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DeepPartial<string[]> was producing (string | undefined)[] instead of string[], breaking type compatibility with generated RPC types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
Same fix as Node — fire-and-forget Send leaks pending CAPI requests into subsequent tests via the shared CLI process. SendAndWait ensures the conversation completes before the test ends. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Write the test image to workDir so the model can view it successfully. Without the file on disk, the model runs platform-specific shell commands (Get-ChildItem vs ls) making the snapshot non-portable. Also use SendAndWait to prevent request leaking into subsequent tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These Content→Content.String changes came from session events codegen regeneration and are unrelated to the model capabilities PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The session events codegen regeneration introduced a breaking DataContent union type change that is unrelated to this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These tests were updated for the DataContent breaking change that we've now reverted. Restore them to match main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The DataContent union type from session events codegen was reverted, so these tests need to use *string Content again. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2bd6129 to
4f9f91b
Compare
The Go session events codegen was intentionally reverted to avoid a breaking DataContent change (see #1031). This means generated files will be out of sync until that issue is resolved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
4f9f91b to
09f7cee
Compare
Cross-SDK Consistency Review ✅This PR maintains full cross-language parity for the
The No consistency gaps found.
|
Summary
Same as #1022 but new PR needed as that one was prematurely merged in error.
Adds
modelCapabilitiesparameter to the Node SDK'screateSessionandsetModelAPIs, allowing SDK consumers to override individual model capabilities (e.g., vision support) without specifying the full capabilities object.Closes #994
Changes
modelCapabilitiestomodel.switchTo)ModelCapabilitiesOverridetype, exported from indexcreateSessionandsetModelaccept optionalmodelCapabilitiessetModelchanges image processing behaviorUsage
` ypescript
// Override vision at session creation
const session = await client.createSession({
model: "my-byok-model",
provider: { type: "openai", baseUrl: "..." },
modelCapabilities: { supports: { vision: true } },
});
// Toggle vision mid-session
await session.setModel("my-byok-model", {
modelCapabilities: { supports: { vision: false } },
});
`
Testing
E2E tests run against a local runtime build and verify that:
setModelcausesimage_urlcontent to appearsetModelcausesimage_urlcontent to stop appearing