Support custom Model Provider Services in Pi - #289
Open
FantomKun wants to merge 1 commit into
Open
Conversation
A Unity Catalog Model Provider Service with provider_type EXTERNAL_MODEL_PROVIDER_TYPE_CUSTOM — a self-hosted, OpenAI-compatible model — could not be used by any agent. `_TOOL_PROVIDER_TYPES` had no `custom` entry and no `pi` key, so `resolve_provider_service` rejected it with "which pi can't route to (supported: none)", and `ucode pi` had no `--provider` flag to begin with. Such a service is addressed differently from a vendor one: the `Databricks-Model-Provider-Service` header selects it and the request body's `model` carries the bare target name (the fully-qualified name returns NOT_FOUND). The dialect each target serves is declared in `targets[].native_api_types`, which the listing parsed and discarded. - databricks.py: keep `native_api_types` per target; add `build_native_api_base_url` mapping a dialect to its gateway path, `custom_openai_chat_targets`, `CUSTOM_PROVIDER_TYPES`, and the usability/resolve guards. All agent-agnostic, so a second agent is additive. - agents/pi.py: emit a `databricks-custom` provider (openai-completions on /ai-gateway/openai/v1) with the service header and the three compat flags that actually change pi's behavior for an unknown backend. Listed in PROVIDER_NAMES so it's stripped when a later launch drops --provider. - agents/pi.py: fix `_refresh_token_once` raising under a provider-only launch. `_refresh_forever` swallowed the error, so the token silently stopped refreshing and the session died at expiry. - cli.py: `--provider` on `ucode pi` (extracted to a shared ProviderOption), pi in the interactive picker, and `configure --provider-context-window`. The Model Provider Service API exposes no context-window metadata, so a value has to be assumed. The directions are not symmetric: understating costs earlier compaction, while overstating is unrecoverable — pi compacts to `contextWindow - reserveTokens`, so a window above the server's real limit makes the compact-and-retry overflow again and the turn ends. Default to a conservative 32768, print the assumption at launch, and make it overridable. Verified end to end against a live custom service: discovery, resolve, generated config, a real tool round-trip through pi, token refresh with zero Databricks models on the workspace, the context-window override, and cleanup leaving a hand-added provider intact. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #288.
A Unity Catalog Model Provider Service with
provider_type: EXTERNAL_MODEL_PROVIDER_TYPE_CUSTOM— a self-hosted, OpenAI-compatible model — could not be used by any agent._TOOL_PROVIDER_TYPEShas nocustomentry and nopikey, soresolve_provider_servicerejects it with "which pi can't route to (supported: none)", anducode pihas no--providerflag (it's registered withignore_unknown_options, so the flag is silently ignored and the launch quietly proceeds on a Databricks model).Such a service is addressed differently from a vendor one: the
Databricks-Model-Provider-Serviceheader selects it, and the request body'smodelcarries the bare target name. Passing the fully-qualified name asmodelreturnsPERMISSION_DENIED; dropping the header returnsNOT_FOUND. The dialect each target serves is declared intargets[].native_api_types, which the listing parsed and discarded.Changes
databricks.py— keepnative_api_typesper target astarget_api_types(targetskeeps its plain-id shape, whichmap_bedrock_claude_modelsand the e2e tests rely on); addbuild_native_api_base_urlmapping a dialect to its gateway path,custom_openai_chat_targets,CUSTOM_PROVIDER_TYPES, and theservice_usable_for_tool/resolve_provider_serviceguards. All dialect-level and agent-agnostic, so adding a second agent later is additive.agents/pi.py— emit adatabricks-customprovider (openai-completionson/ai-gateway/openai/v1) carrying the service header, with the targets as its models. Listed inPROVIDER_NAMESso it's stripped when a later launch drops--provider, which also getsucode reverthandling for free.agents/pi.py— fix_refresh_token_onceraising under a provider-only launch._refresh_foreverswallows the error, so on a workspace with no Databricks models the token silently stopped refreshing and the session died at expiry (~1h).agents/__init__.py—resolve_provider_modelsreturns the target list for a custom service;configure_toolgains apibranch alongside claude's, withisinstancenarrowing so a Bedrock-style dict can't leak into pi's list parameter.cli.py—--provideronucode pi(the option was byte-identical on codex/claude, so it's extracted to a sharedProviderOption), pi added to the interactive picker, andconfigure --provider-context-window.New public parameters on
render_overlay/write_tool_configare keyword-only and appended, to minimise conflict with #256.On the context window
The Model Provider Service API exposes no context-window metadata (see #288 for the full config dump), so a value has to be assumed. The two directions are not symmetric — with pi, understating costs earlier compaction and shorter replies, while overstating is unrecoverable: pi compacts to
contextWindow - reserveTokens, so a window above the server's real limit makes the compact-and-retry overflow again and the turn ends with "Context overflow recovery failed after one compact-and-retry attempt."This defaults to a conservative 32768, prints the assumption at launch, and makes it overridable via
ucode configure --provider-context-window. The override is load-bearing rather than cosmetic: ucode strips its own provider block on every write, so a user cannot durably hand-editmodels.jsonto correct a large endpoint. Concretely, the endpoint I tested against reported 32768 when first measured and 327680 a day later — the server had been resized, with no way for a client to notice.Only three
compatflags are emitted (maxTokensField,supportsDeveloperRole,supportsStore). Pi'sdetectCompatalready returns correct defaults forsupportsReasoningEffort/supportsUsageInStreaming/supportsStrictModeon an unrecognized base URL, and atest_compat_is_exactly_the_behavior_changing_flagsset-equality assertion keeps redundant or invalid flags from creeping back in. Noreasoning/thinkingLevelMapis claimed, since the API exposes no capability metadata and asserting it would make pi sendreasoning_effortto a server that may reject it.Testing
The 7 remaining failures are pre-existing on
main(Rich terminal-width assertions intest_cli.py/test_mcp.py, plus two network-dependenttest_e2e_user_agent.pycases) and fail identically without this change.Live e2e against a real custom service (
llm_gov.gateway.deepseek-v4-flash, SGLang behind PrivateLink):Also verified by hand: discovery and resolve on the path that previously errored; the generated config; a real tool round-trip (pi reading a file with its own tools and reporting the contents, with no
--modelso it resolves through the pinned default); token refresh with zero Databricks models in state; the context-window override reaching the generated config; and a subsequent non-provider launch removingdatabricks-customwhile leaving a hand-added provider untouched.Note for CI: a custom service fronts a caller-operated endpoint that can be down independently of ucode (the gateway surfaces that as a 502 wrapping the upstream 503). The e2e test skips in that case, mirroring the existing
USE CONNECTION/EXECUTEpermission skip. I confirmed the guard is narrow — an unrelated non-502 failure still fails the test.Out of scope
Other agents (codex needs
openai/v1/responses, a different dialect and gateway path; copilot/opencode/gemini each need their own config dialect and verification), othernative_api_typeson a custom service (_NATIVE_API_GATEWAY_PATHSis a one-line extension point), pi overanthropic/openai/amazon_bedrockservice types, and multiple simultaneous services per agent (provider_servicesis{tool: name}).This pull request and its description were written by Isaac.