Skip to content

Support custom Model Provider Services in Pi - #289

Open
FantomKun wants to merge 1 commit into
databricks:mainfrom
FantomKun:custom-mps-pi
Open

Support custom Model Provider Services in Pi#289
FantomKun wants to merge 1 commit into
databricks:mainfrom
FantomKun:custom-mps-pi

Conversation

@FantomKun

Copy link
Copy Markdown

Fixes #288.

A Unity Catalog Model Provider Service with provider_type: EXTERNAL_MODEL_PROVIDER_TYPE_CUSTOM — a self-hosted, OpenAI-compatible model — could not be used by any agent. _TOOL_PROVIDER_TYPES has no custom entry and no pi key, so resolve_provider_service rejects it with "which pi can't route to (supported: none)", and ucode pi has no --provider flag (it's registered with ignore_unknown_options, so the flag is silently ignored and the launch quietly proceeds on a Databricks model).

Such a service is addressed differently from a vendor one: the Databricks-Model-Provider-Service header selects it, and the request body's model carries the bare target name. Passing the fully-qualified name as model returns PERMISSION_DENIED; dropping the header returns NOT_FOUND. The dialect each target serves is declared in targets[].native_api_types, which the listing parsed and discarded.

Changes

  • databricks.py — keep native_api_types per target as target_api_types (targets keeps its plain-id shape, which map_bedrock_claude_models and the e2e tests rely on); add build_native_api_base_url mapping a dialect to its gateway path, custom_openai_chat_targets, CUSTOM_PROVIDER_TYPES, and the service_usable_for_tool / resolve_provider_service guards. All dialect-level and agent-agnostic, so adding a second agent later is additive.
  • agents/pi.py — emit a databricks-custom provider (openai-completions on /ai-gateway/openai/v1) carrying the service header, with the targets as its models. Listed in PROVIDER_NAMES so it's stripped when a later launch drops --provider, which also gets ucode revert handling for free.
  • agents/pi.py — fix _refresh_token_once raising under a provider-only launch. _refresh_forever swallows the error, so on a workspace with no Databricks models the token silently stopped refreshing and the session died at expiry (~1h).
  • agents/__init__.pyresolve_provider_models returns the target list for a custom service; configure_tool gains a pi branch alongside claude's, with isinstance narrowing so a Bedrock-style dict can't leak into pi's list parameter.
  • cli.py--provider on ucode pi (the option was byte-identical on codex/claude, so it's extracted to a shared ProviderOption), pi added to the interactive picker, and configure --provider-context-window.

New public parameters on render_overlay / write_tool_config are keyword-only and appended, to minimise conflict with #256.

On the context window

The Model Provider Service API exposes no context-window metadata (see #288 for the full config dump), so a value has to be assumed. The two directions are not symmetric — with pi, understating costs earlier compaction and shorter replies, while overstating is unrecoverable: pi compacts to contextWindow - reserveTokens, so a window above the server's real limit makes the compact-and-retry overflow again and the turn ends with "Context overflow recovery failed after one compact-and-retry attempt."

This defaults to a conservative 32768, prints the assumption at launch, and makes it overridable via ucode configure --provider-context-window. The override is load-bearing rather than cosmetic: ucode strips its own provider block on every write, so a user cannot durably hand-edit models.json to correct a large endpoint. Concretely, the endpoint I tested against reported 32768 when first measured and 327680 a day later — the server had been resized, with no way for a client to notice.

Only three compat flags are emitted (maxTokensField, supportsDeveloperRole, supportsStore). Pi's detectCompat already returns correct defaults for supportsReasoningEffort / supportsUsageInStreaming / supportsStrictMode on an unrecognized base URL, and a test_compat_is_exactly_the_behavior_changing_flags set-equality assertion keeps redundant or invalid flags from creeping back in. No reasoning / thinkingLevelMap is claimed, since the API exposes no capability metadata and asserting it would make pi send reasoning_effort to a server that may reject it.

Testing

uv run ruff check .                  # pass
uv run ruff format --check src/ tests/   # pass
uv run ty check src/                 # pass
uv run pytest                        # 1262 passed, no regressions (+49 tests)

The 7 remaining failures are pre-existing on main (Rich terminal-width assertions in test_cli.py/test_mcp.py, plus two network-dependent test_e2e_user_agent.py cases) and fail identically without this change.

Live e2e against a real custom service (llm_gov.gateway.deepseek-v4-flash, SGLang behind PrivateLink):

tests/test_e2e.py::TestModelProviderLaunch::test_launch_pi_through_custom_provider PASSED

Also verified by hand: discovery and resolve on the path that previously errored; the generated config; a real tool round-trip (pi reading a file with its own tools and reporting the contents, with no --model so it resolves through the pinned default); token refresh with zero Databricks models in state; the context-window override reaching the generated config; and a subsequent non-provider launch removing databricks-custom while leaving a hand-added provider untouched.

Note for CI: a custom service fronts a caller-operated endpoint that can be down independently of ucode (the gateway surfaces that as a 502 wrapping the upstream 503). The e2e test skips in that case, mirroring the existing USE CONNECTION/EXECUTE permission skip. I confirmed the guard is narrow — an unrelated non-502 failure still fails the test.

Out of scope

Other agents (codex needs openai/v1/responses, a different dialect and gateway path; copilot/opencode/gemini each need their own config dialect and verification), other native_api_types on a custom service (_NATIVE_API_GATEWAY_PATHS is a one-line extension point), pi over anthropic/openai/amazon_bedrock service types, and multiple simultaneous services per agent (provider_services is {tool: name}).

This pull request and its description were written by Isaac.

A Unity Catalog Model Provider Service with provider_type
EXTERNAL_MODEL_PROVIDER_TYPE_CUSTOM — a self-hosted, OpenAI-compatible
model — could not be used by any agent. `_TOOL_PROVIDER_TYPES` had no
`custom` entry and no `pi` key, so `resolve_provider_service` rejected it
with "which pi can't route to (supported: none)", and `ucode pi` had no
`--provider` flag to begin with.

Such a service is addressed differently from a vendor one: the
`Databricks-Model-Provider-Service` header selects it and the request
body's `model` carries the bare target name (the fully-qualified name
returns NOT_FOUND). The dialect each target serves is declared in
`targets[].native_api_types`, which the listing parsed and discarded.

- databricks.py: keep `native_api_types` per target; add
  `build_native_api_base_url` mapping a dialect to its gateway path,
  `custom_openai_chat_targets`, `CUSTOM_PROVIDER_TYPES`, and the
  usability/resolve guards. All agent-agnostic, so a second agent is
  additive.
- agents/pi.py: emit a `databricks-custom` provider (openai-completions
  on /ai-gateway/openai/v1) with the service header and the three compat
  flags that actually change pi's behavior for an unknown backend. Listed
  in PROVIDER_NAMES so it's stripped when a later launch drops
  --provider.
- agents/pi.py: fix `_refresh_token_once` raising under a provider-only
  launch. `_refresh_forever` swallowed the error, so the token silently
  stopped refreshing and the session died at expiry.
- cli.py: `--provider` on `ucode pi` (extracted to a shared
  ProviderOption), pi in the interactive picker, and
  `configure --provider-context-window`.

The Model Provider Service API exposes no context-window metadata, so a
value has to be assumed. The directions are not symmetric: understating
costs earlier compaction, while overstating is unrecoverable — pi
compacts to `contextWindow - reserveTokens`, so a window above the
server's real limit makes the compact-and-retry overflow again and the
turn ends. Default to a conservative 32768, print the assumption at
launch, and make it overridable.

Verified end to end against a live custom service: discovery, resolve,
generated config, a real tool round-trip through pi, token refresh with
zero Databricks models on the workspace, the context-window override, and
cleanup leaving a hand-added provider intact.

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Custom (EXTERNAL_MODEL_PROVIDER_TYPE_CUSTOM) Model Provider Services are unusable: no tool routes to 'custom', and pi has no --provider

1 participant