Re-land Custom model endpoint provider with the provider-coverage gaps closed - #1437
Merged
Paul Lizer (paullizer) merged 11 commits intoSep 5, 2026
Merged
Conversation
Re-applies the changes from PR #1228, which was merged prematurely and reverted by PR #1431. This restores the feature onto current Development as a working base so the remaining provider-coverage gaps can be closed before it is proposed for merge again. Refs #1228 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
SimpleChat decided what a model could do by pattern-matching its name, so any model the shipped catalog did not know about -- an on-premises or customer supplied model reached through a Custom endpoint -- silently got the wrong answer for vision, tool calling, streaming, and reasoning. A model named "corp-llm-v2" was treated as having no capabilities at all. Capability answers now resolve through per-model override, endpoint override, catalog entry, then the original name heuristic, so the catalog can be incomplete without blocking an administrator and unknown models behave exactly as they did before. Two matching rules keep this honest. "family" is never used to match, because members of one family disagree on capabilities: the phi-4 family holds both the multimodal and the text-only Phi models, and each gpt-5.x family holds a non-vision "-chat" member. And a longer identifier prefix wins, with a digit-leading remainder rejected as a version continuation rather than a variant, so gpt-5.1-chat-v2 resolves to gpt-5.1-chat and gpt-5.3 no longer resolves to gpt-5. The catalog gains supportsStreaming and reasoning for all 65 existing models and 10 Google Gemini records, a provider it did not cover at all. Claude, Llama 4, and Phi-4 multimodal are now correctly recognised as vision-capable, and the gpt-5.x "-chat" variants correctly as not. A published JSON schema plus a coverage test means a malformed record fails a test rather than degrading capability answers at runtime. Refs #1228 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Custom endpoints supported exactly three API types, each hard-coded in five places: the allowlist, the request-model resolver, the protocol inference chain, the admin template option list, and the admin JavaScript. Adding a provider meant editing all five, and missing one failed silently. An API type is now one declarative record carrying its wire protocol, model identifier field, URL policy, accepted auth types, and version field. The admin option list, model identifier label, version fields, validation, and protocol inference all derive from it. The three existing API types are unchanged and an unregistered api_type is still refused. Google Gemini is registered, reached through its OpenAI-compatible surface so it keeps running on the validated-DNS pinned transport rather than needing a vendor SDK with its own transport. This also fixes URL construction for any surface that already carries a version segment. SimpleChat appended /v1 unconditionally, turning Gemini's .../v1beta/openai base into .../v1beta/openai/v1 and a 404. The URL policy is now per provider, so /v1 is appended only where it belongs. The registry reaches the browser as server-rendered inline JSON, through the modal's data-api-types attribute and a window helper in base.html for scripts that build model lists without the endpoint editor present. No hard-coded api_type comparison remains in the admin or workspace scripts, which the existing UI contract test now asserts. Refs #1228 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Every Custom endpoint failure produced the same sentence, "Custom model request failed.", with the cause discarded by "from None" before it could reach the log. A wrong path, a wrong key, a wrong model name, a TLS failure, and a blocked address were indistinguishable, and nothing anywhere explained which had happened. Seven failure paths did this. Sanitizing what reaches the browser is right, because an upstream error body can echo back a URL, a header, or an API key. Destroying it before it reaches the server log is not. The browser message stays sanitized and now carries a short reference id. The real cause is recorded server-side with the API type, the resolved request URL, the upstream status code, and the upstream body, stamped with the same id so an administrator can join a reported message to the entry that explains it. The resolved request URL is included deliberately: URL normalization rewrites what the administrator typed, and that rewrite was previously invisible, so a 404 caused by a rewritten path could not be diagnosed at all. Credentials are redacted before anything is logged, covering api_key fields, Authorization and bearer tokens, x-api-key, x-goog-api-key, key query parameters, and sk- style keys. Detail is truncated so a large upstream body cannot flood the log, and a logging failure is swallowed so it can never replace the model error. The existing provider test asserted the message was exactly "Custom model request failed."; it now asserts the security property, that no provider detail leaks, plus the presence of a reference id. Refs #1228 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
SimpleChat appended /v1 to every Custom OpenAI-compatible endpoint, even when the configured URL already said where the API lived. A gateway at https://apim.example.com/inference/chat/completions was called at https://apim.example.com/inference/v1/, and any base carrying its own version segment was broken the same way. The rewrite was also invisible, so a misdirected request looked identical to a correct one. /v1 is now appended only when the URL does not already name the API surface. A last path segment matching a version, such as v1, v2, or v1beta, is left alone, and a full operation URL is treated as stating the base exactly, so its suffix is stripped and the remainder used as given. Some gateways mount the API at a path that cannot be inferred, where both the given path and a /v1 child are plausible. Rather than guess, the endpoint editor gains a "Use this URL exactly as entered" option, stored as connection.url_mode and honoured for any API type. Test Connection now reports the URL that was actually called, so the rewrite is always verifiable instead of silent. Refs #1228 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
SimpleChat supports one response mode, streaming, but some responses cannot be streamed by the model. Tool calling is the clearest case: a tool call has to arrive complete, so the request is made without streaming. That path already pushed the completed answer through the streaming interface, but as a single chunk, so the user saw nothing and then everything at once, which reads as a hang. Agents and plugins rely on tool calling, so this was a common path rather than an exotic one. A completed answer is now split at word boundaries and delivered progressively. Chunking is lossless, which matters because the frontend accumulates chunks by concatenation: the tokenizer preserves every character, including runs of whitespace and newlines, so the reassembled text is identical to the original. Function calls cannot be partially delivered, so non-text items ride on the final message alongside the finish reason and usage metadata. Emitting metadata exactly once keeps token usage from being multiplied by the chunk count. This also restores usage reporting on streaming responses. stream_options is how an OpenAI-compatible stream reports token usage, and it was stripped from every request, including from endpoints that accept it. Support is now declared per provider and the option is dropped only for surfaces that reject it. Refs #1228 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The gate named "allow private Custom endpoint hosts" did not permit the two address forms on-premises deployments actually use. An IP address and a short single-label host name were both rejected even with the gate enabled, and both were refused with a message claiming the URL was an IP address, which was simply wrong for the short host name and named no setting that would allow it. Even once an address was accepted, the connection could not complete. The outbound transport trusts only certifi public roots and deliberately ignores SSL_CERT_FILE, so a gateway presenting an internally issued certificate could never be validated. On-premises inference was unreachable in practice. With the gate enabled, IP addresses, short host names, and private-range hosts are now accepted, and administrators can name a CA bundle to trust an internal issuer. Ambient environment variables are still ignored, so widening trust stays a recorded configuration decision, and a bundle that cannot be loaded fails the request rather than silently falling back to public roots. Plaintext HTTP gets its own second gate for isolated networks, labelled with its consequence, and requires the private-hosts gate as well. None of this weakens the outbound protections. Loopback, link-local, and cloud metadata addresses stay blocked with every gate enabled, and addresses are still revalidated at connection time. Saving an endpoint no longer requires the host to resolve from the application tier, so configuration can be seeded or restored ahead of connectivity. Only resolution is tolerated; policy violations are still refused at save time, and are reported as policy violations rather than resolution failures. Egress proxies remain unsupported on purpose: with a proxy the CONNECT target is resolved by the proxy, so connection-time pinning would protect nothing, and replacing it with a host allowlist is a security decision rather than plumbing. Refs #1228 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Custom endpoints accepted one scheme: an API key sent in whichever header the built-in providers happened to use. That covers OpenAI and Anthropic and nothing else, so a gateway reading the key from a different header, a corporate gateway issuing short-lived tokens, and an appliance requiring a client certificate were all unreachable. The API key header name and value prefix are now configurable, which is what lets one scheme cover Authorization: Bearer, Anthropic's x-api-key, Google's x-goog-api-key, and bespoke gateway headers. Each provider supplies its own default so the common case needs no configuration. Static bearer tokens are supported, as is the OAuth2 client credentials grant. Tokens are cached per token URL, client, and scope, and refreshed ahead of expiry so one cannot lapse mid-request. Two things about the token endpoint matter. It is a different host from the inference endpoint, so it is validated against the same outbound policy and cannot become an unchecked request target: a token URL pointing at cloud metadata is refused exactly as an inference endpoint would be. And its error body routinely echoes the client id or secret, so failures are sanitized to the browser and recorded server-side with credentials redacted. mTLS is modelled as a transport concern rather than an auth type, because a client certificate combines with any scheme. Certificates are referenced by file path, never by value, so a private key is mounted into the deployment instead of being written to the configuration database and replicated with it. Refs #1228 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Validate the OAuth2 token endpoint when the token is fetched, not only when the endpoint is saved. Checking at save time leaves the request itself unguarded, because settings can be written by another path, restored from backup, or changed after validation, so code scanning was right to call this a server-side request forgery. The token URL is now revalidated against the same outbound policy as the inference endpoint and the request runs on the same pinned transport, so its addresses are validated at connection time. That transport refuses redirects, which the previous code avoided on the stated grounds that token endpoints commonly redirect. That reasoning was wrong: redirects belong to the browser-based authorization-code flow, while a client-credentials token endpoint answers a server-to-server POST with a JSON body. A regression test now asserts a blocked token URL is refused before any HTTP client is constructed. Remove backtracking from the version-segment pattern. Its optional suffix could begin with a digit, making it ambiguous with the preceding digits and quadratic on a long run of them: 8,000 characters took about 0.19s to reject and now takes 0.0003s. Requiring the suffix to begin with a letter matches exactly the same segments. Move the auth-type constants and their normalizer to the provider registry so validation no longer imports the authentication module. That removes the real coupling behind several cyclic-import findings and lets the token fetch import validation normally rather than through a deferred import whose comment would otherwise have become untrue. Drop genuinely dead imports, and declare the deliberate re-exports in functions_model_endpoint_types with __all__, since functions_settings and the provider tests import those constants through it. Close file handles in tests and stop importing the same module both ways. The remaining cyclic-import findings are pre-existing: the same single nine-module cycle is present on Development, and none of the modules in this change participate in an import-time cycle. Refs #1228 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ease Development published v0.261.010 and v0.261.011 for the Azure Managed Redis work while this branch was using the same numbers, so both release-notes sections and config.py collided. Development's sections are authoritative and are kept exactly as published. This branch's nine sections move above them, to v0.261.012 through v0.261.020, and config.py becomes 0.261.020. The restored PR #1228 entry is renumbered too: its original v0.261.010 claim was vacated when that PR was reverted, so it re-lands alongside the fixes stacked on it. Version references in this branch's tests and feature documentation are updated to match, so each still names the version its behaviour ships in. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ease Development published v0.261.012 for the Yamcs basic authentication work while this branch was using that number for the restored PR #1228 entry, so the release notes and config.py collided again. Development's sections are kept exactly as published. This branch's nine sections shift up one, to v0.261.013 through v0.261.021, and config.py becomes 0.261.021. Version references in this branch's tests and feature documentation move with them. The sections belonging to this branch are now identified by content rather than by a hardcoded list of numbers, so a further collision while this PR is open is resolved the same way without guessing which sections are whose. The generated release-notes pages are rebuilt from the merged source rather than hand-resolved, since they are build output. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
PR #1228 was merged prematurely and reverted by #1431. This restores that work and closes the gaps that made it not-ready, so Custom endpoints actually support the provider shapes people need rather than only the three they were built against.
The first commit is a plain revert-of-revert, so the restore can be reviewed separately from the seven fixes stacked on top.
0a359d6add18f41588e5224217e3862a3bbbfcdb934a557726c3344b51178c36The gaps, and what changed
Capabilities were guessed from the model's name.
functions_model_capabilities.pynever openedmodel_capabilities.json— it was pure regex. "Is this Claude?" was"claude" in name; "is this a reasoning model?" wasstartswith("gpt-5"). A customer's on-premises model namedcorp-llm-v2therefore got the wrong answer to every capability question. Capabilities now resolve through per-model override → endpoint override → catalog → the original heuristic, so unknown models behave exactly as before and administrators can describe one without waiting on a catalog release.Two matching bugs surfaced while building it and are fixed:
familywas lettingphi-4inheritphi-4-multimodal-instruct's vision capability, andgpt-5.3was prefix-matchinggpt-5because normalization collapses.and-. The catalog gainedsupportsStreaming,reasoning, 10 Google Gemini records — a provider it did not cover at all — and a JSON schema so a malformed record fails a test rather than degrading answers at runtime.Adding a provider meant editing five places. The api-type allowlist, the request-model resolver, the protocol inference chain, the admin template's option list, and the admin JavaScript. An API type is now one declarative registry entry carrying its wire protocol, model identifier field, URL policy, auth types, and version field. Google Gemini is registered, reached through its OpenAI-compatible surface so it keeps running on the validated-DNS pinned transport rather than needing a vendor SDK with its own.
URLs were rewritten into 404s.
/v1was appended unconditionally, sohttps://apim.example.com/inference/chat/completionswas called at…/inference/v1/, and any base carrying its own version segment broke the same way. It is now appended only when the URL does not already name the API surface, with a "use this URL exactly as entered" escape hatch for gateways at an unguessable path. Test Connection reports the URL actually called, because that rewrite was previously invisible.On-premises was unreachable. The gate named "allow private Custom endpoint hosts" did not permit IP literals or short host names — the two forms on-premises deployments use — and refused both with a message claiming the URL was an IP address. Even once an address passed, the transport trusted only certifi's public roots and ignored
SSL_CERT_FILE, so an internally issued certificate could never validate. Both are fixed, with a CA bundle setting and a separate loud plaintext-HTTP gate. Loopback, link-local, and cloud metadata stay blocked with every gate enabled.Every failure said the same thing. Seven paths did
raise RuntimeError("Custom model request failed.") from None, discarding the cause before it could reach the log. Sanitizing what reaches the browser is right; destroying it before the server log is not. The browser message stays generic and gains a reference id; the log gets API type, resolved URL, status code, and upstream body with credentials redacted.Auth was API-key-only. Header name and value prefix are now configurable, which is what lets one scheme cover
Authorization: Bearer, Anthropic'sx-api-key, Google'sx-goog-api-key, and bespoke gateway headers. Bearer tokens and the OAuth2 client-credentials grant are supported, with token caching and refresh ahead of expiry.Tool-calling responses did not really stream. Tool calling cannot stream, so the completed answer was pushed through the streaming interface as a single chunk — the user saw nothing, then everything. Agents and plugins rely on tool calling, so this was a common path. Answers are now chunked losslessly, with tool calls and usage metadata on the final chunk only.
stream_optionswas also being stripped from every OpenAI-compatible request, silently suppressing usage reporting; it is now dropped only for surfaces that reject it.Two decisions worth reviewer attention
Egress proxies are refused deliberately, not overlooked. With an HTTP proxy the client issues
CONNECT host:443and the proxy resolves DNS, so connection-time address pinning would protect nothing. Supporting a proxy means replacing that control with an administrator-managed host allowlist, which is a security decision rather than plumbing, so the proxy is refused instead of silently unprotected.mTLS certificates are referenced by path, never by value. A private key pasted into a settings field would be written to Cosmos and replicated wherever that database goes. The key is mounted into the deployment and its path named instead.
Out of scope
Embeddings remain Azure-only and are untouched.
functions_content.generate_embeddingstill constructsAzureOpenAIdirectly, so a pure-OpenAI, pure-Anthropic, or air-gapped deployment still needs an Azure OpenAI resource for RAG. That is a separate piece of work.Testing
Seven suites, 49 assertions, all passing at
0.261.017:test_model_capability_catalog_resolution.pytest_model_endpoint_provider_registry.pytest_custom_model_endpoint_diagnostics.pytest_custom_model_endpoint_synthetic_streaming.pytest_custom_model_endpoint_on_prem.pytest_custom_model_endpoint_auth.pytest_custom_model_endpoint_provider.pyAlso green: the workspace, admin-persistence, cloud-environment, Claude, Foundry-streaming, workflow-capabilities, tabular, docs-integrity, and route-policy suites.
Eleven other
*endpoint*tests fail, along with eight relative docs links and the exact-version assertions in the tabular export tests. Each was baselined at the restore commit and fails identically there, so none are introduced here.Documentation: five feature pages, one fix page, and release-notes entries for every version in the stack.
Refs #1228, #1431