fix: AI Proxy fixes

[michaeljguarino]

• bumps bifrost + some api refactors as a result

Fixes PROD-4851

Test Plan

Test environment: https://console.plrldemo.onplural.sh/cd/clusters/a1748282-ce8b-48ab-ae7e-326e74fce04e/services/f3f89a54-d1a7-4bc8-9152-daa07ede918d/settings/secrets

Checklist

• I have added a meaningful title and summary to convey the impact of this PR to a user.
• If required, I have updated the Plural documentation accordingly.
• I have added tests to cover my changes.
• I have deployed the agent to a test environment and verified that it works as expected (required only when changing agent code).
  • Agent starts successfully.
  • Service creation works without any issues when using raw manifests and Helm templates.
  • Service creation works when resources contain both CRD and CRD instances.
  • Service templating works correctly.
  • Service errors are reported properly and visible in the UI.
  • Service updates are reflected properly in the cluster.
  • Service resync triggers immediately and works as expected.
  • Sync waves annotations are respected.
  • Sync phases annotations are respected. Phases are executed in the correct order.
  • Sync hook delete policies are respected. Resources are not recreated once they reach the desired state.
  • Service deletion works and cleanups resources properly.
  • Services can be recreated after deletion.
  • Service detachment works and keeps resources unaffected.
  • Services can be recreated after detachment.
  • Service component trees are working as expected.
  • Cluster health statuses are being updated.
  • Agent logs do not contain any errors (after running for at least 30 minutes).
  • There are no visible anomalies in Datadog (after running for at least 30 minutes).

Plural Flow: console

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	github.com/​golangci/​golangci-lint/​v2@​v2.11.4

View full report

[greptile-apps]

Greptile Summary

This PR fixes the AI Proxy (Nexus) by bumping Bifrost and refactoring the OpenAI routing layer to support CHAT/RESPONSES/AUTO method selection with cross-format fallbacks, and extends the deployment-operator harness with proxy model prefixing, OpenAI-compatible custom endpoints, and Docker-in-Docker bash permission improvements.

• Nexus routing: Adds OpenAiMethod proto enum and wires it through gRPC → openAIHTTPPolicyFromConfig → Bifrost AllowedRequests; implements chat↔responses cross-format fallbacks for both streaming and non-streaming paths, with raw-response passthrough disabled during translation.
• Base URL normalization: Extracts bifrostNetworkBaseURL to strip the /v1 suffix Console clients include, and a matching openai_base_url/1 in the Elixir Nexus provider to avoid double-appending /v1.
• Deployment operator: Centralizes proxy model prefixing in the new model.ProxyModel helper, adds OpenCodeOpenAICompatibleConfig for custom OpenAI-compatible endpoints, strengthens secret ref validation with secretKeySelectorSet, and makes API-key fields omitempty so aiProxy runs without a key secret.

Confidence Score: 4/5

Generally safe to merge; one stream-cancellation gap in the cross-format fallback path could leave upstream Bifrost streams open longer than needed when a client disconnects mid-stream on the responsesViaChat path.

The responsesViaChat streaming branch in handleStreaming logs write errors from writeChatStreamAsResponses but does not call cancel() or return, unlike every other write-error site in that function. If a client disconnects during a long multi-chunk response on this path, the upstream stream stays running and each subsequent chunk re-enters the branch, fails, and is logged until the channel drains. All other routing paths are well-tested and the change is otherwise well-structured.

go/nexus/internal/router/router.go — the responsesViaChat streaming branch around line 409

Important Files Changed

Filename	Overview
go/nexus/internal/router/router.go	Added responsesViaChat cross-format handling and releaseChatToResponsesStreamState defer; the responsesViaChat streaming path doesn't call cancel() on write errors, unlike all other write-error paths in this function.
go/nexus/internal/router/openai_method.go	New file implementing HTTP policy for OpenAI method routing (CHAT/RESPONSES/AUTO) with cross-format fallback via BifrostContext keys; logic and tests look correct.
go/nexus/internal/router/openai_routing_stream.go	New file wrapping chat→responses stream conversion with pool state management; writeStreamSSE duplicates SSE-writing logic from router.go but is otherwise correct.
go/nexus/internal/router/openai_raw_response.go	New helper that gates raw-response passthrough when cross-format fallbacks are active; correctly skips passthrough for chatViaResponses and responsesViaChat paths.
go/nexus/internal/router/openai_allowed_requests.go	Configures Bifrost AllowedRequests gating to match the configured OpenAI method; clean and well-tested.
go/nexus/internal/router/account_keys.go	Migrated Bedrock and Azure deployments from KeyConfig.Deployments to Key.Aliases, matching updated bifrost API.
go/nexus/internal/router/provider_base_url.go	Strips trailing /v1 from Console-style base URLs for Bifrost compatibility; well-tested including edge cases.
lib/console/ai/provider/nexus.ex	Extracted openai_base_url/1 helper to avoid double-appending /v1; tests verify no-double-append and trailing-slash cases.
lib/console/grpc/server.ex	Added openai_method_to_pb/1 to serialize the new method field; nil maps to :AUTO which falls through the same default case as UNSPECIFIED in Go.
go/deployment-operator/api/v1alpha1/agentruntime_types.go	Added OpenAICompatible config block for custom endpoints, aiProxy helpers, secretKeySelectorSet guard, and omitempty on raw API-key fields; changes are backwards-compatible.
go/deployment-operator/pkg/agentrun-harness/model/proxy.go	New package providing ProxyProvider and ProxyModel helpers; replaces ad-hoc openai/ prefix logic scattered across tool packages.
go/deployment-operator/pkg/agentrun-harness/tool/opencode/opencode_types.go	Refactored EnsureProvider to pass through arbitrary provider slugs, added ProviderOpenAICompatible constant, and extracted resolveOpenCodeSettings for cleaner provider/model wiring.
go/deployment-operator/pkg/agentrun-harness/tool/opencode/templates/opencode.json.gotmpl	Added DindEnabled bash permission toggle, optional baseURL for non-plural providers, and openai-compatible npm block; template conditionals produce valid JSON in all branches.

_{Reviews (2): Last reviewed commit: "better ai proxy documentation" | Re-trigger Greptile}

[michaeljguarino]

plural deploy this, and use modify the tag for the ai proxy too alongside it

[linear]

PROD-4851

[michaeljguarino]

plural deploy this, and also modify the ai proxy tag alongside it

[michaeljguarino]

plural deploy this, alonside the ai proxy as well

[michaeljguarino]

@greptileai

[greptile-apps]

Stream not cancelled on write error in responsesViaChat path

When writeChatStreamAsResponses returns an error — typically because the client disconnected and a write to w failed — cancel() is not called, so the upstream Bifrost stream keeps running. Every subsequent chunk in the channel will re-enter this branch, fail again, and be logged, burning server resources until Bifrost drains the whole response. The parallel non-fallback write paths both call cancel() and return on write failures (lines 451, 466). This path should do the same.

[michaeljguarino]

there's a defer cancel at the top of the function

[michaeljguarino]

plural deploy this, and the nexus proxy image too

[michaeljguarino]

plural deploy this, and the ai proxy tag too

[michaeljguarino]

plural deploy this, plus the ai proxy tag

[michaeljguarino]

plural deploy this

[michaeljguarino]

plural deploy this

[floreks]

Mostly LGTM. I am not fully sure of the impact of AI proxy changes. I have deployed and checked this version and opencode harness no longer receives errors from Nexus on unsupported model configuration. It just passed without providing any messages.

[michaeljguarino]

yea I'm not positive as well, we should probably merge it and work on opencode separately in another pr so this doesn't back up any further.