AuthBridge injection emits no_proxy with embedded newline → httpx "Invalid non-printable ASCII character in URL" breaks all agent egress

[webchang]

Summary

The AuthBridge injection webhook (inject.rossoctl.io, rossoctl-operator 0.3.0-rc.1) injects the lowercase no_proxy env var into the agent container with an embedded literal newline + indentation (\n ) in the middle of the value. Because the agent's HTTP client is httpx (which reads no_proxy via trust_env), httpx rejects the value and every outbound request from the agent fails before it leaves the pod — including MCP tool calls and LLM calls.

Symptom (user-facing)

Weather-agent chat fails with:

Error: Cannot connect to MCP weather service at
http://weather-tool-mcp.team1.svc.cluster.local:8000/mcp.
Please ensure the weather MCP server is running.
Error: Invalid non-printable ASCII character in URL, '\n' at position 37.

The MCP tool is actually healthy (returns 401 missing Authorization header to an unauthenticated probe) — the failure is purely client-side URL/proxy parsing.

Root cause

Inside the running agent container, os.environ['no_proxy'] (lowercase) contains a stray newline:

no_proxy = '...,otel-collector.rossoctl-system\n  .svc.cluster.local,keycloak.keycloak.svc.cluster.local'
#                                              ^^^^  literal "\n  " (newline + 2 spaces)

httpx's trust_env proxy handling parses no_proxy, and the \n triggers
httpx.InvalidURL: Invalid non-printable ASCII character in URL, '\n' at position 37
when it builds its proxy mounts. The uppercase NO_PROXY is clean, but httpx reads the
lowercase variant too, so it still fails.

The \n (newline + two spaces) is characteristic of a YAML block-scalar / multi-line
template join in the operator's env-composition code — the base no-proxy list and the
appended cluster hosts (.svc.cluster.local,keycloak.keycloak.svc.cluster.local) are being
concatenated with a newline instead of a comma.

Environment

• Operator: ghcr.io/rossoctl/operator/rossoctl-operator:0.3.0-rc.1
• Webhook: rossoctl-operator-chart-mutating-webhook-configuration → inject.rossoctl.io
• Cluster: OpenShift (HyperShift hosted), Istio ambient
• Injected container: agent (alongside the authbridge-proxy sidecar)

Reproduction

1. Deploy any agent that gets the AuthBridge sidecar injected (e.g. weather-service in team1).
2. kubectl exec <agent-pod> -c agent -- /app/.venv/bin/python -c "import os; print(repr(os.environ['no_proxy']))"
3. Observe the embedded \n in the value.
4. Any httpx call from the agent (MCP tool, LLM) fails with the InvalidURL error above.

Expected

The injected no_proxy/NO_PROXY must be a single-line, comma-separated string with no
newlines or embedded whitespace between entries.

Workaround (applied manually)

Override the env on the deployment with a clean single-line value; it survives webhook
re-injection because the container already defines the var:

CLEAN='127.0.0.1,localhost,<external-llm-host>,.svc,.svc.cluster.local,.cluster.local,<tool-host>,keycloak.keycloak.svc.cluster.local,otel-collector.rossoctl-system'
kubectl set env deployment/<agent> -n <ns> -c agent no_proxy="$CLEAN" NO_PROXY="$CLEAN"

Suggested fix

In the operator's proxy-env composition, join no-proxy entries with , and never emit a
YAML block scalar / newline. Add a guard/test asserting the composed value matches
^[^\s]+(,[^\s]+)*$ (no whitespace) before setting it on the container.

Related

• proxy-sidecar: NO_PROXY too restrictive — breaks K8s API and external endpoints #384 (proxy-sidecar: NO_PROXY too restrictive) — same injection path, different symptom
  (over-restriction vs. malformed value). Fixing both in the composition code makes sense.

---

Assisted-By: Claude Code

[webchang]

Closing — misdiagnosis on my part. The embedded newline in no_proxy did NOT come from the operator injection. Root cause was an operator-external kubectl set env command whose NO_PROXY/no_proxy value was split across two shell lines, baking a literal \n (+ indent) into the env var. Confirmed by an exact character-for-character match with the failing pod value, including a deployment-specific external LLM host the operator never sees.

The operator does inject HTTPS_PROXY=http://127.0.0.1:8081 and the authbridge-proxy sidecar, but it did not introduce the newline. No operator change is warranted from this report. Apologies for the noise.

(Unrelated #384 — NO_PROXY too restrictive — remains valid and stays open.)

Assisted-By: Claude Code