feat(llm): apikey_header_only auth scheme + record invoked LLM URL on llm_call

[initializ-mk]

Two related gateway-debuggability changes, motivated by an agent behind a Kong AI Gateway 401ing on invalid x-api-key.

1. auth_scheme: apikey_header_only (gateway header, native header suppressed)

apikey_header (#302) sends the provider-native header (x-api-key / Authorization) in addition to the gateway header. That is correct when the gateway replaces the upstream credential (Kong request-transformer replace, or ai-proxy with allow_override: true) or the key passes through. But when the gateway injects the native header with an add-if-absent transform (Kong request-transformer add), Forges native header blocks the injection, so the gateway key reaches the provider and it 401s.

New model.auth_scheme: apikey_header_only sends only the gateway header and suppresses the provider-native header, mirroring how aws_sigv4 already skips it. The gateway becomes the sole injector of the real upstream credential.

• Native x-api-key (anthropic) / Authorization: Bearer (openai) suppressed for the new scheme; setGatewayAPIKeyHeader fires for both gateway schemes.
• apikey_header stays additive and unchanged — no behavior change or broken tests for existing deployments (the LLM client: apikey-header auth scheme for Kong AI Gateway (key-auth) #302/feat(llm): apikey_header auth scheme for Kong AI Gateway key-auth #303 additive contract and its tests are preserved).
• forge validate accepts the new scheme; the native-header-collision guard and auth_header_name applicability extend to it via a shared isGatewayScheme helper.

Pick apikey_header when the gateway replaces the native header; apikey_header_only when it adds.

2. Record the invoked LLM URL on llm_call (payload-capture independent)

The llm_call audit event recorded model/provider/tokens but not the URL, so a base-URL/gateway misroute (e.g. ANTHROPIC_BASE_URL pointing at Kong) was invisible unless you enabled payload capture, and a hard 401 emitted no llm_call at all.

• llm.ChatResponse gains an internal Endpoint field (json:"-"), set by the openai + anthropic clients to the URL they POSTed to ({base}/v1/messages, {base}/chat/completions).
• registerAuditHooks records fields.url on every llm_call, independent of payload capture — the URL is header-authed metadata and carries no secret.
• Provider errors now name the invoked URL (... calling <url>: ...), so a 401 misroute is debuggable straight from the agent loop error log even though no llm_call fires.

Tests

• apikey_header_only: native header suppressed + gateway header set (openai + anthropic); custom auth_header_name honored; validate accepts the scheme + collision guard.
• Endpoint recorded on success (openai + anthropic); error string includes the URL on a 401.

Docs

• docs/reference/forge-yaml-schema.md, docs/core-concepts/runtime-engine.md, .claude/skills/forge.md — the new scheme + additive-vs-suppress guidance.
• docs/security/audit-logging.md — the llm_call fields.url.

forge-core + forge-cli build/vet/lint clean; llm + validate suites green.

[initializ-mk]

Review: apikey_header_only scheme + record invoked LLM URL

Change 1 (auth scheme) is clean and correct. Change 2 (URL recording) is a genuinely useful debugging aid with one security issue and one coverage gap. Merge-ready after finding 1.

Change 1 — apikey_header_only: correct, no regression ✅

• Native-header suppression is symmetric and right: anthropic suppresses x-api-key, openai suppresses Authorization: Bearer for apikey_header_only (joining aws_sigv4), while setGatewayAPIKeyHeader fires for both gateway schemes.
• apikey_header stays additive and untouched — no behavior change for existing #302/#303 deployments.
• Validate wiring is tidy: isGatewayScheme unifies the native-header-collision guard and auth_header_name applicability, the new scheme is added to knownModelAuthSchemes, and the provider warning is extended. The key rides in exactly one header.

Change 2 — findings (see inline)

1. Low/Medium (security): the recorded URL isn't sanitized, so a base URL with userinfo (https://user:pass@gateway) leaks a credential into the audit stream (fields.url) and agent loop error logs. The "carries no secret" comment holds only for a clean base URL — and userinfo-in-URL is a legitimate gateway config, i.e. exactly this PR's audience.
2. Low (completeness): only non-streaming Chat sets Endpoint; the streaming path doesn't, so streaming llm_call events (the common agent path) carry no fields.url — which is where you'd most want it during a gateway misroute.

Also good

• Endpoint string json:"-" is the right call — internal-only, not serialized into response payloads, explicitly surfaced into the audit field.
• Naming the URL in the error string genuinely fixes the hard-401 case (no llm_call fires there), the motivating bug.
• CI fully green; tests + docs included.

[initializ-mk]

Low/Medium (security): sanitize before recording. The comment above says the URL "carries no secret," but that's only true for a clean base URL. Endpoint = c.baseURL + "/path", and baseURL comes from model.base_url / ANTHROPIC_BASE_URL / OPENAI_BASE_URL. A gateway configured with userinfo — https://user:pass@gateway.internal (a legitimate setup; Go's http.NewRequestWithContext extracts it into basic auth) — makes Endpoint = https://user:pass@gateway.internal/v1/messages, and that password then lands in the audit stream here (typically shipped to SIEM + retained) and, via the provider error strings (anthropic.go / openai.go ... calling <endpoint>: ...), in agent loop error logs.

Strip userinfo before recording, in one shared spot ideally:

safe := endpoint
if u, err := url.Parse(endpoint); err == nil {
    u.User = nil          // or u.Redacted() to mask the password
    safe = u.String()
}

Apply the same to the two provider error strings. That turns "carries no secret" from an assumption into an invariant.

[initializ-mk]

Low (completeness): the streaming path never sets Endpoint. Chat sets result.Endpoint here, but ChatStream doesn't — its aggregated ChatResponse builder (anthropic.go:371, and openai's at openai.go:281) omits it, and ChatStream even re-inlines c.baseURL+"/v1/messages" rather than reusing an endpoint var. So streaming llm_call events carry no fields.url — and streaming is the common path for interactive agents, exactly where you'd want the URL when a gateway misroutes. Hoist the URL into an endpoint var in ChatStream like Chat does and assign it on the aggregated response. Same in the openai client. (Whatever sanitization lands for finding 1 should cover this path too.)

[initializ-mk]

Addressed both findings in 3c3c746:

Finding 1 (Low/Medium security — sanitize userinfo). Added a shared sanitizeEndpoint helper that strips user:pass@ from the URL. The request still uses the raw URL (so userinfo becomes Basic auth as before), but every recorded/logged form is sanitized: ChatResponse.Endpoint (→ llm_call fields.url) and the provider error strings in both Chat and the streaming paths (anthropic + openai + responses). "Carries no secret" is now an invariant, not an assumption. Tested: a https://user:secret@… base URL yields an Endpoint and an error with no secret/user: in them, plus a sanitizeEndpoint table (incl. an unparseable → unchanged case).

Finding 2 (Low completeness — streaming). You were right that an aggregated ChatResponse was missing it — the real one is ResponsesClient.Chat (responses.go), which aggregates a stream into a ChatResponse and is the OpenAI OAuth executor path. It now sets Endpoint, so OAuth llm_call events carry fields.url like the key-based path. The ChatStream methods (anthropic/openai/responses) also stop re-inlining c.baseURL+"/…" — hoisted into an endpoint var and named (sanitized) in their error strings.

Worth noting on scope: Forges executor (loop.go) only ever calls client.Chat (non-streaming) — ChatStream returns a <-chan StreamDelta with no ChatResponse to stamp and isnt used in the agent loop, so there are no client-level streaming llm_call events to miss. The one place streaming is aggregated into a ChatResponse is ResponsesClient.Chat, now covered. OllamaClient inherits the fix (it embeds *OpenAIClient).

llm suite green; vet + lint clean.

[initializ-mk]

Re-review: fix commit 3c3c746d — both findings resolved

Verified against the diffs and the audit data-flow. Build / Lint / Test / Doc-link green (Integration still running — unrelated; this diff is llm provider-only).

Finding 1 (userinfo leak) — ✅ resolved, cleanly

New shared sanitizeEndpoint does url.Parse → u.User = nil → u.String(), stripping the entire userinfo (stronger than Redacted(), which keeps the username). The separation is exactly right: the HTTP request still uses the raw endpoint (so user:pass@ becomes a working Basic auth header), while every recorded/logged form uses the sanitized one:

• result.Endpoint (→ fields.url) in openai/anthropic Chat and ResponsesClient.Chat
• all provider error strings, including the ChatStream error paths (which now also name the sanitized URL — a real debuggability win even though they don't feed llm_call)
• OllamaClient inherits it via embedded *OpenAIClient

Sanitizing at the source rather than only in the audit hook is the better call — it covers the error-log vector too. The url.Parse-error → return-raw fallback is a non-issue in practice (url.Parse doesn't error on http(s) URLs with userinfo).

Finding 2 (streaming Endpoint) — ✅ resolved at the correct path

I traced the audit path end-to-end: the executor loop calls e.client.Chat and fires AfterLLMCall with that response, so llm_call is only ever fed by .Chat. The one streaming path reaching it is ResponsesClient.Chat (OAuth/Responses, which streams + aggregates internally), and the fix sets Endpoint there. My original inline anchor pointed at the plain ChatStream aggregate, but that response never feeds llm_call — so this diagnosed the real audit-feeding path more precisely than my comment and fixed exactly it. Streaming llm_call events now carry fields.url on the path that actually emits them.

Verdict: both findings resolved, sanitization applied at every sink, streaming coverage on the genuinely correct path. Nothing dangling — merge-ready.

[initializ-mk]

Added the actual root cause fix in 62cd126 (per the deployment forensics): a gateway base URL never reached the model client, so the anthropic client fell back to api.anthropic.com and sent the gateway key as x-api-key — the exact 401. Two silent drops, both fixed:

1. os.Environ never merged into envVars. The runner built the model-resolution env map from only the .env file + the secret overlay, and the overlay fetches only secret keys (EnvProvider.List() returns nil by design). So non-secret model-config keys injected into a deployed pod — ANTHROPIC_BASE_URL / OPENAI_BASE_URL / OLLAMA_BASE_URL / FORGE_MODEL_PROVIDER / MODEL_NAME / OPENAI_ORG_ID / AWS_REGION / FORGE_MODEL_FALLBACKS — were invisible to ResolveModelConfig (while LLM_API_KEY, a builtin secret key, flowed in fine). New mergeModelConfigEnv fills them from the process env as a fallback; .env / secrets still win.

2. model.base_url in forge.yaml was never copied to the client — it only fed the build-time egress allowlist. ResolveModelConfig now applies cfg.Model.BaseURL (before the env overrides, so *_BASE_URL still wins).

This means any deployed agent using a gateway via the anthropic provider was broken the same way, and the platform’s OPENAI_BASE_URL/ANTHROPIC_BASE_URL deploy-injection contract now actually routes the client. Combined with the apikey_header_only scheme and the llm_call fields.url already in this PR, the full gateway path — reach Kong, auth cleanly, and see the real target in the audit stream — works end to end.

Tests: base_url-from-yaml + env-override precedence; mergeModelConfigEnv fills from os env, .env wins, secret keys excluded. Build/vet/lint clean; forge-core runtime/llm + forge-cli runtime suites green.

[initializ-mk]

Review: commit 62cd1265 — gateway base URL now reaches the client

This is the root-cause fix behind the original 401 (the earlier commits were debuggability + the apikey_header_only workaround; this makes the base URL actually take effect). Correct, well-scoped, properly tested. No security concern. Full CI green.

Why it's safe

1. mergeModelConfigEnv fills model-config keys from os.Environ as a fallback, but iterates a fixed 8-key allowlist (*_BASE_URL, FORGE_MODEL_PROVIDER, MODEL_NAME, OPENAI_ORG_ID, AWS_REGION, FORGE_MODEL_FALLBACKS) — not the whole environment, so no env-surface broadening, and all eight are non-secret config.
2. model.base_url is now copied to mc.Client.BaseURL (previously it only fed the build-time egress allowlist).

Precedence is correct and the tests pin every edge:

• mergeModelConfigEnv skips keys already in envVars, so .env/secrets win over os.Environ.
• Secret keys are explicitly excluded — the test asserts ANTHROPIC_API_KEY does not flow this path (stays with overlaySecrets). This is the key safety invariant and it's locked.
• model.base_url is applied before the *_BASE_URL env override, so per-deploy env wins over yaml (tested both directions).
• The CLI --model override still beats os.Environ MODEL_NAME (merge runs before the override block).

Nice synergy

This closes the loop on the PR: base_url now routes the client, and the sanitized fields.url audit from 3c3c746d records where it went — so a misroute is both fixable and visible. And since the key now reaches the configured gateway instead of the provider's public host, it complements apikey_header_only correctly.

Minor behavior-change note (not a finding)

Env now overrides yaml base_url including from the ambient shell — a stray OPENAI_BASE_URL/ANTHROPIC_BASE_URL in a dev's local shell will silently redirect the client where before it was ignored. Standard 12-factor env-over-config convention and defensible, and the new audit URL surfaces it if it happens — just worth being aware it's a local-dev behavior change, not only deployed pods.

Verdict: correct root-cause fix, scoped allowlist, secret-exclusion tested, precedence tested, CI green. No issues.