Skip to content

fix(test): pin autoevals judge to the OpenAI API - #617

Merged
aidandaly24 merged 1 commit into
mainfrom
fix/autoevals-openai-client
Aug 4, 2026
Merged

fix(test): pin autoevals judge to the OpenAI API#617
aidandaly24 merged 1 commit into
mainfrom
fix/autoevals-openai-client

Conversation

@jariy17

@jariy17 jariy17 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #614, which took the evaluation group from 6 failures to 3. The three DeepEval tests now pass — confirming both the span-fixture fix and the OPENAI_API_KEY wiring work. The three remaining AutoEvalsAdapter tests fail with:

openai.AuthenticationError: Braintrust gateway error: auth failed
  [401 Unauthorized]: Invalid API Key ... org: None

The key is valid — it was being sent to the wrong service. autoevals does not call OpenAI directly by default (autoevals/oai.py):

if base_url is None:
    base_url = os.environ.get("OPENAI_BASE_URL") or _gateway_url()   # https://gateway.braintrust.dev
    if _is_gateway_url(base_url):
        api_key = os.environ.get("BRAINTRUST_API_KEY") or os.environ.get("OPENAI_API_KEY")

With OPENAI_BASE_URL unset it routes to the Braintrust AI gateway, which rejects an OpenAI key. This is documented behaviour, not a bug — init()'s own docstring example pairs the gateway base_url with BRAINTRUST_API_KEY.

Fix

Pass an explicit client, which is the provider-selection path autoevals documents. Its api_key / base_url arguments are marked "Deprecated: Use the client argument", so this is the forward-compatible option rather than setting OPENAI_BASE_URL in CI.

The client is built once in a fixture; each test passes it to the scorer, keeping the provider choice visible at the call site.

Scoped to the Autoevals tests only — DeepEval is untouched and already passing.

Testing

# an injected client bypasses the gateway: probe with a deliberately invalid key
Error code: 401 - Incorrect API key provided: sk-fake-*robe ...
#   ^ OpenAI's own error, no Braintrust in the message (previously: "Braintrust gateway error")

ruff format --check  -> 1 file already formatted
ruff check           -> All checks passed!

I could not run these tests against the real key locally: the value lives only in the central DevX Secrets Manager account, and secretsmanager:GetSecretValue on it is granted solely to DevXWorkflowSecretsReader-SDKPython for the CI OIDC subjects — my ReadOnly role is denied by design. So the routing fix is verified structurally, and the first true green run for these three tests will be in CI.

Reviewer note

safety-gate does not apply here — this touches no workflow files, so the integration tests should run automatically on this PR.

The autoevals default judge model is gpt-5-mini. If the provisioned key's account lacks access to it, expect a model 404 rather than a 401; that would be a key-provisioning follow-up, not a code issue.

#614 fixed the span fixture and wired OPENAI_API_KEY, taking the
evaluation group from six failures to three. The three remaining
AutoEvalsAdapter tests fail with:

  Braintrust gateway error: auth failed [401 Unauthorized]: Invalid API Key

autoevals does not call OpenAI directly by default. When base_url is
unset it resolves to the Braintrust AI gateway and prefers
BRAINTRUST_API_KEY over OPENAI_API_KEY, so our OpenAI key is presented to
a service that does not accept it. The key is valid; it was being sent to
the wrong endpoint.

Passing an explicit client is the provider-selection path autoevals
documents; its api_key and base_url arguments are marked deprecated in
favour of the client argument. The client is built once in a fixture and
each test passes it to the scorer, so the provider choice stays visible
at the call site.

Scoped to the Autoevals tests. DeepEval is unaffected and already passes,
since it reaches OpenAI directly.

Verified: an injected client bypasses the gateway; a probe with a
deliberately invalid key returns OpenAI's own 401 ("Incorrect API key
provided") with no Braintrust in the message. ruff format and ruff check
are clean.
@jariy17
jariy17 requested a review from a team August 4, 2026 18:56
@github-actions github-actions Bot added the size/s PR size: S label Aug 4, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 4, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 4, 2026
@aidandaly24
aidandaly24 merged commit 703ccfd into main Aug 4, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/s PR size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants