chore(aio): update online-eval skill to unpinned provider key - #70447
Merged
Radu-Raicea merged 4 commits intoJul 27, 2026
Conversation
The evaluation API and MCP tool now require a valid provider and model for an llm_judge eval, while provider_key_id may be null (unpinned). The creating-online-evaluations skill still gated on an ok-state provider key and told the agent to stop when none existed, which blocks creating valid unpinned llm_judge evaluations. Reframe the gate around provider + model, mark pinning a provider key as optional across the table, gate section, example payloads, tips, and the payload reference. Generated-By: PostHog Code Task-Id: e2717ec9-87dc-4d47-b6c1-6433bd1a87e0
posthog
Bot
added
the
skip-agent-review
Re-align the tool and eval-type tables with oxfmt so the markdown format check passes. Generated-By: PostHog Code Task-Id: e2717ec9-87dc-4d47-b6c1-6433bd1a87e0
Contributor
|
This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the |
Radu-Raicea
marked this pull request as ready for review
July 27, 2026 17:13
Radu-Raicea
approved these changes
Jul 27, 2026
Radu-Raicea
enabled auto-merge (squash)
July 27, 2026 17:14
Radu-Raicea
deleted the
posthog-code/fix-online-eval-skill-provider-key-gate
branch
July 27, 2026 17:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The evaluation API and MCP tool were changed so an
llm_judgeevaluation requires a validproviderandmodel, whileprovider_key_idmay benull(see #70113). Thecreating-online-evaluationsskill was still written against the old contract: it gatedllm_judgecreation on anok-state provider key and told the agent to stop and send the user to the UI when none existed. That stale gate can stop an agent from creating a perfectly valid unpinnedllm_judgeevaluation.Changes
Reframe the gate around what the API now actually requires: a valid
provider+model, with pinning a provider key as an optional step (provider_key_idmay benull).SKILL.md: updated the frontmatter description, the tools table, the eval-type table, the renamed "Pick a provider and model" gate section, thellm_judgeexample payload, and the tips checklist.references/evaluation-payload.md.j2:provider_key_idis now optional (was marked required), and the LLM-judge example usesprovider_key_id: null.Docs-only change to an agent skill. No production code touched.
How did you test this code?
No automated tests apply — this only edits a skill markdown file and its Jinja reference template. I verified the new wording against the actual contract in commit
4a3274caf3(products/ai_observability/backend/api/evaluations.pyandproducts/ai_observability/mcp/tools.yaml):model_configurationrequiresproviderandmodel, andprovider_key_idis nullable.Automatic notifications
🤖 Agent context
Autonomy: Fully autonomous
Authored by Claude Code (Opus 4.8) acting on an inbox report flagging the stale provider-key gate. No repo skills needed invoking for a docs-only skill edit. I extended the fix to the linked
references/evaluation-payload.md.j2, which had the same staleprovider_key_id: requiredcontract, so the skill and its reference stay consistent.Created with PostHog Code from an inbox report.