feat(llma): add ai-failures and online-evals skills - #67707
Merged
carlos-marchal-ph merged 4 commits intoJul 3, 2026
Conversation
Contributor
|
Reviews (1): Last reviewed commit: "feat: add failures and evals skills" | Re-trigger Greptile |
Contributor
|
Size Change: 0 B Total Size: 70.2 MB ℹ️ View Unchanged
|
Contributor
Query snapshots: Backend query snapshots updatedChanges: 16 snapshots (16 modified, 0 added, 0 deleted) What this means:
Next steps:
|
Radu-Raicea
approved these changes
Jul 2, 2026
|
|
||
| For each trace, note in plain language what went wrong — and jot down the trace's earliest-event timestamp | ||
| alongside the note (it's right there in the trace you just read, and in `query-llm-traces-list`'s | ||
| `first_timestamp`). That timestamp and the trace ID is all you need to build a resolvable deep link in Step 4, |
Member
There was a problem hiding this comment.
Suggested change
| `first_timestamp`). That timestamp and the trace ID is all you need to build a resolvable deep link in Step 4, | |
| `createdAt`). That timestamp and the trace ID is all you need to build a resolvable deep link in Step 4, |
| "name": "Output is valid JSON", | ||
| "description": "Fails when the assistant message can't be parsed as JSON", | ||
| "evaluation_type": "hog", | ||
| "evaluation_config": { "source": "try { jsonParseStr(event.properties.$ai_output_choices[1].content); return true; } catch { return false; }" }, |
Member
There was a problem hiding this comment.
Doesn't the Hog function get access to a normalized output? Doing event.properties.$ai_output_choices[1].content seems wrong.
Generated-By: PostHog Code Task-Id: c43291cb-8f62-4c94-82f2-ed8cbc999b97
carlos-marchal-ph
deleted the
posthog-code/aiobs-creating-online-evals-skill
branch
July 3, 2026 08:13
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds two new skills, bundled in a single commit because they have a dependency and I tested them together:
exploring-ai-failuresexplores the failure space of AI traces and sample and understands what goes wrong with existing traces, creating a taxonomy of failures (based on current best practices)creating-online-evaluationssets up new evals in an evidence based manner, that fit the user's domain. Leverages theexploring-ai-failuresskill when the user is not clear on what to focus the evaluation onIt additionally bundles a fix that prevented the evaluation creation tools from working by being too strict with UUID input validation.
Sample run of the skill, which I used to create this eval and this other eval:
How did you test this code?
Automated tests I actually ran:
stripUuidFormatinschema-transforms.test.ts, including the nullable array-formtype: ["string","null"]case that this fix targets. Ran vitest — all pass.hogli build:openapi-schema+hogli build:openapi-mcpand confirmedprovider_key_idnow generates as.string(), with the diff limited to.uuid()→.string()on nullable fields.tsgo --noEmit) passes; oxlint clean on the changed source.The skills themselves are markdown plus a Jinja reference template, so there's no runtime path to exercise beyond authoring.
Automatic notifications
Docs update
Skills only, no posthog.com docs change.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
I drove this; Claude Code (Opus 4.8) did the code. The skills were authored first (this branch's original
wip). The MCP fix came out of debugging why pinning a provider key failed during a spot-run of the eval skill.Claude root-caused the rejection to
stripUuidFormat's exact-stringtypecheck missing the nullable array form, rather than to the serializer. It chose to relocate the transform into the sharedschema-transforms.mjslib (next tostripEnumMinLength) instead of patching it inline, so it could be unit-tested. Skills invoked:/writing-tests(gate for the new unit test).