test(evals): eval suite for launchdarkly-metric-create#97
Draft
sruthykumar wants to merge 1 commit into
Draft
Conversation
First metrics eval in the main harness (supersedes the lighter tests/evals/metrics/metric-create.eval.yaml). Five cases by the three-lens method, grounded in usage data (click 46% / pageview 45% / custom 9%): the golden path is a pageview or click metric, not custom. Adds shared metric tool defs + mocks (create-metric, get-metric, list-metrics, list-metric-events, get-environment), manifest entry, and npm scripts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the first metrics eval in the main harness, for the launchdarkly-metric-create skill. Supersedes the lighter
tests/evals/metrics/metric-create.eval.yaml(raw-text-into-Haiku, rubric-only, no tool mocking) with a proper mocked-tool + trajectory suite.Grounded in usage data (Snowflake GOALS: click 46% / pageview 45% / custom 9%), so the frequency-weighted golden path is a pageview or click metric, not custom — the case the prior thin eval underweighted.
Five cases (three-lens method):
Shared infra (new metric tool support in the harness): adds
create-metric,get-metric,list-metrics,list-metric-events,get-environmenttool defs + mocks, manifest entry, and npm scripts.Testing
ANTHROPIC_API_KEY; CI (ornpm run eval:metric-create) will execute it. Expect some assertion tuning on the first live run.🤖 Generated with Claude Code
via LD Research 🤖