docs: require running added/changed eval cases with baseline#237
Merged
Conversation
Previously the eval guidance said running evals was "recommended but not required" for improvements, and told contributors to skip running entirely when changes were "purely additive". That conflated two distinct things: 1. Running the case you just added or changed — this is code you wrote, and an unrun case can be silently mis-scoped (e.g. a prompt that says "just the command" paired with a behavior that expects an explanation). It must be run to confirm it passes with the skill and shows a real baseline delta. 2. Re-running the untouched existing suite as a regression check — this stays conditional on whether you modified content those cases cover. Update both `.claude/CLAUDE.md` (PR eval requirements + upstream-sync checklist) and the `improve-ic-skill` skill (Step 7) to make (1) mandatory with baseline and keep (2) conditional.
Skill Validation ReportNo skill files were changed in this PR — validation skipped. |
lwshang
approved these changes
Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tightens the eval-running policy for skill improvements. The current guidance is inconsistent: it says running evals is "recommended but not required" and tells contributors to skip running entirely when changes are "purely additive". That conflates two genuinely different things:
This surfaced during the mops-cli upstream sync (#236): a newly added eval was mis-scoped and only caught because the case was actually run with baseline.
Changes
.claude/CLAUDE.md— rewrote the Skill improvement bullet under PR eval requirements and the Evals reviewed item in the upstream-sync checklist to make (1) mandatory-with-baseline and keep (2) conditional..agents/skills/improve-ic-skill/SKILL.md— restructured Step 7 into "A. always run cases you added/changed (with baseline)" vs "B. re-run untouched existing cases only when relevant", and updated the command examples so--no-baselineis framed as a quick spot-check, not for PR results.Docs-only; no skill content or site changes.
🤖 Generated with Claude Code