Ship evidence-first skill security and CI gates - #11
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR evolves Skillbench into an “evidence-first” workflow by separating portable skill validation from strict authoring lint, adding a static security audit plus repository CI gate (and reusable GitHub Action), and expanding task A/B evaluation into repeated counterbalanced challenges with richer ROI metrics and reporting.
Changes:
- Add static skill-package security auditing (with suppressions) and enforce it in registry publish/install, CI gate, and a composite GitHub Action.
- Split
validate(portable) fromlint(opinionated authoring requirements) and introduce a repo-widecheck/cigate with discovery of conventional skill roots. - Upgrade task evaluation to support repeated AB/BA runs, runtime skill package copying without
evals/leakage, additional rubric types, and updated CLI/TUI + website/docs.
Reviewed changes
Copilot reviewed 37 out of 46 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/validate.test.ts | Adds coverage for validate-vs-lint split and task rubric leak warning. |
| tests/task-eval.test.ts | Extends rubric scoring tests; validates runtime resource copying, evals non-leak, and counterbalanced orders. |
| tests/security.test.ts | Introduces fixtures covering key security audit findings and suppression behavior. |
| tests/registry.test.ts | Ensures registry publication rejects skills with high-risk security findings. |
| tests/cli.test.ts | Updates CLI help expectations to include new commands. |
| tests/check.test.ts | Adds tests for repo-wide gate discovery and strict lint gating. |
| tests/action.test.ts | Verifies pinned third-party actions and safe versioned package execution in the composite action. |
| src/validate.ts | Exports markdown parser, removes authoring lint from portable validation, adds rubric-literal leak warning. |
| src/tui/TaskEvalApp.tsx | Adds runs/order/seed support and report UX updates for task challenges. |
| src/tui/HomeApp.tsx | Adds new Audit and Repository Gate screens and task challenge configuration UI. |
| src/task-eval/types.ts | Expands rubric types, execution metadata (tokens/commands), per-run results, metrics, and verdict. |
| src/task-eval/score.ts | Implements negative file/final rubrics and command-based rubrics and exit-code checks. |
| src/task-eval/run.ts | Adds runtime skill installation (excluding evals/), repeated AB/BA ordering, ROI metrics, and verdict. |
| src/task-eval/load.ts | Parses new rubric types and validates command exit-code expectations. |
| src/task-eval/codex-runner.ts | Adds JSON-line metadata parsing for token + command traces; includes skill root in prompt. |
| src/security/types.ts | Defines security report model (finding/suppression/summary). |
| src/security/audit.ts | Adds static threat scanner with severity gates and narrow suppressions. |
| src/registry/registry.ts | Enforces security audit on publish/install/doctor/installed-skill checks. |
| src/lint.ts | Implements strict authoring lint (Process/Done/length) layered on portable validation. |
| src/cli.tsx | Adds new CLI commands/options, JSON report writing, security report output, and challenge command. |
| src/check.ts | Adds repo-wide skill discovery and deterministic validation+lint+audit gate. |
| site/styles.css | Updates responsive layout + adds new security/case-study sections styling. |
| site/script.js | Updates website command/evidence mode rendering for challenge + labels. |
| site/index.html | Refreshes marketing copy and adds Security section + evidence links; bumps site asset version. |
| site/evidence/verify-real-outcome-v0.4.0.json | Adds published “redundant” dogfood evidence artifact. |
| site/evidence/responsive-release-proof-v0.4.0.json | Adds published “proven” dogfood evidence artifact. |
| SECURITY.md | Documents static audit, suppressions, and CI supply-chain posture. |
| scripts/check-site.ts | Extends site smoke checks for new required content. |
| README.md | Updates docs for validate/lint/audit/check/challenge workflow and new rubrics/verdict semantics. |
| package.json | Bumps package version to 0.4.0. |
| examples/responsive-release-proof.brief.json | Adds new example brief used to generate the responsive-release-proof skill. |
| examples/generated/responsive-release-proof/SKILL.md | Adds generated skill content including evidence contract. |
| examples/generated/responsive-release-proof/references/evidence-schema.md | Adds schema doc for responsive evidence artifact. |
| examples/generated/responsive-release-proof/evals/tasks.yaml | Adds task eval suite with command + evidence rubrics for the example skill. |
| examples/generated/responsive-release-proof/evals/fixtures/mobile-release-proof/styles.css | Adds fixture reproducing mobile overflow regression. |
| examples/generated/responsive-release-proof/evals/fixtures/mobile-release-proof/scripts/check-responsive.ts | Adds proxy script used by the fixture for deterministic checks. |
| examples/generated/responsive-release-proof/evals/fixtures/mobile-release-proof/package.json | Adds fixture script entrypoint for QA proxy command. |
| examples/generated/responsive-release-proof/evals/fixtures/mobile-release-proof/index.html | Adds fixture page demonstrating long CI rail + table overflow issue. |
| examples/generated/responsive-release-proof/evals/cases.yaml | Adds trigger/near-miss cases for discovery evaluation of the example skill. |
| examples/generated/responsive-release-proof/agents/openai.yaml | Adds agent interface metadata for the generated example skill. |
| CONTRIBUTING.md | Documents new invariants (evals hidden, validate vs lint split, security-rule fixtures, pinned actions). |
| CHANGELOG.md | Adds 0.4.0 release notes summarizing new audit/gate/challenge features. |
| action.yml | Introduces reusable composite GitHub Action for the Skillbench gate. |
| .github/workflows/release.yml | Pins third-party actions by commit SHA; tightens tag/version verification and env usage. |
| .github/workflows/pages.yml | Pins third-party actions by commit SHA for pages build/deploy. |
| .github/workflows/ci.yml | Pins third-party actions by commit SHA; adds Skillbench check with evidence artifact upload. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+102
to
+106
| function verdict(taskMetrics: TaskEvalMetrics, thresholds: TaskEvalSuite["thresholds"]): TaskEvalReport["verdict"] { | ||
| const epsilon = 0.000_001; | ||
| if (taskMetrics.averageDelta < -epsilon) return "harmful"; | ||
| if (taskMetrics.averageSkillScore >= thresholds.minSkillScore && taskMetrics.averageDelta > epsilon) return "proven"; | ||
| const efficientByDuration = taskMetrics.durationDeltaPercent !== undefined && taskMetrics.durationDeltaPercent <= -0.15; |
Comment on lines
+41
to
+47
| filter(source) { | ||
| const resolved = path.resolve(source); | ||
| if (resolved === evalsRoot || resolved.startsWith(`${evalsRoot}${path.sep}`)) return false; | ||
| const relative = path.relative(path.resolve(skillPath), resolved); | ||
| const topLevel = relative.split(path.sep)[0]; | ||
| return topLevel !== ".git" && topLevel !== "node_modules"; | ||
| }, |
Comment on lines
+122
to
+123
| if ("value" in rubric && typeof rubric.value === "string" && rubric.value.length >= 8 | ||
| && evalCase.prompt.toLocaleLowerCase().includes(rubric.value.toLocaleLowerCase())) { |
Comment on lines
+56
to
+58
| if [[ ! "$SKILLBENCH_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+([+-][0-9A-Za-z.-]+)?$ ]]; then | ||
| echo "version must be an exact semver, not a tag or range" >&2 | ||
| exit 2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Evidence
Release
Package version is 0.4.0. Merge first; tag v0.4.0 only after CI succeeds.