You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With/without skill-eval harness (benchmarks/evals/) — a benchmark that runs the
same task twice (plugin installed vs. not), grades both blind on the outcome, and
reports the reproducible per-expectation gap (pass^k vs pass@k). The evals.json and grading.json schemas and the blind two-arm grader are taken from Anthropic skill-creator's eval system; the pass^k aggregation, benchmark.json shape, and
HTML viewer are our own and are not compatible with skill-creator's eval-viewer. First eval set is
6 credit-card-fraud cases (4 positive core-discipline cases, 2 negative-trigger
guards) exercising the accuracy trap, planted-metric framing, target leakage, and
test-set peeking. README.md maps all ten skill-eval best practices to how the
harness implements each; run_eval.py/aggregate.py are stdlib-only.
Second eval set + a committed worked example.benchmarks/evals/house-prices/
adds 6 cases built around the target-leakage trap (a neighbourhood-mean-of-SalePrice
feature) on a skewed regression target. benchmarks/evals/example/ is a committed,
graded with/without comparison of eval 1 for both datasets — the with_skill arm
graded against the repo's real shipped benchmark pipelines, the without_skill arm a
committed reproduciblenaive_run.py (real numbers: fraud accuracy 0.9995 /
ROC-AUC 0.944, house-prices in-sample RMSE $11,058 / R² 0.98) — showing a pass^k gap
of 0.875 (fraud) and 0.80 (house-prices). An HTML eval-viewer renders the comparison.
aggregate.py also emits benchmark.skill-creator.json in skill-creator's exact
eval-viewer schema (runs[] keyed by configuration, nested result.pass_rate, run_summary.<config>.pass_rate.{mean,stddev}), alongside our pass^kbenchmark.json
— so results are portable to skill-creator's viewer without giving up the pass^k view.
/ds-frame now takes an information inventory — a framing-time step that writes
down what will actually be known at prediction time versus what only becomes known
after the fact, recorded in 00-frame.md. It's the proactive complement to /ds-prep's per-feature "known at prediction time?" check, and it surfaces available
signal (e.g. prior-period totals) before feature-building instead of after.
Changed
/ds-frame and metric-selection now treat over- vs under-prediction asymmetry as
a first-class question.metric-selection gains a regression row for asymmetric cost
(quantile/pinball loss at a chosen service level) alongside the existing
classification-only F-beta row; /ds-frame asks whether over- and under-shooting cost
the same before locking a symmetric metric.
/ds-baseline now warns against strawman baselines — a global mean/majority on
data with strong temporal, seasonal, or hierarchical structure is trivial to beat, so
a too-weak baseline inflates apparent lift. Guidance and a Red Flag now push toward the
strongest simple anchor (last value, same period last cycle, or the rule already in
use).
/ds-frame flags forecasting as out of scope when it sees it — a target that is a
future value of a time-indexed series now trips a Red Flag pointing at README → Scope,
stating plainly that these gates aren't a forecasting stack (weak mean baseline, no
lag/rolling feature machinery) rather than underperforming silently.
Fixed
Stale structure docs after the deployment mile. The README Project Structure tree
listed only 27 skills (missing ds-package, ds-deploy, and the data-science-project
entry-point skill) while claiming 30, and both README and CONTRIBUTING still said "15
slash commands" (now 17).