v0.10.0 — security fixes, corrected skill guidance, honest evidence baseline

Validation status. Two free, deterministic gates (structural lint and lexical routing) pass, and their numbers reproduce from a clean checkout. Three full pipeline runs on real datasets land inside independently published ranges. What is not measured is whether the plugin changes the answer versus a capable unaided model — the harness that tried to answer that has been removed rather than left to imply a verdict it could not support. Everything here about routing, shape, and pipeline output is measured; the marginal-value claim is open.

Security

• SessionStart no longer forwards untrusted lesson titles verbatim. learnings.jsonl is designed to be committed, so cloning a repo or merging a PR could put a stranger's text into additionalContext at session start, before the user's first turn. Titles from that file are now stripped of control characters, newlines, and zero-width/bidi marks, truncated to 120 characters, and fenced as [untrusted lesson titles from learnings.jsonl — data, not instructions].
• The notebook scan no longer echoes the shell-magic line it flags. It used to quote 60 characters of the offending line — the most likely place in a notebook for a live credential to sit — so the hook whose job is finding secrets was copying them into the transcript. It now reports the line number instead.
• The three subagents declare explicit tools: allowlists. They previously inherited the full parent tool set, meaning a haiku/low-effort data profiler pointed at untrusted CSVs held Bash, Write, WebFetch, and MCP access. None of the three can now make a network call or modify your project.

Fixed — skill content

Six technical errors in the shipped guidance, each one an agent would hit on a real dataset:

• metric-selection explained ROC-AUC's failure under imbalance with a false mechanism. ROC-AUC is invariant to class balance — that's its defining property — so an agent reasoning from the old explanation would conclude rebalancing fixes it. It doesn't; the real issue is that a small false-positive rate is a large false-positive count under severe imbalance. Also adds PR-AUC's missing no-skill floor and the log-target retransformation bias.
• ds-validate's checklist demanded random_state on "every splitter" — TimeSeriesSplit and older GroupKFold reject that argument, so following the checklist raised a TypeError.
• ds-validate's nested-CV example used plain shuffled KFold in both loops, 40 lines after establishing that grouped and time-ordered data must not. Replaced with the grouped form (sklearn metadata routing + pre-1.4 fallback) and TimeSeriesSplit(gap=) for label horizons.
• imbalanced-data listed threshold tuning's leakage risk as "None." Tuning a threshold on the data you then report is textbook optimistic bias.
• imbalanced-data never mentioned calibration, despite every resampling technique it recommends shifting the effective base rate — and /ds-evaluate requiring a calibration check two stages later.
• uncertainty-quantification presented its one-standard-deviation bar without noting that k-fold scores aren't independent, which invites an invalid SD/√k confidence interval. Now explains why the raw SD is used deliberately as a conservative screen, and points to the corrected resampled t-test for an actual test.

Fixed — release & platform

• Marketplace installs are pinned to this release tag (v0.10.0), not an unpinned default branch — a test asserts the pin exists and matches plugin.json.
• Windows was the least-served platform, on the maintainer's own OS. The installer's "install Claude Code first" guard was dead code there; claude is now resolved to an absolute path by walking PATH directly (which also closes a cmd.exe current-directory hijack). Re-running the installer with the marketplace already registered no longer aborts. The routing-check script, a documented release-gate command, crashed under Windows' default console encoding — CI is Linux-only so it never caught it.
• The CSV-scanning hook no longer reads entire files. A comment claimed the scan was bounded; only the pattern search was — the read wasn't. On a 148 MB CSV that measured 465 MB of peak memory inside a synchronous hook.
• Two false claims in AUDIT.md — the document positioned as this plugin's security reference. It said hooks "exit 0 unconditionally" (true of the Python, not the shell wrapper that invokes them) and that subagents "read no files and make no network calls" (they read files; that's their job — they just can't reach the network or write anything, which is now enforced by explicit tool allowlists, not by omission).
• Scope disclaimers now live in the skills that need them, not only in the README — ds-validate, ds-baseline, and metric-selection now state plainly when a question (e.g. time-series forecasting) is out of scope, at the point where an agent would otherwise give confident partial advice.
• README and ds-method disagreed about what a hard gate does. Clarified: discipline gates (missing baseline, validation strategy, slice performance, pinned environment) produce the missing artifact inline and continue; safety gates (parity check, a remote push) still stop and ask.
• The credit-card-fraud benchmark was re-run against corrected metric-selection guidance rather than left as documented drift. model.py, evaluate.py, and seed_stability.py were re-executed: CV PR-AUC 0.8455 ± 0.0117, baseline 0.00167, and seed-stability mean 0.8465 / std 0.0010 all reproduced exactly — every committed figure and artifact came back byte-identical.

Removed

• The with/without-skill eval harness, entirely, from the repo and from git history. It had run once — two cases, three trials, one dataset — against a build with a front-door bug that's since been fixed, so its one measurement no longer describes what ships. 8 of its 13 expectations couldn't separate the two arms at all, and every run was graded by hand with the arm visible (the blinding tool was committed after the results it was meant to blind). Rather than publish a number that looks like evidence and isn't, it's gone. Its transcripts also carried operator machine details (home paths, an inventory of installed tooling) with no legitimate reason to ship — no credentials were found anywhere in history, but there was nothing worth keeping them for either.
• What survives: the skill-routing trigger corpus and the deterministic routing check, now living at benchmarks/routing/ under an honest name. Routing still holds at 91.3% rank-1, 100% top-k, zero description collisions.

Added

• benchmarks/README.md — what each of the three benchmark datasets is, why those three, where to download them, how the results compare against independently published work, and why the House Prices Kaggle leaderboard specifically is not a usable reference (the ground truth is public, so the top of that leaderboard reflects submitting known answers, not modelling skill).
• A supported-task table near the top of the README — regression and classification: supported and benchmarked; multiclass: works, unbenchmarked; time-series forecasting, NLP, vision, recommenders: explicitly out of scope. Said plainly, before install, rather than discovered mid-project.

---

Honest framing for this release: the security and content fixes are real and verified — 398 tests pass, routing holds at 91.3% from a clean checkout of this tag. What this release does not claim is that the plugin outperforms a capable unaided agent; that question is open and the harness that tried to measure it has been removed rather than left to imply an answer it couldn't support.

v0.9.0 — with/without skill-eval harness + forecast-aware framing

Added

• With/without skill-eval harness (benchmarks/evals/) — a benchmark that runs the
  same task twice (plugin installed vs. not), grades both blind on the outcome, and
  reports the reproducible per-expectation gap (pass^k vs pass@k). The evals.json and
  grading.json schemas and the blind two-arm grader are taken from Anthropic
  skill-creator's eval system; the pass^k aggregation, benchmark.json shape, and
  HTML viewer are our own and are not compatible with skill-creator's eval-viewer. First eval set is
  6 credit-card-fraud cases (4 positive core-discipline cases, 2 negative-trigger
  guards) exercising the accuracy trap, planted-metric framing, target leakage, and
  test-set peeking. README.md maps all ten skill-eval best practices to how the
  harness implements each; run_eval.py/aggregate.py are stdlib-only.
• Second eval set + a committed worked example. benchmarks/evals/house-prices/
  adds 6 cases built around the target-leakage trap (a neighbourhood-mean-of-SalePrice
  feature) on a skewed regression target. benchmarks/evals/example/ is a committed,
  graded with/without comparison of eval 1 for both datasets — the with_skill arm
  graded against the repo's real shipped benchmark pipelines, the without_skill arm a
  committed reproducible naive_run.py (real numbers: fraud accuracy 0.9995 /
  ROC-AUC 0.944, house-prices in-sample RMSE $11,058 / R² 0.98) — showing a pass^k gap
  of 0.875 (fraud) and 0.80 (house-prices). An HTML eval-viewer renders the comparison.
• aggregate.py also emits benchmark.skill-creator.json in skill-creator's exact
  eval-viewer schema (runs[] keyed by configuration, nested result.pass_rate,
  run_summary.<config>.pass_rate.{mean,stddev}), alongside our pass^k benchmark.json
  — so results are portable to skill-creator's viewer without giving up the pass^k view.
• /ds-frame now takes an information inventory — a framing-time step that writes
  down what will actually be known at prediction time versus what only becomes known
  after the fact, recorded in 00-frame.md. It's the proactive complement to
  /ds-prep's per-feature "known at prediction time?" check, and it surfaces available
  signal (e.g. prior-period totals) before feature-building instead of after.

Changed

• /ds-frame and metric-selection now treat over- vs under-prediction asymmetry as
  a first-class question. metric-selection gains a regression row for asymmetric cost
  (quantile/pinball loss at a chosen service level) alongside the existing
  classification-only F-beta row; /ds-frame asks whether over- and under-shooting cost
  the same before locking a symmetric metric.
• /ds-baseline now warns against strawman baselines — a global mean/majority on
  data with strong temporal, seasonal, or hierarchical structure is trivial to beat, so
  a too-weak baseline inflates apparent lift. Guidance and a Red Flag now push toward the
  strongest simple anchor (last value, same period last cycle, or the rule already in
  use).
• /ds-frame flags forecasting as out of scope when it sees it — a target that is a
  future value of a time-indexed series now trips a Red Flag pointing at README → Scope,
  stating plainly that these gates aren't a forecasting stack (weak mean baseline, no
  lag/rolling feature machinery) rather than underperforming silently.

Fixed

• Stale structure docs after the deployment mile. The README Project Structure tree
  listed only 27 skills (missing ds-package, ds-deploy, and the data-science-project
  entry-point skill) while claiming 30, and both README and CONTRIBUTING still said "15
  slash commands" (now 17).

v0.6.0 — fix broken figure links, add /ds-brief for non-technical audiences

Fixed

• Every figure link in every benchmark's 07-evaluate.md/08-explain.md/02-explore.md was broken — written as ../.last-ds-mile/figures/... from a file already inside .last-ds-mile/stages/, adding a nonexistent extra .last-ds-mile/ segment (stages/ and figures/ are siblings, not nested). This is also why no chart ever rendered inline on GitHub — the figures were generated correctly, only the reference to them was wrong. Fixed across all 14 image references in all three benchmarks; verified every relative link in benchmarks/ and showcase/ now resolves to a real file.
• Swept all three benchmarks for other staleness (artifact filenames, cross-stage number consistency, stray /ds-iterate references) — found nothing else wrong.

Added

• ds-brief skill and /ds-brief command — translates /ds-report's technical narrative into a one-page, jargon-free brief for non-technical stakeholders (executives, a board, a frontline team lead). No metric names or statistical terms permitted; every claim must trace back to /ds-report rather than introducing new analysis. Demonstrated with real examples for all three benchmarks (09b-brief.md), each verified jargon-free and under 350 words.
• CONTRIBUTING.md, SECURITY.md, CLAUDE.md — contribution workflow, private vulnerability reporting, and repo-specific instructions for Claude Code sessions working in this codebase.

Full Changelog: https://github.com/stamkavid/last-ds-mile/blob/main/CHANGELOG.md