Skills refactor: trim bloat, add model-evaluation + pydantic-ai, pushy trigger descriptions by ortizeg · Pull Request #6 · ortizeg/whet

ortizeg · 2026-07-23T19:58:03Z

P0/P1 cleanup from the skill-library review. 32 skills total.

What changed

1. Trim bloat (progressive disclosure)

All 19 SKILL.md bodies over Anthropic's 500-line guideline are now under it — restated "what is X" prose and near-duplicate code blocks removed, every distinct pattern/gotcha/anti-pattern kept.

Skill	Before	After
kubernetes	776	480
onnx	720	499
abstraction-patterns	687	464
matplotlib	675	452
github-actions	632	412
pypi	616	335
…13 more	—	all <500

Avg SKILL.md 551 → ~440 lines; ~2.2k net lines removed. (Next step, not in this PR: split survivors into thin index + references/.)

2. New skills (real coverage gaps)

model-evaluation — detection mAP/IoU, confusion matrices, per-class + per-size breakdowns, threshold selection, per-slice failure analysis, eval-as-CI gate. Built on supervision (import supervision as sv), the detection-native library already used alongside RF-DETR/YOLOX, with torchmetrics retained for classification/calibration. Feeds GSD's eval-planner; previously only generic testing existed.
pydantic-ai — typed LLM/VLM structured outputs; VLM-in-the-loop auto-labeling (Gemini/Moondream → validated detections).

The supervision usage was verified against 0.29.1 and encodes the real traps: import from supervision.metrics (the top-level sv.MeanAveragePrecision is deprecated for 0.31.0 and inconsistent with pycocotools); MetricTarget.MASKS is silently ignored by mAP in released versions (it scores boxes — use pycocotools for mask mAP); don't pre-filter confidence before mAP; -1 is an "absent" sentinel, not a score.

3. `pydantic-strict` → `pydantic`

Renamed for a simpler name and broader scope (strict remains the default stance). Updates the dir, docs page, nav, archetype refs, and all skill.toml cross-refs — no dangling references.

4. Pushy "use-when" descriptions

Every skill's description: rewritten as a third-person, trigger-first "Use this skill when…" with keyword coverage and explicit disambiguation between adjacent skills (code-quality vs pre-commit vs vscode; docker-cv vs kubernetes vs gcp; pydantic vs pydantic-ai; mlflow vs wandb vs tensorboard).

5. Env manager: pixi stays canonical

Kept pixi as the ML env manager (handles CUDA/system deps beyond Python). Incidental run-commands are tool-agnostic (pytest, not pixi run pytest); dependency installs use pixi add.

6. No vertex-ai skill — folded into `gcp` instead

A separate vertex-ai skill was duplicating the gcp skill, which already covers Vertex AI training jobs (custom + custom-container, GPU selection reference, prebuilt containers) alongside GCS and Artifact Registry. Instead, gcp gains the one genuinely non-obvious bit:

Retrieving Artifacts After Training — the training VM is ephemeral, so write to AIP_MODEL_DIR and explicitly sync artifacts down when the job ends.

Verification

uv run pytest tests/ → 213 passed
uv run ruff check . → clean
uv run ruff format --check . → clean

Notes for review

The pydantic rename touches archetype .toml skill lists and cross-refs — worth a glance that nothing dangles.
Worth sanity-checking the trimmed skills you use most (pytorch-lightning, onnx, fastapi) for anything cut that you wanted kept.
model-evaluation's supervision snippets assume sv.DetectionDataset.from_coco; swap to from_yolo if that matches your loader.

🤖 Generated with Claude Code

…escriptions Progressive-disclosure + fit cleanup from the skill-library review. - Trim all 19 over-limit SKILL.md bodies under Anthropic's 500-line guideline (kubernetes 776->480, onnx 720->499, pypi 616->335; ~2.2k net lines removed). Cut restated "what is X" prose and near-duplicate code blocks; kept every distinct pattern, gotcha, and anti-pattern. - Add skills: model-evaluation (mAP/IoU, PR curves, eval sets, slice analysis, eval-as-CI), pydantic-ai (typed LLM/VLM structured outputs / VLM auto-labeling), vertex-ai (custom-container training on Vertex, the gcp-workflow pattern). - Rename pydantic-strict -> pydantic (broaden scope; strict stays the default stance). Updates dir, docs, nav, archetype refs, and skill.toml cross-refs. - Rewrite every skill description as a third-person, pushy "use-when" trigger with disambiguation between adjacent skills. - Keep pixi as the canonical ML env manager (handles CUDA/system deps beyond Python); incidental run-commands are tool-agnostic, installs use `pixi add`. 33 skills total. 218 tests pass; ruff check + format clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The gcp skill already covered Vertex AI training jobs (custom + custom-container, GPU selection reference, prebuilt containers) alongside GCS and Artifact Registry, so a separate vertex-ai skill was duplicating an existing skill rather than filling a gap — the exact redundancy this refactor is removing. - Remove skills/vertex-ai and docs/skills/vertex-ai.md - Add "Retrieving Artifacts After Training" to gcp: the training VM is ephemeral, so write to AIP_MODEL_DIR and explicitly sync artifacts down when the job ends - Add the matching best-practice bullet and renumber the list - Repoint the dangling skill references (gcp + aws-sagemaker descriptions), drop the docs table row and mkdocs nav entry - 32 skills; 213 tests pass, ruff + format clean Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Switch detection evaluation to `import supervision as sv` — the detection-native library already used alongside RF-DETR/YOLOX, which speaks the same sv.Detections object as those model outputs and whose metrics are pycocotools-aligned (>=0.26). Verified against supervision 0.29.1 and encodes the real traps: - Import from `supervision.metrics`, NOT the top-level `sv.MeanAveragePrecision` (deprecated in 0.27.0, removed in 0.31.0, and inconsistent with pycocotools) - MetricTarget.MASKS is silently ignored by mAP in released versions — it scores boxes instead, so use pycocotools for instance-segmentation mAP - Do not pre-filter confidence before mAP (needs the full score-ranked list); ConfusionMatrix, by contrast, wants a real deployment threshold - `-1` is an "absent" sentinel, not a score - len(preds) != len(targets) raises — pad with sv.Detections.empty() - class_id is the COCO category id; xyxy is absolute pixels torchmetrics retained for classification metrics/calibration (supervision is detection-only). 272 lines; 213 tests pass, ruff + format clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Follows [#6](#6) (merged). Result: **32 skills, 0 agents.** ## Why remove the agents They were never Claude Code subagents. Four independent pieces of evidence: 1. **4 of 6 had no YAML frontmatter** — `expert-coder`, `ml-engineer`, `code-review`, `test-engineer` began directly with an H1. They installed with the heading as their entire description (`expert-coder: Expert Coder Agent`) and were effectively untriggerable. The two that worked were the two that happened to have frontmatter. 2. **All 6 installed to `.claude/skills/`, not `.claude/agents/`.** `discover_agents()` returned `Skill` objects; there was no `Agent` model anywhere in `src/`. 3. **`agent.toml` was never parsed** — only a doctor check and a test asserted the file existed. The `advisory`/`blocking` field was read exactly once, by a test. 4. **Both "blocking" `action.yml` files were broken** — they ran `pixi run` against a repo with no `pixi.toml` (whet itself uses uv), and no workflow referenced them. Content-wise ~77% of their 3,047 lines duplicated existing skills. Because they were written independently they *diverged*: `test-engineer` timed GPU inference without `cuda.synchronize()` (meaningless) while `skills/testing` does it correctly; `expert-coder` recommended `logging.getLogger` against the repo's own Loguru mandate. GSD already ships 33 correctly-formatted subagents. **GSD owns agency; whet owns domain knowledge — and knowledge is a skill.** ## Promoted to a skill - **`skills/data-pipelines`** (481 lines) from `data-engineer` — storage-format decision matrix, **group-aware leakage-preventing splitting** (frames from one match must not straddle splits), schema evolution/migration, Great Expectations suites. ## Also removed (per review) - **`dvc`** — not part of the workflow. Removed the skill, its docs page, nav entry, archetype/`skill.toml` references, and rewrote prose that pointed at it. Where DVC appeared as a *generic* data-versioning concept, the guidance is now tool-neutral (object storage plus a versioned/content-hashed manifest) rather than naming an unused tool. - **`cv-model-selection`** — not wanted. ## Salvaged before deletion | From | To | |---|---| | devops-infra containerization tree | `docker-cv` | | devops-infra deployment tree ("is K8s right?") | `kubernetes` | | devops-infra CI/CD tree | `github-actions` | | test-engineer coverage thresholds + naming convention | `testing` | ## Removed plumbing `agents/`, `docs/agents/`, `discover_agents()`, `--agents-only`/`--skills-only`, `agents_dir` config, the two doctor agent checks (remaining checks renumbered 1–8), `tests/test_agents.py`, and the agents nav/how-to sections. ## Verification - `uv run pytest tests/` → **192 passed** - `uv run ruff check .` / `ruff format --check .` → clean - `uv run mypy src/whet/ tests/ --strict` → clean - `uv run mkdocs build --strict` → clean (caught and fixed dangling agent links) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Third pillar of the library review. Skills (#6, #8) and agents (#7) were audited and fixed; the 6 archetypes had never been reviewed. An audit that actually **rendered** each one and ran the toolchain against the output found they were in worse shape than either. ## Why it went unnoticed **Nothing ever rendered an archetype in CI.** The tests checked README length and file existence — which is exactly how `cv-inference-service` shipped for months without the `fastapi` skill. Every bug below was invisible to a green build. ## What was broken **4 of 6 archetypes generated nothing.** `whet init` fell through to a 4-file stub; `model-zoo` and `research-notebook` produced *byte-identical* output apart from one description string. `research-notebook` shipped no `notebooks/` directory — the one thing its name promises. **The 2 with templates produced projects that did not run:** | Archetype | Failure | |---|---| | `pytorch-training-project` | **Zero runtime dependencies declared** — couldn't import its own entry point. Crashed on the first training step (`batch["image"]` against `FakeData`, which yields tuples). **Hydra was a required skill but entirely absent**, so documented CLI overrides silently did nothing — worse than crashing. Failed its own `ruff` gate. | | `cv-inference-service` | Named for ONNX, requires the `onnx` skill, shipped **no `onnxruntime`, no inference engine, no `/predict`** — its request/response schemas were dead code. `Dockerfile` COPY'd a `uv.lock` that doesn't exist, so `docker build` failed. | And the docs described a different product than the code: every README said `pixi`, every template said `uv`; all six listed a `pixi.toml` no template shipped. ## What this PR does **Templates: 19 files → 125.** All six now render into a project that passes its own gates. - **pytorch-training-project** — deps declared; tuple unpacking; a real `@hydra.main` driving Pydantic configs; `transforms.py` and `configs/trainer/debug.yaml` added; 13 tests including a genuine `fast_dev_run`. A *second* latent crash surfaced during verification and was fixed: the `LazyLinear` head left `configure_optimizers` iterating uninitialized params. - **cv-inference-service** — real ONNX Runtime engine, wired `/predict`, pixi-based Dockerfile, and the app starts cleanly with **no model artifact present** (covered by a test, so the suite needs no weights). - **data-processing-pipeline** — group-aware, leakage-preventing splits (split on video/match/scene, never the row) with a disjointness assertion. - **library-package** — the packaging furniture the README promised: `LICENSE`, `CHANGELOG.md`, `py.typed`, entry point. - **model-zoo** — model-card schema + **sha256-verified** weight fetch, with a test proving a corrupted file is rejected. - **research-notebook** — an actual `notebooks/` dir with a valid `.ipynb`, plus `nbstripout` so outputs stay out of git. **The guardrails are the real point:** - `tests/test_archetype_templates.py` renders every archetype through whet's **real** `render_template` and asserts the generated project has no unsubstituted placeholders, parses, passes `ruff`, **declares every package it imports**, and has src-layout + tests. That import-vs-dependency check catches the "zero dependencies declared" class of bug without installing torch. - `tests/test_archetypes.py` — archetype skills resolve to real skills, no `extra`-tier skill sits in `required`, `required` is non-empty. - New **`archetypes` CI job** running both suites. **Staleness swept** — the deleted Code Review / Test Engineer agents were still referenced in 5 archetype READMEs, `docs/archetypes/index.md`, `master-skill`, and two guides (`code-review.yml` → a plain `ci.yml`). Templates now use pixi consistently with tool-agnostic run commands; all `uv` references dropped from generated projects. ## Verification - `uv run pytest tests/` → **250 passed** - `uv run ruff check .` / `ruff format --check .` → clean - `uv run mypy src/whet/ tests/ --strict` → clean - `uv run mkdocs build --strict` → clean ## Notes for review - Deliberately **not** in this PR: further skill-composition additions the audit suggested (no archetype pulls in `pixi` despite every README opening with `pixi install`; `pydantic-ai` is referenced by zero archetypes). Kept separate to keep this diff to "make the templates work." - Each archetype still has up to three conflicting directory trees documented (archetype README vs docs page vs actual `template/`). Worth a follow-up making `template/` the single source of truth. - Two caveats from template verification: mypy on a real 3.11 interpreter can trip over current numpy stubs (`--python-version 3.12` is clean); and pixi warns that `[project]` is deprecated in favor of `[workspace]` — left as-is because `skills/pixi/SKILL.md` documents `[project]` repo-wide, so changing it should be coordinated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Release branch for **v0.3.0**, merging `develop` into `main`. ## ⚠️ Two things to know before merging **1. A release blocker was found and fixed here.** I installed the built wheel into a clean venv and ran the CLI outside any checkout — what a user actually does — and got: ``` $ whet list No skills found. ``` The wheel packaged **only `src/whet`**: 36 files, zero `SKILL.md`. The skills were listed for the *sdist* but never the wheel, and the bundled dirs resolved via `Path(__file__).parents[3]`, which is the repo root in a checkout but points outside `site-packages` once installed. **The currently published 0.2.1 almost certainly has this same defect.** Fixed here; the wheel now carries 430 files (32 skills, 155 references, 125 template files). **2. Version had drifted.** PyPI carries **0.2.1** while both `main` and `develop` declared **0.1.0**, and the repo has **no git tags** — 0.2.1 was published outside this tag-triggered workflow. Releasing from 0.1.0 would have built a package *older* than what is live. ## Version: 0.3.0 Breaking, so a minor bump under pre-1.0 semver: - the `agents/` concept and `--agents-only` / `--skills-only` flags are removed - skills `dvc` and `cv-model-selection` deleted; `pydantic-strict` renamed to `pydantic` - `whet init` now installs skills instead of printing a hint ## What's in the release Thirteen merged PRs (#6–#18) auditing all three pillars against Anthropic's Agent Skills guidance: - **Skills** — always-resident context down **73%** (16.5k → 4,523 lines); 29 of 32 split into a thin index plus `references/`; every description rewritten as a trigger-first statement. - **Agents** — removed. Four of six had no frontmatter and were untriggerable; all six installed as skills; `agent.toml` was never parsed. GSD owns agency, whet owns knowledge. - **Archetypes** — all six now generate a project that runs, gated by CI that renders each one and runs the toolchain against the output. - **Enforcement** — the authoring rules are now CI-enforced rather than upheld by discipline. Full detail in [`CHANGELOG.md`](CHANGELOG.md), added in this PR. ## Verification - `uv run pytest tests/` → **541 passed** (5 new packaging guards) - ruff, format, `mypy --strict`, `mkdocs build --strict` → clean - `uv build` → `whet-0.3.0` wheel + sdist - **Installed-wheel smoke test**: clean venv, outside the repo — `whet list` shows 32 skills, `whet doctor` passes, and `whet init cv-inference-service` scaffolds a project with 8 skills and 43 reference files installed ## After merge Tagging `v0.3.0` triggers `release.yml`, which **publishes to PyPI** and creates a GitHub Release. That step is irreversible — a PyPI version can never be reused — so I'll confirm before pushing the tag. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

ortizeg and others added 2 commits July 23, 2026 15:57

ortizeg changed the title ~~Skills refactor: trim bloat, add eval/pydantic-ai/vertex-ai, pushy trigger descriptions~~ Skills refactor: trim bloat, add model-evaluation + pydantic-ai, pushy trigger descriptions Jul 23, 2026

ortizeg merged commit 5bf59a8 into develop Jul 23, 2026
2 checks passed

ortizeg deleted the skills-p0-refactor branch July 23, 2026 20:23

ortizeg mentioned this pull request Jul 23, 2026

P1: remove agents/, promote data-pipelines into skills #7

Merged

ortizeg mentioned this pull request Jul 24, 2026

Make every archetype generate a working project, and gate it in CI #11

Merged

ortizeg mentioned this pull request Jul 24, 2026

Release v0.3.0 #19

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skills refactor: trim bloat, add model-evaluation + pydantic-ai, pushy trigger descriptions - #6

Skills refactor: trim bloat, add model-evaluation + pydantic-ai, pushy trigger descriptions#6
ortizeg merged 3 commits into
developfrom
skills-p0-refactor

ortizeg commented Jul 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ortizeg commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

1. Trim bloat (progressive disclosure)

2. New skills (real coverage gaps)

3. pydantic-strict → pydantic

4. Pushy "use-when" descriptions

5. Env manager: pixi stays canonical

6. No vertex-ai skill — folded into gcp instead

Verification

Notes for review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ortizeg commented Jul 23, 2026 •

edited

Loading

3. `pydantic-strict` → `pydantic`

6. No vertex-ai skill — folded into `gcp` instead