Skip to content

Remove SkillGym suite and repo-health snapshot infrastructure - #1480

Merged
thymikee merged 4 commits into
mainfrom
claude/agent-device-scope-review-bmrfyc
Jul 29, 2026
Merged

Remove SkillGym suite and repo-health snapshot infrastructure#1480
thymikee merged 4 commits into
mainfrom
claude/agent-device-scope-review-bmrfyc

Conversation

@thymikee

@thymikee thymikee commented Jul 29, 2026

Copy link
Copy Markdown
Member

Removes the SkillGym suite and the repo-health snapshot per the 2026-07-28 descope recorded on #1412: two parallel LLM-judge suites answered one question (the help-conformance bench is now the single non-gating small-model oracle), and the observatory track is deferred until a concrete consumer exists (#1424 and PR #1477 closed).

Summary

  • SkillGym removed end-to-end (test/skillgym/, 770 LOC + config): the smoke suite, runner environment, local-cli-help-policy helpers, and the bin/agent-device wrapper. The test:skillgym* package scripts and the skillgym devDependency are gone (lockfile regenerated), along with the .skillgym-results/ entries in .gitignore/.oxfmtrc.json, the unit-lane include in vitest.config.ts, and the fallow config globs + stale fallow-baselines/health.json entry.
  • check-affected simplified: the skillgym check id, catalog entry, and ownership rule are removed. Behavior change: skills/*.md now classifies as docs-only in the selector — previously it selected the SkillGym suite; without an owning suite it would otherwise fail open to the full check set on every SKILL.md edit.
  • scripts/repo-health removed (~1.2k LOC, 6 files). Forced by the descope and by this diff: repo-health/run.ts statically imported the deleted SkillGym suite. Its only gating assertion duplicated the Layering Guard job, and its sole planned consumer (obs: evolve the size-report PR comment into a quiet quality-delta comment + metric history on main #1424 / PR obs: quiet quality-delta PR comment + repo-health metric history on main #1477) is closed. The depgraph/layering models remain the queryable data. Also drops the now-consumerless SLOW_TEST_RATCHET re-export from scripts/vitest-slow-test-budgets.ts and two stale comments pointing at repo-health (scripts/layering/check.ts, budgets module).
  • Docs updated to route command-planning guidance checks at the help-conformance bench instead: AGENTS.md, docs/agents/testing.md, docs/agents/pull-requests.md, test/output-economy/README.md, test-app README + one fixture comment.

Validation

Local: oxfmt --check, oxlint, tsc, check:layering (all R-rules green, R6 at 7, R9 at 102), fallow audit --base origin/main (no issues in the 33 changed files), the check-affected node --test suites (31/31), and the full unit run (unit-core + subprocess-stub, 4,700+ tests, exit 0; two tests over budget inside the 2x load-variance band, non-failing). Build/smoke lanes left to CI — the diff is tooling/docs-only outside examples/test-app/src/screens/FormScreen.tsx, which changes a comment.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FUv7bvbWNryuXgSBuqTtep

Remove the SkillGym harness (test/skillgym/), its check-affected lane,
package scripts, and devDependency — the help-conformance bench is now
the single non-gating small-model oracle. skills/ markdown classifies
as docs in the affected-check selector instead of failing open.

Remove scripts/repo-health: its only gating assertion duplicated the
Layering Guard job, its case-count metric imported the deleted SkillGym
suite, and its sole planned consumer (#1424 / PR #1477) was closed with
the Track C descope on #1412.

Verified: check-affected node --test suites, oxfmt, oxlint, tsc,
check:layering, fallow audit vs origin/main, and the full unit suite
(unit-core + subprocess-stub) all pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FUv7bvbWNryuXgSBuqTtep
@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-29 09:46 UTC

@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.87 MB 1.87 MB 0 B
JS gzip 602.8 kB 602.8 kB 0 B
npm tarball 720.6 kB 720.5 kB -93 B
npm unpacked 2.52 MB 2.52 MB -499 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 30.0 ms 29.3 ms -0.6 ms
CLI --help 60.3 ms 59.8 ms -0.5 ms

Top changed chunks: no changes in the largest emitted chunks.

…-exports

The Fallow production-exports gate flagged all three budget exports:
their in-file consumer (SLOW_TEST_RATCHET) and the repo-health entry
point that kept the module reachable were both removed in the descope,
leaving the config-loaded reporter as the only consumer — invisible to
--production analysis. The data-only module's second consumer is gone,
so per the boundaries-are-earned norm the constants move into the
reporter instead of gaining a suppression.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FUv7bvbWNryuXgSBuqTtep
@thymikee

Copy link
Copy Markdown
Member Author

Two P2 cleanup findings remain at bced9062:

  1. docs/agents/testing.md still says every skills/ change requires pnpm format and describes formatting as always-on, while selectChecks() now short-circuits skills/*.md as docs-only with checks: [] and the updated regression pins that behavior. Make the matrix/narrative match the intentional no-local-check policy, or retain format selection.

  2. The merge result still leaves examples/README.md saying test-app is used for SkillGym experiments, contradicting the claimed end-to-end removal; update that root examples index too.

The #1412/#1424 descope and removal wiring otherwise look coherent. Completed exact-head gates are green; iOS Smoke is still in progress. No ready-for-human label until these are resolved.

claude added 2 commits July 29, 2026 08:55
Address both P2 review findings on #1480: the testing-matrix row and
the selector's formatGate both still claimed oxfmt covers skills/,
while selectChecks classifies skills/*.md docs-only (oxfmt ignores
**/*.md, so the claim was a no-op even before). The matrix now states
the docs-only policy and formatGate drops the dead underSkills fact.
The merged examples/README.md index (from #1469) loses its skillgym
mention.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FUv7bvbWNryuXgSBuqTtep

Copy link
Copy Markdown
Member Author

Both P2 findings addressed in ca50809 (which also merges current main so the fix applies to the merge result):

  1. Docs now state the intentional docs-only policy rather than retaining format selection: the Testing Matrix row says skills/ is Markdown-only guidance that oxfmt ignores (**/*.md) and the selector classifies docs-only. The claim was a no-op even before the descope — oxfmt never formatted skills Markdown — so the matrix was describing a gate that didn't exist. formatGate drops the dead underSkills fact so the code stops making the same stale claim (oxfmt covers src/ and test/).
  2. examples/README.md (merged in via feat(examples): add runnable Node.js SDK examples under examples/sdk/ #1469, which is why the branch-side sweep missed it) loses its skillgym mention.

Selector node --test suites (31/31), oxfmt, oxlint, tsc (incl. the new examples/sdk project), fallow audit vs merge result, and check:production-exports all green locally.


Generated by Claude Code

@thymikee

Copy link
Copy Markdown
Member Author

Re-reviewed the current head ca50809 after the follow-up fixes. No remaining findings from me.

The two earlier P2s are resolved in the merge result: the skills/*.md docs-only policy is now consistent between the selector, regression test, and testing guide, and the last stale SkillGym mention in examples/README.md is gone. I also checked the removal wiring and stale references for SkillGym, repo-health, and the extracted slow-test budget module; the remaining SkillGym occurrence is only the explanatory selector comment.

Focused validation: pnpm check:affected:test passes 31/31 locally. Exact-head CI is fully green, including Fallow Code Quality, coverage, integration, iOS/Android/macOS/Linux smoke, and Web Platform Smoke. This matches the #1412/#1424 descope and looks ready to merge.

@thymikee
thymikee merged commit 53e4be5 into main Jul 29, 2026
29 checks passed
@thymikee
thymikee deleted the claude/agent-device-scope-review-bmrfyc branch July 29, 2026 09:45
thymikee pushed a commit that referenced this pull request Jul 29, 2026
Rebase onto main post-#1480: the SkillGym/repo-health descope deleted
scripts/repo-health/run.test.ts, whose CONTENTION_RETRY_FILES entry
would now fail this PR's own missing-file check, and inlined the
slow-test budgets into the reporter, resolving the budgets-module
import. Envelope comments now point at scripts/lib/lane-envelope.ts
instead of the closed #1430.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FUv7bvbWNryuXgSBuqTtep
thymikee added a commit that referenced this pull request Jul 29, 2026
…timeouts only) (#1448)

* test(ci): single-retry policy for enumerated contention-flaky files

* fix: satisfy fallow

* test(ci): read failures through a lane reporter so timeouts stay distinguishable

* test(ci): cover the lane reporter and drop its duplicated boilerplate

* chore(fallow): own the retry lane's tool-loaded export seams

* test(ci): block retries on non-test failures and classify timeouts structurally

* test(ci): decide retry eligibility from runner metadata and route gate verdicts through blockers

* refactor(ci): name the retry policy's rules in code instead of comments

* test(ci): mark runner-aborted timeouts inside the runner instead of inferring them

* test(ci): make timeout provenance a per-run secret, not a writable flag

Cover direct task.meta mutation in the real child-Vitest fixture gate.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ci): retry the failed files in the first run's project and coverage modes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: drop deleted repo-health file from the retry list after #1480

Rebase onto main post-#1480: the SkillGym/repo-health descope deleted
scripts/repo-health/run.test.ts, whose CONTENTION_RETRY_FILES entry
would now fail this PR's own missing-file check, and inlined the
slow-test budgets into the reporter, resolving the budgets-module
import. Envelope comments now point at scripts/lib/lane-envelope.ts
instead of the closed #1430.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FUv7bvbWNryuXgSBuqTtep

---------

Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants