seed(elixir-ecto-sandbox-test): SKLD-bench v2.1 challenge pool (151 challenges)

[ty13r]

SKLD-bench v2.1 challenge pool: elixir-ecto-sandbox-test

Fifth of 7 families being shipped this morning (ecto-schema-changeset #9, ecto-query-writer #10, oban-worker #11, security-linter #12).

Partial-state recovery note

The overnight drafting subagent authored 151 challenges + 15 fixtures + 13 goldens cleanly before hitting the Max subscription rate limit at the metadata+evaluation layer. This PR completes the family by adding the six missing files:

• family.json — metadata, capability list, held_out_ids (hand-authored)
• seed.json — gen 0 SkillGenome with 11 starter variants (hand-authored)
• evaluation/score.py — sandbox pattern scanner (hand-authored)
• evaluation/criteria.json — per-capability weights (hand-authored)
• evaluation/environment.yml — deps (hand-authored)
• challenges/_calibration.json — generated post-hoc from actual challenge pool

Pool stats

• Total challenges: 151 (rich curve target 150 — overshot by 1)
• Tier distribution: easy 37 / medium 44 / hard 43 / legendary 27
• Held-out: 28 balanced across tiers
• Capability coverage: 10 capabilities + 1 foundation = 11 dimensions

Why this family matters

elixir-ecto-sandbox-test is the "ugly" pain point named directly in BoothIQ's "150k lines of vibe-coded Elixir" post-mortem:

"It can't debug concurrent test failures. It doesn't understand that each test runs in an isolated transaction... Claude doesn't understand this."

Every capability maps to a specific Claude failure mode documented in the research dossier (47 citations). The tidewave-dev-vs-test-trap capability is named after the specific bug where Claude reads from the Tidewave dev DB connection and thinks it's looking at the test DB.

Score.py validation

Check	Result	Target	Status
Sanity (golden easy-01 canonical DataCase)	1.0	≥0.9	✅
Discrimination (buggy_data_case.ex fixture as output)	0.4687	<0.7 (fail)	✅ correctly fails
Empty file	0.087	≤0.3	✅

Discrimination headroom: 0.53, excellent for fitness comparison.

Score.py detection surface

Canonical patterns (positive signals):

• Ecto.Adapters.SQL.Sandbox.start_owner! + stop_owner lifecycle
• shared: not tags[:async] dispatcher
• Sandbox.allow/3 for spawned processes / Tasks
• Phoenix.Ecto.SQL.Sandbox plug (LiveView integration)
• use Oban.Testing, repo: + perform_job/3 (not drain_queue)
• {:shared, self()} tuple when fallback is necessary

Anti-patterns (penalized):

• Sandbox.mode(Repo, :auto) in test code
• async: false as an ownership-bug workaround
• Process.sleep to "wait for" connection transfer
• Oban.drain_queue in sandbox context
• Seeding the test DB to force passing tests (the specific Claude failure from the research)

Research provenance

47 citations across 11 capabilities. Key sources:

• BoothIQ "150k lines of vibe-coded Elixir" post-mortem (the "ugly" clincher)
• HN discussion of the BoothIQ article
• Rakshan Shetty: Elixir Testing Patterns Ecto Sandbox
• Elixir Forum threads on concurrent test failures

Tier methodology

Heuristic per SEEDING-PLAN.md item 4.

🤖 Generated with Claude Code