feat(experiments): expose trade_off_table as an MCP tool - #487
Merged
Conversation
Adds experiments/_tools/trade_off_table.py, following the ENG-02 one-module- per-tool layout and the ENG-04/ENG-10 pydantic input contract. The tool returns the two-level factorial trade-off grid: one cell per (run budget, factor count), each carrying the design label, its resolution and the generators that build it. Cells where the budget exceeds the full factorial report replication rather than an error; impossible combinations come back blank with the reason attached, so the agent can explain why. Run and factor axes are caller-supplied and bounded at 128 runs / 12 factors, which keeps the minimum-aberration search behind each cell interactive. Only the trade-off table is exposed; the simulators and dataset loaders are deliberately left off the tool surface. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzfBSecQWwr9AhCN5MqTe1
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
10 tasks
kgdunn
added a commit
that referenced
this pull request
Aug 10, 2026
The branch was cut from 0e8e8d1 on 7 August, before #486 through #490 merged, and had gone stale. Four files conflicted. Version metadata. This branch set 1.63.0 in pyproject.toml and CITATION.cff, but 1.63.0 was released from main on 8 August by #486, and main now stands at 1.66.1. Taking this branch's value would have moved the version backwards onto a number already in use, so main's 1.66.1 is kept in both files and no bump is made here. CHANGELOG. The "## [1.63.0] - 2026-08-07" heading is dropped for the same reason and its body moves under [Unreleased], leaving main's released history untouched. The link-reference footer is taken from main, which carries the four releases this branch predates. experiments/__init__.py. Both sides appended to __all__ in the same region: this branch added moment_aberration, main added manufacture and omars_minimum_runs. All three are kept, alphabetical order restored, and the list checked programmatically against the module's imports. evaluate.py merged without conflict. One thing the merge could not flag. SKILL.md stated that ten @tool_spec DOE tools are reachable through the dispatcher and listed them in a table. Main added an eleventh, trade_off_table, in #487. The count and the table are corrected, and the documented list is now checked against the modules actually present in experiments/_tools. Verification after the merge: ruff check . and ruff format --check . clean mypy src/process_improve clean, 146 source files pytest: 2530 passed, 5 skipped, coverage 94.12% moment_aberration still reproduces the paper's Table 1 pattern exactly (3.27, 11.67, 42.47, 157.27, 591.27, 2251.67, 8666.47) at resolution IV, both directly and through the evaluate_design metric all three skill scripts run, including the non-zero exit path: verify_design.py --require-resolution 5 exits 3 on a resolution IV design Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1daHPaXLs7PPRsq8CmMMm
kgdunn
added a commit
that referenced
this pull request
Aug 10, 2026
#485 landed the doe-designer skill and minimum moment aberration. Merging it in exposes call sites that this branch's renames break, none of which git flags. CHANGELOG was the only textual conflict: both sides had added to [Unreleased]. Entries from both are kept, main's first in each subsection. Three schema-key call sites in the skill. The n_ prefix sweep renamed generate_design's `center_points` and `replicates` inputs, and the skill passes those keys as live JSON: examples/design_spec.json "center_points": 0, invoked by SKILL.md, examples/README.md and worked-example.md references/worked-example.md:59 the prose describing that call references/worked-example.md:199 a runnable CCD command Left alone, the skill's flagship example would fail input validation rather than degrade. All three now use n_center_points, and all three documented commands were run to confirm it. The English words are untouched: "center points or replicates", "extra replicates", "3 replicates at the optimum". Only keys in payloads changed. Three tests in test_moment_aberration.py called generate_design with center_points=; updated to n_center_points=. Also corrected, unrelated to the rename: the CHANGELOG entry for the skill said it exposes "the ten designed-experiments tools". trade_off_table (#487) made it eleven, which SKILL.md already reflected after the #485 merge. Citation narrowed in three places. SKILL.md, references/verification.md and scripts/verify_design.py each stated that the Vazquez, Rother and Charles-Gonzalez study (arXiv:2512.17113) ran GPT-5.1 and Gemini 2.5 Flash over 36 tasks with ten replicates, and that the failures were resolution 1 or 2, non-regular arrays offered as regular fractions, and tables with missing cells. The paper's abstract supports the run sizes, the factor range, the GPT and Gemini families, and reliability up to about eight factors; it does not name model versions, task or replicate counts, or those specific failure modes. The text now claims only what is verifiable, and the authors are named in full. Verification after the merge: ruff check . and ruff format --check . clean mypy src/process_improve clean, 146 source files pytest: 2538 passed, 5 skipped API naming scan over the merged code: no scalar count missing the n_ prefix, including everything #485 added all 44 numeric checks behind the pid-book sections still pass MCP surface as intended: trade_off_table still keyed on runs and factors, generate_design now on n_center_points and n_replicates skill scripts run end to end, including verify_design.py --require-resolution 5 exiting 3, and the CCD command from worked-example.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1daHPaXLs7PPRsq8CmMMm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
trade_off_tableas an agent-callable tool, inexperiments/_tools/trade_off_table.py. Follows the ENG-02 one-module-per-tool layout and the ENG-04 / ENG-10 pydantic input contract (ConfigDict(extra="forbid"), parsed model as the single positional argument).popcorn,grocery,manufacture) and the dataset loaders are deliberately left off the tool surface.The tool returns the two-level factorial grid: one cell per (run budget, factor count), each carrying the design label, its resolution and the generators that build it. Two cases an agent needs to explain rather than just report:
2^3 (4 times),replicates: 4) instead of erroring.table, and the matching entry incellscarriesexists: falseplus the reason ("8 runs cannot accommodate 9 factors: only 7 factors fit into 8 runs").Run and factor axes are caller-supplied, bounded at 128 runs and 12 factors so the minimum-aberration search behind each cell stays interactive. Output is
clean()ed and JSON round-trips.Test plan
tests/test_experiments_tools.py, all driven end to end throughexecute_tool_call: default grid against the course-notes table, per-cell generators, the blank-cell and replication cases, JSON serialisability, both validation errors, schema rejection of unknown keys (SEC-15), and registry membership._tools/trade_off_table.pyis fully covered.ruff check .andmypy src/process_improveclean.Checklist
pyproject.toml(1.63.0 -> 1.64.0, MINOR: new tool), withCITATION.cffkept in syncruff check .passesCHANGELOG.mdupdatedGenerated by Claude Code