feat(devtools): bound test execution inside agent jobs - #4540
Conversation
Nine of every ten pytest runs in the last two days were started by lanes themselves, outside admission, with affected selections of a fifth to a half of the corpus. Inside an agent job the affected and complete tiers refuse with the job command to use, focused runs cap at two workers, and bare pytest exits at configuration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGi8wLWGR2HYBh8p8fXFXz
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0eff95cc96
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if _has_worker_flag(selection): | ||
| return [] | ||
| requested = configured_pytest_worker_request(os.environ) | ||
| requested = agent_worker_cap(configured_pytest_worker_request(os.environ), os.environ) |
There was a problem hiding this comment.
When an agent invokes devtools test ... -n 8 or --numprocesses=auto, _has_worker_flag returns before this new cap is applied, so build_pytest_cmd forwards the unbounded request unchanged. This defeats the resource bound and can recreate the host oversubscription the change is intended to prevent; explicit worker arguments should also be rewritten or rejected when they exceed AGENT_MAX_PYTEST_WORKERS.
Useful? React with 👍 / 👎.
| if requested is None or requested < 1: | ||
| return AGENT_MAX_PYTEST_WORKERS | ||
| return min(requested, AGENT_MAX_PYTEST_WORKERS) |
There was a problem hiding this comment.
When an agent sets the documented POLYLOGUE_PYTEST_WORKERS=0 override to request an in-process run, configured_pytest_worker_request returns 0, but this branch converts it to 2. That unexpectedly enables xdist and can break focused runs that deliberately disable multiprocessing; only the absent request should receive the agent default, while zero is already within the cap.
Useful? React with 👍 / 👎.
| bare = refuse_bare_pytest(os.environ) | ||
| if bare is not None: | ||
| raise pytest.UsageError(bare) |
There was a problem hiding this comment.
When an agent invokes pytest --noconftest, documented by pytest --help as “Don't load any conftest.py files,” this module is never imported and pytest_configure cannot call the refusal; a run such as python -m pytest --noconftest -o addopts='' therefore executes under SINNIXD_PRINCIPAL=agent-control. This still permits broad, unbounded runs outside the managed harness, so the prohibition must be enforced at an entrypoint that pytest arguments cannot disable.
AGENTS.md reference: AGENTS.md:L139-L140
Useful? React with 👍 / 👎.
| if not inside_agent_job(env) or "--quick" in argv: | ||
| return None |
There was a problem hiding this comment.
When an agent invokes the existing devtools verify --commit mode, this predicate rejects it solely because it lacks --quick, even though _scope classifies commit as NON_TEST and build_verify_steps omits every pytest step when commit=True. The new restriction therefore blocks a safe static-only gate despite claiming to prohibit test tiers; decide from the parsed verification scope rather than whitelisting one spelling.
Useful? React with 👍 / 👎.
| refusal = refuse_verify_tier(list(argv or []), os.environ) | ||
| if refusal is not None: | ||
| sys.stderr.write(refusal + "\n") | ||
| return 2 |
There was a problem hiding this comment.
When an agent requests devtools verify --json, the refusal occurs before argument parsing and writes only prose to stderr, so stdout is empty despite the command's machine-readable mode. Automation consuming the verifier's JSON result cannot classify this expected refusal and instead receives a parse failure; parse --json first and emit a structured terminal result through the existing _emit route.
Useful? React with 👍 / 👎.
#4682) ## Summary The corpus and the runner's affected tier run at one fixed width of eight workers, sized to the pytest pool's 12 GiB cgroup ceiling. The descriptor no longer overrides the width, the pytest slot launch timeout is 7,200 s, and the selection gate reads the width from the constant. ## Problem `CORPUS_MAX_WORKERS = 2` made a full corpus run take about seven hours (measured 2026-09-05: 5% after 40 minutes), and the runner's affected tier could not finish a stale-graph run inside its 3,600 s slot timeout, so every pull request's required `verify` check timed out (runs 3669329 and 600822, both at 38%). ## Solution Width 8 (47 minutes for 20,860 tests, measured uncontended 2026-09-03), one constant for both tiers, timeout 7,200 s. ## Verification - `devtools verify --quick`: all gates ok, including `testmon-selection` reporting `workers=8`. - Descriptor contract and slot tests updated for the new timeout. ## Residual risk Bead polylogue-m3018 asked for a modest width after eight workers plus lane-side pytest runs drove oomd kills on 2026-09-04; lane-side pytest is now refused (#4540, #4677) and the pool ceiling bounds the run. Browser responsiveness during a corpus run is the receipt m3018 still wants. Beads: polylogue-m3018 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DNGJ3awfNrsLaMdHgQZvid --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 participant
Summary
Inside an agent job (
SINNIXD_PRINCIPAL=agent-control),devtools verifywithout--quickrefuses and names the job to use,devtools testcaps at two workers, and barepytestraises a usage error at configuration.Problem
Audit of the last 48 hours: 1,170 pytest runs, 42 hours of wall time, peak 11 concurrent. 1,164 of them were started by lanes in their worktrees, not by verification jobs, so admission never saw the load. Affected selections were 3,398 to 10,899 tests (median 22% of the corpus). The worker contract forbids this in prose; prose did not hold.
Solution
One module,
devtools/agent_env.py, with three pure decisions read from the environment;verify._main, the worker-args helper inrun_tests, andtests/conftest.py::pytest_configureconsult it. Harness-launched runs carryPOLYLOGUE_PYTEST_RUN_IDand pass the conftest guard.Verification
Residuals
The declared
verify_affectedjob is the only test-tier route for a lane; the reactor dispatches it once per head.🤖 Generated with Claude Code
https://claude.ai/code/session_01YGi8wLWGR2HYBh8p8fXFXz