You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The target repo's CI then failed on that exact commit:
ImportError while importing test module tests/test_dataset.py
from torchsummary import summary
E ModuleNotFoundError: No module named 'torchsummary'
The session wrote a test importing a module whose dependency the target repo had never declared. --test-command had been pointed at the operator's existing local .venv (per #15/#20, so the session would not spend its wall clock on a cold install), and that venv happened to have the package installed from earlier ad-hoc work. So _tests_pass returned True against an environment strictly richer than the declared one.
tests_passed: true currently means "green in whatever environment the operator supplied", but it is consumed — by the PR readiness checklist and by anyone reading the ledger — as "green". Those differ exactly when the target's declared dependencies are incomplete, which is precisely the bug class a fresh-environment run would catch.
This is the direct trade-off against #20: the cheap path (reuse a warm environment) and the trustworthy path (build from declared dependencies) are opposed, and right now the tool takes the cheap path while reporting the trustworthy one's verdict.
Acceptance criteria
tests_passed distinguishes "green in a supplied environment" from "green in one built from the target's declared dependencies"; the ledger and the PR body carry which one it was.
When a supplied --test-command is used, the PR body says so explicitly rather than implying an unqualified green.
Measured on a real run against an out-of-fleet target.
my-coder reported the run as fully successful:
{"outcome": "success", "detail": "opened draft PR #10", "issue": 2, "files_touched": ["main.py", "tests/test_dataset.py"], "tests_passed": true, "cost_usd": 0.77}The target repo's CI then failed on that exact commit:
The session wrote a test importing a module whose dependency the target repo had never declared.
--test-commandhad been pointed at the operator's existing local.venv(per #15/#20, so the session would not spend its wall clock on a cold install), and that venv happened to have the package installed from earlier ad-hoc work. So_tests_passreturnedTrueagainst an environment strictly richer than the declared one.tests_passed: truecurrently means "green in whatever environment the operator supplied", but it is consumed — by the PR readiness checklist and by anyone reading the ledger — as "green". Those differ exactly when the target's declared dependencies are incomplete, which is precisely the bug class a fresh-environment run would catch.This is the direct trade-off against #20: the cheap path (reuse a warm environment) and the trustworthy path (build from declared dependencies) are opposed, and right now the tool takes the cheap path while reporting the trustworthy one's verdict.
Acceptance criteria
tests_passeddistinguishes "green in a supplied environment" from "green in one built from the target's declared dependencies"; the ledger and the PR body carry which one it was.--test-commandis used, the PR body says so explicitly rather than implying an unqualified green.