Skip to content

--test-command can verify against a richer environment than CI, producing a false green #22

Description

@lorenzoliuzzo

Measured on a real run against an out-of-fleet target.

my-coder reported the run as fully successful:

{"outcome": "success", "detail": "opened draft PR #10", "issue": 2,
 "files_touched": ["main.py", "tests/test_dataset.py"],
 "tests_passed": true, "cost_usd": 0.77}

The target repo's CI then failed on that exact commit:

ImportError while importing test module tests/test_dataset.py
    from torchsummary import summary
E   ModuleNotFoundError: No module named 'torchsummary'

The session wrote a test importing a module whose dependency the target repo had never declared. --test-command had been pointed at the operator's existing local .venv (per #15/#20, so the session would not spend its wall clock on a cold install), and that venv happened to have the package installed from earlier ad-hoc work. So _tests_pass returned True against an environment strictly richer than the declared one.

tests_passed: true currently means "green in whatever environment the operator supplied", but it is consumed — by the PR readiness checklist and by anyone reading the ledger — as "green". Those differ exactly when the target's declared dependencies are incomplete, which is precisely the bug class a fresh-environment run would catch.

This is the direct trade-off against #20: the cheap path (reuse a warm environment) and the trustworthy path (build from declared dependencies) are opposed, and right now the tool takes the cheap path while reporting the trustworthy one's verdict.

Acceptance criteria

  • tests_passed distinguishes "green in a supplied environment" from "green in one built from the target's declared dependencies"; the ledger and the PR body carry which one it was.
  • When a supplied --test-command is used, the PR body says so explicitly rather than implying an unqualified green.
  • Ideally the declared-dependency environment is resolved from a warm cache, so the trustworthy path is also the cheap one (see Building the target environment from scratch consumes the whole wall clock #20) rather than a choice between them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions