How should source-only deepagents-evals CLI bugs be reported?

[Lucien (dakjdakd)]

I found a reproducible exit-code bug in the deepagents-evals run CLI on monorepo main at 0c756c4f8. libs/evals/tests/evals/pytest_reporter.py::pytest_sessionfinish() intentionally resets pytest exit status 1 to 0 after failed tests so aggregation can finish. libs/evals/deepagents_evals/cli.py::_cmd_run() checks only that subprocess status, so a run whose report contains {"failed": 1} exits 0. Automated callers therefore see success for a failed evaluation. trials and aggregate already inspect report failure counts.

A network-free CLI reproduction:

from pathlib import Path
from subprocess import CompletedProcess
from tempfile import TemporaryDirectory
from unittest.mock import patch

from deepagents_evals import cli

with TemporaryDirectory() as tmp:
    report = Path(tmp) / "report.json"

    def fake_run(cmd, **kwargs):
        Path(cmd[cmd.index("--evals-report-file") + 1]).write_text('{"failed": 1}')
        return CompletedProcess(cmd, 0)  # the reporter masks pytest exit 1

    with patch.object(cli.subprocess, "run", side_effect=fake_run):
        print(cli.main(["run", "--model", "fake:model", "--report", str(report)]))

Observed on main: 0; expected: 1 when the report records a failed evaluation. A separate check through the real reporter hooks produced pytest_exit=0 and report_failed=1. Related #3221 introduced the unified CLI, and #2932 addresses the no-tests-run exit, not this path.

I would report this with the Bug Report template, but its required checklist says the bug must reproduce with the latest released version. The published deepagents-evals==0.0.1 package is a placeholder; this CLI exists only in monorepo source, so I cannot truthfully check that item. What is the preferred route for a source-only evals bug? If this discussion can serve as the approved contribution link, I can provide a focused fix and regression tests once a maintainer approves the work and assigns me (@dakjdakd). Otherwise I can file an Issue through the route you recommend.

[Riconas Sun (RiconasSun)]

I reproduced this on current main at 44bd956bf8ca86c7cf0831132d69d38e06dd1573.

For the source-only run path:

• the eval report records failed=1
• pytest_reporter intentionally normalizes pytest exit status 1 → 0
• _cmd_run currently trusts the subprocess return code
• result: deepagents-evals run returns 0 even though the report contains failures

I also checked trials and aggregate; those already derive failure status from the report/summary counts and correctly return 1.

The smallest fix appears to be in libs/evals/deepagents_evals/cli.py, keeping the reporter behavior unchanged and making run derive eval failure status from a fresh report as well, including when the user does not pass --report.

If this Discussion is sufficient as the contribution approval thread, I'm happy to add focused regression tests and a minimal fix. Otherwise, I can file a separate Issue first.

[Lucien (dakjdakd)]

Thanks, Riconas Sun (@RiconasSun), for independently confirming this on current main. I opened this discussion to find the right reporting route for the source-only evals CLI. I already have a focused fix and regression tests pushed to my fork: dakjdakd/evals/report-run-failures (630f37a3c). The patch makes run use a fresh report's failure count for its exit status, including when --report is omitted, while leaving the reporter's aggregation behavior unchanged.

I'd like to submit this existing patch and avoid duplicate work. Could a maintainer confirm how to report this source-only bug and whether I can be assigned to implement it? The Bug Report form requires a latest-release reproduction that I cannot truthfully confirm for the placeholder PyPI package. If an Issue is required for assignment and PR linkage, please advise how to handle that form requirement; I can then link the focused PR to the approved Issue.