Skip to content

v3.15.5: honest coverage and quality results

Choose a tag to compare

@proffesor-for-testing proffesor-for-testing released this 12 Oct 18:04
· 565 commits to main since this release
d65ede1

What's New

Honest coverage and quality results: AQE no longer reports a fake 0%, approves
without evidence, or saves a coverage estimate as if it were measured. Also
fixes aqe code index measuring the wrong folder.

Fixed

  • aqe code index measures the real project (#994, #997). The project
    root is now found from the target you pass: the nearest folder with
    package.json, .git, go.mod, Cargo.toml or pyproject.toml, walking up.
    aqe code index . at a monorepo root measures the root. Previously it often
    measured src/ or .agentic-qe/workers and reported 0 tests. It never falls
    back to the filesystem root.
  • Coverage analysis on a subfolder or an existing report (#995, #1000):
    • a subfolder target (src/) runs the tests from the project root and reports
      only the files under the target, instead of a "measured" 0%;
    • an existing report (lcov.info, coverage-final.json,
      coverage-summary.json, Cobertura XML) can be passed as the target;
    • when nothing measured the target, the result is unmeasured, not 0%. A
      file that ran with 0 hits is still a real 0%;
    • nothing is written into your project (coverage/, .vitest/, results
      files); reports go to a temp folder that is always removed;
    • the test runner is started without a shell or npx (nothing is
      downloaded), and its whole process group is stopped on timeout or when the
      MCP server exits, so no test processes are left running;
    • aqe coverage -F json prints only JSON; metrics a report never collected
      (for example branch data) show as n/a and don't count against the threshold.
  • quality_assess no longer approves without evidence (#996, #1001).
    When a requested metric is unmeasured, the result is
    status: insufficient-evidence with the missing metrics listed, and it
    escalates instead of approving. A measured metric that fails its threshold
    blocks even when others are unmeasured. Request and configured thresholds are
    honoured. The fixed security score of 85 is gone (unmeasured is reported as
    unmeasured).
  • Coverage estimates are never saved as measured evidence (#1008, #1013).
    Static estimates, runs where the runner was missing, crashed or timed out, and
    coverage data submitted by callers are no longer stored as coverage:latest
    or quality evidence. A background job that overwrote the last measurement
    with an empty record every few seconds is fixed.
  • Security: sharp updated to 0.35.5 (#993).

Changed

  • aqe coverage exits 3 when the target is unmeasured, and 0 (not 1)
    for a report without branch or function data that meets the threshold on the
    metrics it has.
  • aqe quality exits 3 for insufficient evidence; aqe ci's quality
    artifact uses the same insufficient-evidence status.
  • quality_assess validates metrics (a non-empty list of known names) and
    threshold (0–100). Ungated static approvals report confidence 0.7 at most.
  • Coverage persistence is fail-closed: programmatic callers of the coverage
    analysis API must pass measured: true for results to be stored.

Known issues

  • Five of the seven quality-gate metrics have no command that records them yet,
    so quality_assess --runGate can't pass for most projects (#1009).
  • The coverage runner's Windows path is unit-tested only, not run on Windows.

Getting Started

npx agentic-qe init --auto

See CHANGELOG for full details.