`scan.report.json` is constructed without `status` or `errors` and never reads them from step reports

[gadievron]

Evidence provenance (added 2026-08-22). The run-derived figures in this issue come from a
private scan that a reader cannot reproduce (raptor-e2e-20260818, a 9.85-hour run against
a Python target). They are reported for completeness, not as independently checkable evidence.
Every code claim below is pinned to public source at b5019628 and is checkable; where a
conclusion rests on the private numbers rather than on the code, treat it as my report of what
that run showed. This label was missing when the issue was filed.

Summary

_write_scan_report builds the aggregate StepReport(step="scan", ...) with no status= and no
errors= keyword, and reads only cost_usd, duration_seconds, token_usage and step from the
per-step reports it aggregates. StepReport.status defaults to "success", so the aggregate report
is unconditionally green. Repo-wide there is exactly one writer of status="error" —
core/step_report.py:53 — and it sits in an except block that re-raises, so a step that
catches its own exception and records it in ctx.errors keeps status="success".

Evidence

libs/openant-core/core/scanner.py:1268-1318 (abridged — summary={ ... } marks omitted keys) (HEAD b501962) — the constructor, abbreviated to the
keywords actually passed:

    scan_report = StepReport(
        step="scan",
        summary={ ... },
        inputs={"repo_path": ...},
        outputs={ ... },
        cost_usd=round(total_cost, 6),
        duration_seconds=round(total_duration, 2),
        token_usage={ ... },
    )

No status=, no errors=. The only reads from step_reports are at core/scanner.py:1259-1266
(cost_usd, duration_seconds, token_usage) and :1274 (sr.get("step") for
steps_completed). Confirm:

cd libs/openant-core
awk 'NR>=1253 && NR<=1322' core/scanner.py | grep -n 'status\|errors'
#   -> 33:            "parse_errors": result.parse_errors,      (from ScanResult, not a step report)

libs/openant-core/core/step_report.py:50-57 — the sole writer of a non-default status:

    try:
        yield report
    except Exception as exc:
        report.status = "error"
        report.errors.append(str(exc))
        print(f"[{step}] ERROR: {exc}", file=sys.stderr)
        traceback.print_exc(file=sys.stderr)
        raise

Every status assignment on a step report, repo-wide, excluding tests:

grep -rn 'status = "' --include='*.py' libs/openant-core | grep -v '/tests/' | grep -v 'test_'

Six of the ten hits assign a step-report status field; the other four are unrelated local variables
(experiment.py:499, checkpoint.py:202, result_collector.py:104, cli.py:945):

core/step_report.py:53   report.status = "error"      <- only on an ESCAPING exception
core/scanner.py:416      ctx.status = "skipped"
core/scanner.py:485      ctx.status = "skipped"
core/scanner.py:788      ctx.status = "skipped"
core/scanner.py:922      ctx.status = "skipped"
core/scanner.py:1024     ctx.status = "skipped"

There is no path that sets a step report to anything other than "success" or "skipped" without
the exception escaping the with block.

On the live run

RUN=~/.openant/projects/gadievron/raptor-e2e-20260818/scans/7dbf9c691d7/python
python3 - <<'EOF'
import json, glob, os
for f in sorted(glob.glob(os.path.expanduser("$RUN/*.report.json"))):
    d = json.load(open(f))
    print("%-30s status=%-8s errors=%-2d cost=%-12s total_tokens=%s" % (
        os.path.basename(f), d.get("status"), len(d.get("errors", [])),
        d.get("cost_usd"), d.get("token_usage", {}).get("total_tokens")))
EOF

analyze.report.json            status=success  errors=0  cost=0.0          total_tokens=216077375
app-context.report.json        status=success  errors=0  cost=0.02958      total_tokens=4400
build-output.report.json       status=success  errors=0  cost=0.0          total_tokens=0
dynamic-test.report.json       status=success  errors=0  cost=0.073281     total_tokens=6411
enhance.report.json            status=success  errors=0  cost=2062.294038  total_tokens=655640726
llm-reachability.report.json   status=success  errors=0  cost=0.0          total_tokens=9288595
parse.report.json              status=success  errors=0  cost=0.0          total_tokens=0
report.report.json             status=success  errors=0  cost=3.145047     total_tokens=447741
scan.report.json               status=success  errors=0  cost=2065.541946  total_tokens=1053918215
verify.report.json             status=success  errors=0  cost=0.0          total_tokens=172452967

All ten reports are status="success" with an empty errors list. In the same directory:

python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(d['summary'])" $RUN/verify.report.json
#   {'findings_input': 223, 'findings_verified': 223, 'agreed': 1, 'disagreed': 40,
#    'confirmed_vulnerabilities': 1, 'needs_review': 134, 'error_count': 48}

verify.report.json reports error_count: 48 in its own summary while its errors list is empty
and its status is "success"; scan.report.json reports errors: [] and lists verify under
steps_completed. llm-reachability reports cost_usd: 0.0 beside total_tokens: 9,288,595, and
verify reports cost_usd: 0.0 beside total_tokens: 172,452,967 — neither is flagged.

The signal is also dropped on the way to the Go consumer

Even the per-step errors that are recorded never reach the rendered report.
libs/openant-core/openant/cli.py:1119-1125 projects each step report into the payload the Go CLI
consumes, and copies five fields — errors is not among them:

                step_reports_data.append({
                    "step": sr.get("step", "unknown"),
                    "duration": dur_str,
                    "cost": cost_str,
                    "status": sr.get("status", "unknown"),
                    "timestamp": sr.get("timestamp", ""),
                })

apps/openant-cli/internal/report/types.go:273-280 then colours that status:

// StatusColor returns a Tailwind text color class based on step status.
func (s StepReport) StatusColor() string {
	switch s.Status {
	case "success":
		return "text-green-400"
	case "error":
		return "text-red-400"

So the HTML report has no error channel at all: errors is dropped at the projection, and status
— which cannot be non-success for a handled failure — decides the colour. Every phase renders green.

Age of the deciding file

git log --oneline -- libs/openant-core/core/step_report.py
#   d710b90 feat: parallelization, HTML report overhaul, Zig parser, dynamic test hardening (#23)
#   0d729f6 initial commit
git log -1 --format='%ad' --date=short -- libs/openant-core/core/step_report.py
#   2026-04-14

Two commits; untouched since 2026-04-14.

Why it matters

status is the machine-readable answer to "did this scan work?". Within this repo it is the field a
CI wrapper, a dashboard, or the Go CLI would branch on, and today it cannot be false for any failure
that a step handles rather than propagates. That includes the two most common degradation shapes in
this pipeline: a step that catches an exception and appends to ctx.errors (see the report step at
core/scanner.py:1059-1078), and a step that completes but whose own summary records failures
(verify with error_count: 48).

Because _write_scan_report never reads status or errors from the reports it aggregates, the
aggregate is strictly less informative than the per-step files it summarises — a partially-degraded
scan and a clean one are indistinguishable at the top level.

Suggested fix

Derive the status from evidence rather than from exception escape, in the finally block:

1. In core/step_report.py, after the body completes, set report.status = "partial" when
   report.errors is non-empty and no exception escaped. Use "partial", not "error" — nothing
   that currently branches on "error" changes behaviour, and "success" stops being a lie.
2. In _write_scan_report, aggregate over the step reports it already loads: pass
   errors=[e for sr in step_reports for e in sr.get("errors", [])] and set the scan status to the
   worst per-step status (error > partial > skipped > success).
3. REQUIRED — steps 1-2 are inert without it. Make steps that count per-item failures surface
   them where the status derivation can see them: either append to ctx.errors, or have the
   derivation also consider integer error_count / errors keys in ctx.summary. Concretely, let
   verify append to ctx.errors when
   error_count > 0, so the new "partial" status has something to fire on.
4. Add a test that drives step_context with a body that appends to ctx.errors without raising,
   and asserts the written report is not "success".

Corrected 2026-08-21 — step 3 was filed as "Optionally". That was wrong. Steps 1-2 alone would
have left this exact run entirely green: no step ever appended to ctx.errors (all ten step reports
carry errors: []), and the only two ctx.errors.append sites in the codebase are both in the
report step (core/scanner.py:1068, :1078). Verify's failures live in summary["error_count"]
and never touch ctx.errors, so a status derived from report.errors would still read "success"
and the aggregate would faithfully aggregate ten successes. The defect this issue documents survives
its own fix unless step 3 is done.

What I am not claiming

• I am not claiming any specific external consumer breaks. The impact statement above is scoped
  to consumers within this repo; I have not surveyed downstream users of scan.report.json.
• The cost_usd: 0.0 values above are reported as an observation that the status field does not
  react to them. Their cause (pricing-registry lookup) is a separate matter and is not diagnosed here.
• I have not measured how often a step catches-and-continues in practice across other targets; the
  evidence is one Python run plus the static fact that only an escaping exception can change the field.

---

Dependency on #300 (added 2026-08-22, verified at b5019628). Fix item 3 above is marked
REQUIRED — steps 1-2 are inert without it, and it asks the status derivation to consider an integer
error_count / errors key in ctx.summary. On the standalone openant verify path that key is
not present to consider — openant/cli.py:498-504 builds the summary with five keys:

498|                    vctx.summary = {
499|                        "findings_input": vresult.findings_input,
500|                        "findings_verified": vresult.findings_verified,
501|                        "agreed": vresult.agreed,
502|                        "disagreed": vresult.disagreed,
503|                        "confirmed_vulnerabilities": vresult.confirmed_vulnerabilities,
504|                    }

error_count and needs_review are absent, which is what #300 reports. So on that command this issue's
required item has no key to read until #300's five-line addition lands — verified by the awk over
cli.py:496-506 quoted above. The two are one
work item and should be scheduled together.

[gadievron]

One more site in the same class, found while checking the standalone commands.

The standalone openant verify step report omits the two fields the pipeline's equivalent
includes. libs/openant-core/core/scanner.py:877-885 builds the verify step summary with
needs_review and error_count; libs/openant-core/openant/cli.py:498-504 and :560-566 build the
standalone command's summary without them.

cd libs/openant-core
sed -n '877,885p' core/scanner.py     # pipeline: includes needs_review + error_count
sed -n '498,504p' openant/cli.py      # standalone: neither
sed -n '560,566p' openant/cli.py      # same

VerifyResult.to_dict (core/schemas.py:293-304) does emit both, so the stdout envelope from the
standalone command is honest — only its step report is not. On our run those two fields are what
carry needs_review: 134 and error_count: 48, i.e. 182 of 223 findings (81.6%); a consumer reading
the standalone command's step report sees neither.

Same shape as this issue: the aggregate is less informative than the data available to it. Folding it
here rather than filing separately, since the fix is the same one-line-per-site addition.

[gadievron]

Superseded by #300 — correcting my own comment above.

That comment said "folding it here rather than filing separately". I then filed it separately as
#300, because the fix sites are different files and functions (openant/cli.py:498-504 and
:560-566) from the ones this issue covers (core/scanner.py:_write_scan_report and
openant/cli.py:1119-1125), so a fix here would not reach them.

Nothing in the comment's evidence changes — only its disposition. This issue's scope is unchanged.

[gadievron]

Correction to the Suggested fix ordering in this report: step 3 is not optional — it is the step
that makes the fix fire on the failure this issue documents.

I listed step 3 ("let a step opt into a stronger signal by having verify append to ctx.errors when
error_count > 0") as "Optionally". A verification pass shows steps 1 and 2 alone would have left
this exact run entirely green:

• No step ever appended to ctx.errors on this run — all ten step reports carry errors: [].
• The only two ctx.errors.append sites in the codebase are both in the report step
  (core/scanner.py:1068, :1078).
• Verify's 48 failures never touch ctx.errors; they live in summary["error_count"].

So a status derived from report.errors (step 1) would still read "success" for verify, and the
aggregate (step 2) would faithfully aggregate ten successes. The defect this issue is about —
verify.report.json reporting status: "success" beside summary.error_count: 48 — survives its own
fix.

Re-ranked fix:

1. Derive step status from evidence in the finally block (as filed).
2. Aggregate status and errors in _write_scan_report (as filed).
3. Required, not optional: have steps that count per-item failures surface them where the status
   derivation can see them — either append to ctx.errors, or have the derivation also consider
   integer error_count / errors keys in ctx.summary. Without this, steps 1-2 are inert for every
   failure mode observed in this run.

A minor precision note on the report body: I wrote "exactly one writer of status = "error"". There is
also core/schemas.py:33, which sets that key on the unrelated CLI envelope. The sentence immediately
after scopes the claim to step reports, so the conclusion is unaffected — recording it for accuracy.

[gadievron]

Prior art I should have cited: PR #257 already named this defect as a known follow-up. Its body,
under "Known follow-ups, out of scope for this PR (each is a separate mechanism)", lists verbatim:

• report-step summary/disclosure failure leaves status="success" (core/scanner.py),
• dynamic-test/report skips are recorded after the pipeline_output snapshot so remain absent from
  the summary,

So the report-step half of this issue is a residual of #257's deliberately bounded scope, not an
oversight — and my issue should have opened by saying so.

Also relevant: PR #254 made the five catch-and-continue sites set ctx.status = "skipped", which
is why the step-level status works for those five and the report step remains the exception.

What my issue adds beyond #257's one-line note is the aggregate half — that _write_scan_report
passes no status=/errors= at all and never reads either from the step reports it aggregates — plus
the runtime evidence and the Go-side projection dropping errors. That part is not on #257's list.

[gadievron]

Fix opened in #376 (CI green, all checks passing). The scan report now derives its status from evidence — step_context yields 'partial' for caught-by-design errors (ctx.errors) or counted failures (summary.error_count, covering both verify's error_count=48 shape and analyze's per-unit errors, which now carries the flat key); _write_scan_report aggregates the worst per-step status (error > partial > skipped > success) and every per-step error; and ScanResult carries scan_status/scan_errors so the envelope and webui see a degraded scan without re-reading the file.

Follow-ups recorded here from review: the Go StatusColor switch doesn't know 'partial' yet, and the CLI's report-data projection drops the errors field (the issue's own Go-side observation) — both are Go-surface changes deliberately left to their own PR; standalone verify/analyze paths write error_count to their summaries only on the scan path (the standalone verbs' summaries were updated for analyze; the standalone verify summary still relies on its report's error_count key already present).