Summary
evaluate_submission() can mark a run/submission as completed even when no results.json exists.
In validator/src/eval_backend/services/eval_runner.py, _prepare_results() returns:
- metrics:
{"results_missing": True}
- score:
None
when results.json is missing.
But evaluate_submission() later unconditionally sets:
run.status = "completed"
submission.status = "completed"
This creates a false-success state.
Impact
A failed or incomplete evaluation can be reported as successful:
- API and PR reporting show
completed while score is pending/null
- consumers cannot reliably distinguish true completion vs missing outputs
- evaluation integrity is weakened by silent success on missing artifacts
This pattern appears in merged PR timelines where results_missing: True was reported under completed status (e.g., PR #1, PR #2).
Proposed fix
Treat missing results as a failure (or explicit non-success terminal state), not success:
- In
evaluate_submission(), after _prepare_results(...), if metrics.get("results_missing") is True (or equivalent missing-output condition), set:
run.status = "failed" (or incomplete)
submission.status = "failed" (or aligned non-success status)
- clear
run.error/run.message stating results.json was not produced
- Keep
completed only for valid parsed outputs.
- Preserve diagnosability by keeping
results_missing in metrics.
Acceptance
- Missing
results.json no longer yields completed
- API/PR reporting clearly indicates non-success status for missing outputs
- Existing success path for valid results remains unchanged
- Unit tests cover:
- missing results -> non-completed terminal status
- valid results -> completed
- malformed/empty outputs -> explicit failure message
Summary
evaluate_submission()can mark a run/submission ascompletedeven when noresults.jsonexists.In
validator/src/eval_backend/services/eval_runner.py,_prepare_results()returns:{"results_missing": True}Nonewhen
results.jsonis missing.But
evaluate_submission()later unconditionally sets:run.status = "completed"submission.status = "completed"This creates a false-success state.
Impact
A failed or incomplete evaluation can be reported as successful:
completedwhile score is pending/nullThis pattern appears in merged PR timelines where
results_missing: Truewas reported undercompletedstatus (e.g., PR #1, PR #2).Proposed fix
Treat missing results as a failure (or explicit non-success terminal state), not success:
evaluate_submission(), after_prepare_results(...), ifmetrics.get("results_missing") is True(or equivalent missing-output condition), set:run.status = "failed"(orincomplete)submission.status = "failed"(or aligned non-success status)run.error/run.messagestatingresults.jsonwas not producedcompletedonly for valid parsed outputs.results_missingin metrics.Acceptance
results.jsonno longer yieldscompleted