Skip to content

eval_runner marks submission as completed when results.json is missing #12

Description

@rust-toml

Summary

evaluate_submission() can mark a run/submission as completed even when no results.json exists.

In validator/src/eval_backend/services/eval_runner.py, _prepare_results() returns:

  • metrics: {"results_missing": True}
  • score: None

when results.json is missing.
But evaluate_submission() later unconditionally sets:

  • run.status = "completed"
  • submission.status = "completed"

This creates a false-success state.

Impact

A failed or incomplete evaluation can be reported as successful:

  • API and PR reporting show completed while score is pending/null
  • consumers cannot reliably distinguish true completion vs missing outputs
  • evaluation integrity is weakened by silent success on missing artifacts

This pattern appears in merged PR timelines where results_missing: True was reported under completed status (e.g., PR #1, PR #2).

Proposed fix

Treat missing results as a failure (or explicit non-success terminal state), not success:

  1. In evaluate_submission(), after _prepare_results(...), if metrics.get("results_missing") is True (or equivalent missing-output condition), set:
    • run.status = "failed" (or incomplete)
    • submission.status = "failed" (or aligned non-success status)
    • clear run.error/run.message stating results.json was not produced
  2. Keep completed only for valid parsed outputs.
  3. Preserve diagnosability by keeping results_missing in metrics.

Acceptance

  • Missing results.json no longer yields completed
  • API/PR reporting clearly indicates non-success status for missing outputs
  • Existing success path for valid results remains unchanged
  • Unit tests cover:
    • missing results -> non-completed terminal status
    • valid results -> completed
    • malformed/empty outputs -> explicit failure message

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions