Skip to content

Report when the gold check had nothing to check against - #2293

Closed
ATMAECHO wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
ATMAECHO:gold-check-unimplemented
Closed

Report when the gold check had nothing to check against#2293
ATMAECHO wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
ATMAECHO:gold-check-unimplemented

Conversation

@ATMAECHO

@ATMAECHO ATMAECHO commented Aug 7, 2026

Copy link
Copy Markdown

The problem

Task.validate returns True unconditionally, so a taskset that does not override it
passes the gold check for every item without anything having been checked.

Of the five in-tree tasksets defining a Task subclass, only lean overrides validate.
harbor, openenv, textarena and nemo_gym inherit the base implementation. For those,
validate --only-gold reports valid_rate: 1.0 — which reads as a passing check, but
measures nothing. The failure is silent and flattering: absence of a check is reported as
agreement.

The change

34 lines in verifiers/v1/cli/validate.py:

  • gold_implemented(task) — whether the taskset actually wrote a gold check
    (type(task).validate is not Task.validate, so inheriting from an overriding class
    still counts).
  • _run_gold marks each row checked: true|false.
  • summarize reports gold_unchecked in gold and all modes (_gold_row reads the
    nested gold row in all mode).
  • run_validate warns once, naming the classes that don't override validate.

Validation behaviour is unchanged. This only separates "checked and valid" from
"never checked".

A stronger alternative, if you'd prefer it

Add an unchecked value to reason so it fails loudly rather than being reported in the
summary. I kept this additive because that would touch FINAL_REASONS, _is_final and
resume semantics — happy to switch if you'd rather have the louder version.

Verification

Per AGENTS.md ("to check your own work, write a temporary script instead of committing new
tests") I used a temporary script, not a committed test. It checks:

  • the predicate on synthetic cases: inherits Task.validate, overrides it, and inherits a
    class that overrides it;
  • all five in-tree taskset classes, imported live — 4 unchecked, lean checked;
  • summarize in gold, all and setup modes, including a row with no checked field;
  • a negative control — the old always-true predicate — which fires, so the checks aren't
    passing vacuously.

All pass. uv run ruff check and uv run ruff format --check are clean.

One caveat I'd rather state than hide: I could not run uv run pytest tests/ on this
machine. verifiers/v1/runtimes/limiters.py imports fcntl, which is Unix-only, so the
package doesn't import on Windows. I stashed the change, collected, un-stashed, and
collected again — the failure set is byte-identical (38 both ways, same 6 collection
errors), so nothing here is caused by this change. Still worth a CI run on Linux.

Note

Report unchecked gold validations when a task does not override Task.validate

  • Adds a gold_unchecked count to the summary in gold and all modes, tracking results that passed without a real gold check being implemented.
  • Each result row from _run_gold now includes a checked boolean derived from whether the task's class overrides Task.validate.
  • Before validation runs, run_validate logs a warning listing any task classes whose gold checks will always pass due to missing validate overrides.

Macroscope summarized 4a74e93.

`Task.validate` returns `True` unconditionally, so a taskset that does not
override it passes the gold check for every item without anything having been
checked. Of the five in-tree tasksets defining a `Task` subclass, only `lean`
overrides `validate`; `harbor`, `openenv`, `textarena` and `nemo_gym` inherit
the base implementation. For those, `validate --only-gold` reports
`valid_rate: 1.0`, which reads as a passing check but measures nothing.

Mark such rows with `checked: false`, count them as `gold_unchecked` in the
summary, and warn once per taskset naming the classes involved. Validation
behaviour is unchanged; this only distinguishes "checked and valid" from
"never checked".
xeophon added a commit that referenced this pull request Aug 29, 2026
## Overview

Represent model-free gold validation as a tri-state result so tasksets
without a gold check are reported as unchecked instead of valid.

This is an alternative implementation of the issue identified by
@ATMAECHO in #2293. It makes the absence of a check part of the
Task.validate contract instead of inferring it from method overrides.

## Details

- Make Task.validate return None by default while preserving explicit
True and False results.
- Persist unchecked as a terminal result with valid: null, including
resume and summary handling.
- Preserve invalid, timeout, and error precedence when gold and setup
checks are combined.
- Show unchecked outcomes in the validation dashboard and leave
valid_rate unset when nothing was checked.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changing the default `Task.validate` from `True` to `None` shifts
reporting for any task that relied on the inherited default without
overriding the method.
> 
> **Overview**
> Gold validation is now **tri-state**: `Task.validate` may return
`None` when there is no model-free check, instead of implicitly counting
those tasks as valid.
> 
> The default `Task.validate` returns **`None`** (was `True`). The
validate CLI persists **`reason: "unchecked"`** with **`valid: null`**,
treats unchecked as **terminal** for resume/summary, and shows it in the
**dashboard** (dim style). **`valid_rate`** is computed only over tasks
that were actually checked (valid + invalid); it stays unset when
nothing was checked. Combined gold+setup runs keep
**error/timeout/invalid** precedence and surface **unchecked** when
neither sub-check failed.
> 
> **Lean** tasks without a gold proof now return **`None`** instead of
passing validation by default.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
d8b1e4a. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- Macroscope's pull request summary starts here -->
<!-- Macroscope will only edit the content between these invisible
markers, and the markers themselves will not be visible in the GitHub
rendered markdown. -->
<!-- If you delete either of the start / end markers from your PR's
description, Macroscope will append its summary at the bottom of the
description. -->
> [!NOTE]
> ### Add `unchecked` tri-state outcome to validation pipeline and
dashboard
> Introduces `valid: bool | None` across the validation system so tasks
without a model-free check are reported as `unchecked` instead of
implicitly passing.
>
> - Changes `Task.validate` and `LeanTask.validate` to return `None`
when no gold proof exists, rather than defaulting to `True`
> - Updates `FINAL_VALUES`, `_classify`, `_row`, `_all_reason`, and
`_all_error` in
[validate.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2466/files#diff-f9da541290540aab599f22057f02b6cfcd418d89dfa7bf9cbe3c325d970d5a67)
to handle the tri-state: `unchecked` is terminal, excluded from error
aggregation, and `valid_rate` is computed over checked items only
> - Adds `unchecked` styling and mark to the dashboard in
[validate.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2466/files#diff-2ded55a5b55a7c27d925b91343c516de853d005e99d07590ceb341bdc9e9149e)
> - Behavioral Change: `LeanTask.validate` no longer returns `True` for
tasks without a gold proof — callers that treated any truthy return as
"valid" will now see `None`; persisted result rows store `valid=None`
for unchecked outcomes
>
> <!-- Macroscope's review summary starts here -->
>
> <sup><a href="https://app.macroscope.com">Macroscope</a> summarized
d8b1e4a.</sup>
> <!-- Macroscope's review summary ends here -->
>
<!-- Macroscope's pull request summary ends here -->
@xeophon

xeophon commented Aug 29, 2026

Copy link
Copy Markdown
Member

superseded by #2266

@xeophon xeophon closed this Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants