Skip to content

fix(suite_C): describe the actual reason a precision level is skipped#46

Merged
JuhaoLiang1997 merged 1 commit into
mainfrom
fix/suite-c-stale-skip-reason
May 15, 2026
Merged

fix(suite_C): describe the actual reason a precision level is skipped#46
JuhaoLiang1997 merged 1 commit into
mainfrom
fix/suite-c-stale-skip-reason

Conversation

@JuhaoLiang1997
Copy link
Copy Markdown
Collaborator

Follow-up to the cleanup in #45. That PR removed the runner-declared quantization-backend gating logic and renamed the obvious skip-reason in the headline print (line 101), but two sibling references to the old strategy were missed:

  • The function-level docstring still claimed format selection intersects with runner.SUPPORTED_QUANTIZATIONS and warns on any format the runner doesn't declare.
  • The per-format final-summary line printed skipped (backend not in SUPPORTED_QUANTIZATION_BACKENDS) even though the skipped list now only ever holds the other full-precision baseline (e.g. FP16 on Ampere where the hw baseline is BF16).

Rewrite both so the docstring describes today's policy (always include the hw-supported full-precision baseline; dispatch every quantized level; let the inference subprocess decide hardware compatibility) and the skip-reason print matches what actually causes the entry.

The result.json field name precision_levels_skipped is kept — it's a stable schema field already indexed by the leaderboard and used by older results, so the name stays; only the human-readable strings around it are corrected.

No functional change.

Summary

Type of change

  • New platform support
  • Bug fix (runner, validator, leaderboard, or tooling)
  • Suite definition change
  • Schema change
  • Leaderboard / UI improvement
  • Documentation
  • Other:

Testing

# Commands used to verify

Checklist

  • I have read CONTRIBUTING.md
  • My change does not break existing result.json files (or I have explained the migration path)
  • If adding a new platform: runner inherits from BenchmarkRunner, produces valid result.json, includes a reference result
  • If changing the schema: validate_submission.py updated and all existing results still validate
  • If changing the leaderboard generator: leaderboard/generate.py produces correct output on existing results
  • I have updated relevant documentation

Related issues

Follow-up to the cleanup in #45. That PR removed the runner-declared
quantization-backend gating logic and renamed the obvious skip-reason in
the headline `print` (line 101), but two sibling references to the old
strategy were missed:

* The function-level docstring still claimed format selection
  intersects with `runner.SUPPORTED_QUANTIZATIONS` and warns on any
  format the runner doesn't declare.
* The per-format final-summary line printed
  `skipped (backend not in SUPPORTED_QUANTIZATION_BACKENDS)`
  even though the `skipped` list now only ever holds the *other*
  full-precision baseline (e.g. FP16 on Ampere where the hw baseline
  is BF16).

Rewrite both so the docstring describes today's policy (always include
the hw-supported full-precision baseline; dispatch every quantized
level; let the inference subprocess decide hardware compatibility) and
the skip-reason print matches what actually causes the entry.

The result.json field name `precision_levels_skipped` is **kept** — it's
a stable schema field already indexed by the leaderboard and used by
older results, so the name stays; only the human-readable strings
around it are corrected.

No functional change.

Co-authored-by: Cursor <cursoragent@cursor.com>
@JuhaoLiang1997 JuhaoLiang1997 merged commit bb99f38 into main May 15, 2026
@JuhaoLiang1997 JuhaoLiang1997 deleted the fix/suite-c-stale-skip-reason branch May 15, 2026 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant