Skip to content

No CI job runs or even compiles the benches #437

Description

@justin13888

The workspace has 15 bench files and 61 benches, and no CI job runs any of them. There is
a mise run bench task, but neither ci.yml nor extended.yml invokes it.

Two consequences:

  • The benches can rot silently. Nothing compiles them, so an API change breaks them and no gate
    notices until someone runs the task by hand.
  • They carry zero assertions, so even run they would pin nothing — yet gamut-cmm bench
    numbers were the cited evidence for filing gamut-cmm: fixed-point/integer + SIMD evaluation paths (6.6x lcms2 gap) #423. Evidence that no gate reproduces is evidence
    with a short shelf life.

Minimum fix is a compile-only check (cargo bench --no-run) in the fast lane, which is cheap and
catches the rot. Whether benches should additionally assert a regression threshold — and where
those numbers would come from on preemptible shared runners — is the harder question and the
reason this is its own issue rather than part of #435.

docs/testing.md explicitly places benches outside what it governs, so this is not blocked on the
testing corpus pass.

Refs #435

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions