Verifiable Evaluation Attestations for repositories and pull requests

[zedarvates]

🏷️ Discussion Type

Product Feedback

💬 Feature/Topic Area

Pages

Body

Problem

GitHub Actions can run arbitrary evaluation pipelines, and GitHub Artifact Attestations can establish the provenance of generated artifacts. However, repositories lack a standard way to publish, verify, display, and enforce third-party evaluation results.

This is particularly important for AI-assisted development, where teams may use local models, specialized evaluators, agent benchmarks, deterministic test suites, or regulated-domain checks that are not part of GitHub Code Quality.

Today, each project must invent its own report format, badge, storage convention, verification process, and pull-request integration.

Proposed capability

Introduce Verifiable Evaluation Attestations: an open, provider-neutral mechanism for publishing evaluation results associated with a commit, pull request, release, model, or software artifact.

An evaluation attestation could contain:

evaluator identity and version;

evaluated commit and scope;

benchmark or dataset digest;

model and execution configuration where applicable;

deterministic seed and environment metadata;

structured metrics and evidence;

uncertainty and limitations;

one of the standard outcomes: passed, failed, inconclusive, or unsupported;

links to human-readable and machine-readable reports.

The result would be published as a GitHub Artifact Attestation using a documented custom predicate type.

Native GitHub integration

GitHub could:

display verified evaluations on pull requests, commits, releases, and the Security & quality tab;

distinguish GitHub-provided, organization-trusted, and self-reported evaluators;

expose evaluation results through REST and GraphQL APIs;

allow repository rulesets to require a named evaluator, predicate version, freshness period, or explicit outcome;

generate non-misleading status badges linked to the underlying attestation and evidence;

support evaluations produced by GitHub-hosted Actions, self-hosted runners, local tools, or authorized GitHub Apps.

Trust model

A cryptographic signature proves provenance and integrity, not the correctness of an evaluator or the universal quality of a repository.

The UI should therefore show:

who produced the evaluation;

which policy trusts that evaluator;

what was evaluated;

which benchmark and evaluator versions were used;

whether the result is reproducible;

whether the result is conclusive.

This should be presented as a verifiable evaluation claim, not as universal “GitHub certification.”

Example use cases

reproducible LLM or agent regression benchmarks;

checks performed by local or private models;

evaluation of generated code against organization-specific requirements;

accessibility, safety, performance, governance, or regulated-domain evaluations;

signed evidence packages for releases;

comparison of evaluation results across commits without requiring a single AI provider.

Why GitHub

GitHub already has the core building blocks: Actions, rulesets, Artifact Attestations, APIs, Code Quality, pull-request checks, and repository-level quality views.

A standard evaluation-attestation layer would connect these capabilities while allowing the evaluator ecosystem to remain open and provider-neutral.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[Vendetaaaa]

As a suggestion about this amazing idea:

Storage & Size Limits: Since evaluation evidence packages sometimes can become large very fast, it would be worth considering how GitHub Artifact Attestations handles payload.

​Ruleset Enforcement: Being able to gate pull requests via repository rulesets

[ralftpaw]

One trust-model edge worth making explicit is replay after the evaluator or repository policy changes. I would bind the subject digest, evaluator workflow identity, benchmark digest, policy/config digest, validity window, and a supersedes/revokes pointer into the predicate. Otherwise an old but correctly signed pass can outlive the authority boundary that made it meaningful.

For agent evaluations, I would also bind the declared tool/side-effect boundary. A rule could then require a trusted issuer + predicate version + freshness + matching subject/policy digests, while keeping the UI wording narrow: “this evaluator observed this outcome for these artifacts,” not universal certification.

The typed passed / failed / inconclusive / unsupported outcomes are especially good. They keep missing evidence from quietly becoming green.