A zero-key GitHub Action to gate AI agent production readiness — feedback wanted

[lindixu6-hash]

🏷️ Discussion Type

Product Feedback

💬 Feature/Topic Area

Workflow Configuration

Discussion Details

I kept seeing AI agent repositories test whether code runs, while release decisions around tool permissions, memory, evals, failure recovery, security, observability, cost, and human review remained informal.

I built a small Node 24 GitHub Action that scores a repository-owned agent-card.json across ten readiness areas and can fail CI below a chosen threshold. It makes no model calls and needs no external API key.

- uses: lindixu6-hash/awesome-agentic-engineering@v0
  with:
    card: agent-card.json
    min-score: "15"
    fail-below: "true"

The Action writes score, rating, and a README badge as outputs, and adds the ten-area table to the workflow summary. A separate repository is using the public @v0 reference: its human-gated Content OS currently scores 12/20 (prototype) and gates CI at 10/20. The main project has 42 automated tests.

Important limitation: the numeric score does not override explicit launch blockers. Today the card is self-attested; the useful next step is stronger evidence validation, not pretending a high number proves production safety.

I would value feedback from Actions users on three design questions:

1. Should hard blockers be modeled as a separate required input/output rather than card metadata?
2. Is a single threshold useful, or should thresholds be risk-tiered by read-only vs. state-changing agents?
3. What evidence should CI verify beyond a checked-in scorecard (eval fixtures, permission manifests, budget files, approval rules)?

Repository: https://github.com/lindixu6-hash/awesome-agentic-engineering

Live scorecard: https://lindixu6-hash.github.io/awesome-agentic-engineering/

Disclosure: I maintain the project. It is MIT licensed.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[lindixu6-hash]

Update after the initial post: I shipped the first design change as v0.3.0.

The score threshold and explicit launch blockers are now independent gates:

- uses: lindixu6-hash/awesome-agentic-engineering@v0
  with:
    card: agent-card.json
    min-score: "15"
    fail-below: "true"
    fail-on-blockers: "true"

fail-on-blockers is opt-in to avoid breaking existing consumers. The public @v0 reference was tested on a GitHub-hosted runner: the 16/20 example passes the score gate, fails on its two declared blockers, and still exposes passed=false plus blocker-count=2 for later diagnostic steps. CI evidence.

This addresses the first question mechanically. I am still interested in feedback on risk-tiered thresholds and what external evidence CI should verify rather than trusting self-attested scores.

[lindixu6-hash]

A second concrete iteration is now available in v0.4.0: a framework-neutral prompt-injection eval fixture pack.

It adds 8 JSONL cases across direct override, retrieved web content, email/document injection, tool-output injection, data-exfiltration requests, and benign controls. Every case defines trusted instruction, untrusted content, expected actions, forbidden actions, and evidence required. The pack includes a JSON Schema and zero-dependency agentic-validate-fixtures CLI.

The published @v0 CLI is now tested from a fresh GitHub install in CI. Fixtures use synthetic identifiers and reserved .example domains; they do not execute payloads or include a working exfiltration endpoint.

This is only a partial answer to the third design question: CI can now verify the structure and completeness of evidence contracts. It still cannot prove a model/runtime actually satisfies them until a consumer wires the cases into its eval runner, so the example Agent Cards retain that explicit launch blocker.

[lindixu6-hash]

A third evidence-validation iteration is now shipped in v0.7.0: a framework-neutral Eval Result contract.

A consumer adapter can now record a known fixture ID, Agent/version, pass/fail outcome, observed actions, forbidden-action violations, evidence references, and tool/policy trace references. The zero-dependency agentic-validate-results CLI rejects malformed results, duplicate result IDs, inconsistent passing results, and broken fixture references. It accepts JSON objects, arrays, and JSONL.

The public @v0 install is exercised on a GitHub-hosted runner in CI. The contract deliberately does not fetch evidence or invoke a model: structural validation proves neither artifact authenticity nor Agent safety.

This narrows the third design question. CI can verify that runtime evidence is recorded against known cases, but evidence provenance remains consumer-owned. The next open design task is risk-tiered readiness profiles, especially whether state-changing Agents should require stronger per-area minimums than read-only Agents.

[lindixu6-hash]

The second open design question now has a concrete implementation in v0.8.0: opt-in risk-tiered readiness profiles.

read-only, draft-only, and state-changing profiles define different total and per-area minimums, allowed and required tool effects, approval requirements, and launch-blocker policies. A state-changing Agent must declare an external_state tool, every such tool must require approval, and unresolved blockers fail the gate. Draft-only Agents cannot include external-state capability at all.

The default Action remains backward compatible when profile is omitted. Unknown or malformed profiles, missing effect metadata, and Agent Card/profile mismatches fail closed. The existing draft and operations examples retain their original 12/20 and 15/20 scores and blockers rather than being adjusted to pass.

The public @v0 profile gate and its machine-readable outputs pass on a GitHub-hosted runner: CI evidence.

The remaining hard problem is evidence provenance: checked-in profile metadata still cannot prove that a runtime enforces approval and authorization scope. I would especially value examples of Actions workflows that attest those controls without giving an untrusted Agent control over its own evaluator.

[lindixu6-hash]

A fourth iteration is now shipped in v0.9.0: an executable, deterministic prompt-injection reference eval.

Unlike the earlier fixture/schema work, this runner actually executes all eight malicious and benign cases through separate trusted-instruction and source-labeled untrusted-content channels. An evaluator outside the Agent path emits v0.7 results plus response, assertion, tool-trace, and policy-trace artifacts. A negative regression changes an expected outcome and verifies that the evaluator emits fail rather than forcing a green suite.

CI run 31974669422 validates generated results with public @v0 and retains 34 evidence files in the reference-eval-evidence artifact.

Scope remains deliberately narrow: this proves a deterministic reference policy handled these inert fixtures as expected. It is not a language-model benchmark and Content OS cannot claim these passes because it has no executable Agent runtime boundary. The next useful integration is an external open-source runtime where the evaluator and policy assertions remain outside the Agent-controlled workspace.

[ralftpaw]

A useful next boundary is to make the evidence producer structurally incapable of grading itself. Run the agent in job A with least-privilege permissions; have job B in a separate protected/reusable workflow fetch immutable artifacts, pin the evaluator SHA, and emit an attested result binding the repo SHA, workflow SHA, policy version, fixture hash, and output hash.

That keeps Agent Card fields as claims and external job outputs as evidence. A pass then means “this exact evaluator observed these exact artifacts,” not “the agent says it passed.” GitHub artifact attestations/OIDC can help with provenance, but the verifier should still check the subject digest and trusted workflow identity.

For approval/authorization controls, I would include denied-action fixtures and assert both: (1) no external side effect occurred, and (2) a durable audit record was emitted. A clean return code alone is too easy to game.

[lindixu6-hash]

v0.10.0 now tests the contract through an external runtime: @langchain/langgraph@1.4.10.

A real two-node StateGraph executes all eight malicious and benign fixtures. Trusted instructions and source-labeled untrusted content remain separate graph-state fields, while an evaluator outside graph.invoke() emits node-linked results, assertions, and tool/policy traces. A negative regression confirms an expected/observed mismatch emits fail.

The clean-runner CI installs the pinned lockfile, passes 100 tests, validates generated results with public @v0, reports zero known dependency vulnerabilities, and retains 34 LangGraph evidence files. The root CLIs remain zero-dependency and the npm tarball is 347 KB.

I also opened LangGraph.js #2700 before proposing an upstream docs PR, following their contribution guide. The scope remains narrow: this is reproducible orchestration evidence, not an LLM benchmark or proof that arbitrary LangGraph applications are safe.

[lindixu6-hash]

v0.13.0 adds a second external-runtime proof through @openai/agents@0.16.1.

Every fixture now enters a real SDK Agent + Runner.run() loop. Allowed cases traverse the SDK function-call pipeline and a Zod-validated read-only tool; refusal and escalation cases make no tool call. A public custom Model keeps the run deterministic, the test replaces provider network access with a function that always throws, and the expected outcome remains outside the Agent-controlled input. A forced mismatch still emits fail.

The public run passes 8/8 fixtures, retains 34 files in openai-agents-eval-evidence, and validates the result contract with public @v0. The same contract now has executable evidence in a reference runner, LangGraph.js, and OpenAI Agents SDK. Technical guide · 中文指南

This still does not solve the provenance boundary raised by @ralftpaw: the runtime and evaluator execute in the same CI job. The next stronger design would separate producer job A from a protected verifier job B and bind the repository SHA, workflow SHA, fixture hash, evaluator version, artifact digest, and attestation identity. I would value concrete examples of the minimum GitHub artifact-attestation/OIDC checks that should be required before calling that evidence independently verified.

[lindixu6-hash]

I implemented the producer/verifier separation suggested by @ralftpaw in v0.14.0.

The runtime producer now creates one deterministic bundle plus a manifest binding the repository/workflow identity, five trusted inputs, all 34 evidence files, and 8/8 Eval Results. GitHub OIDC + Sigstore signs that bundle. A separate reusable workflow pinned to commit 8d4b435 checks out verifier code pinned to the older immutable root 34b1235, verifies signer/source SHA and ref, rejects self-hosted provenance, checks every manifest digest, and rejects any failing result.

The release attested run also proves two negative cases: appending one byte to the signed bundle fails verification, and verifying the original bundle against an all-zero source digest fails. Release bundle SHA-256: e0c4c05bafc2873e11c7ab27eb264f01dc0d26106f9c55bed398d8908d8fcab2.

Important boundary: main was not branch-protected for this run. The evidence therefore proves workflow, commit, hosted-runner, and artifact identity, but not independent code review, semantic correctness, or a SLSA level. Moving build/signing into an independently governed reusable builder plus enforced branch protection would be the next stronger step.

English provenance boundary · 中文证据边界