feat(hub,skills): pre-publish security audit gate for marketplace skills (code + body, tier-gated verdict)

[kovtcharov-amd]

This issue adds an automatic safety check that every skill must pass before it can be published to the hub. A skill can contain both written instructions and runnable code that get loaded straight into an agent, so an unchecked skill is a genuine security risk — a competing platform was once flooded with over a thousand malicious skills. The check scans both the code and the text of a skill, returns a clear pass / needs-review / blocked result, and decides how trusted the skill is allowed to be.

Why this matters

Before: The skills marketplace (#2467) lets anyone publish a SKILL.md (± tools.py/scripts) that other users install and load straight into an agent's context and tool registry. Nothing inspects a skill for malicious code, permission over-reach, or prompt-injection before it reaches users. A competing hub was flooded with 1,184 malicious skills (see #647) — an open, unaudited publish path is the single biggest risk in the whole skills feature.

After: Every skill published to the hub passes an automated security audit gate that inspects both its code and its instruction body, emits a machine-readable ALLOW / REVIEW / BLOCK verdict, and gates publication + the security tier it may claim. Trusted-from-day-one is what differentiates the GAIA marketplace, and this is the mechanism that delivers it.

This also resolves the body-injection scanning open question already flagged in both docs/spec/agent-skills and docs/plans/skill-format.

Approach — reuse what exists

• Mirror the existing .github/workflows/claude-security-audit.yml pattern for the CI/LLM-assisted analysis stage.
• Emit findings as the governance GovernanceDecision shape (DecisionType = ALLOW | REVIEW | BLOCK, src/gaia/governance/schemas.py:17) so verdicts are consistent with the rest of GAIA.
• Hook the gate into the marketplace publish path (workers/agent-hub/src/publish.ts, Hub Worker: per-type publish + validation + R2 storage namespace #1718) so no skill enters the catalog un-audited, and expose the same engine as gaia skill audit ./skill/ so authors can self-check before publishing.

Scope

A. Audit engine (gaia skill audit)

Two analyzers over a skill directory, keyed to the two skill flavors:

• Code analysis (tool skills — tools.py, scripts/):
  • Dangerous-call detection: subprocess/shell, eval/exec, dynamic import, raw sockets/exfil, filesystem writes outside declared scope, environment/credential access.
  • Permission-truth check: does the code touch domains (network, shell, filesystem, env, …) it did not declare in metadata.gaia.permissions? A declared-vs-actual mismatch is a hard finding.
  • Supply-chain: flag undeclared or pinned-to-risky dependencies / node_dependencies.
• Instruction analysis (all skills — the Markdown body):
  • Prompt-injection / jailbreak pattern scan (the body is injected into model context and is untrusted input). Resolves the specs' body-injection open question.

B. Verdict + tier gating

• Output a GovernanceDecision-shaped report: ALLOW (publish proceeds), REVIEW (quarantined pending maintainer sign-off), BLOCK (publish rejected with an actionable findings report naming file/line and the fix).
• Tier-scaled rigor, per Skill marketplace: format spec, security tiers, and AMD Verified seeding #647 tiers:
  • experimental — scan is advisory (findings surfaced, install still requires --allow-experimental).
  • community — automated scan required to pass + publisher signature; dangerous findings force REVIEW.
  • verified — scan required plus the human/AMD audit hook (this issue provides the automated gate the human step builds on, not the audit policy itself).
• A skill cannot be stamped a tier whose gate it did not clear.

C. Publish workflow integration

• Gate runs at publish time in the hub Worker (feat(hub,skills): skills marketplace — publish/install skills as a first-class hub catalog lane #2467 / Hub Worker: per-type publish + validation + R2 storage namespace #1718): BLOCK → rejected, REVIEW → held out of the public lane, ALLOW → catalog entry written with the audit result + tier recorded.
• CI workflow (.github/workflows/) that runs the same audit on skill-submission PRs, mirroring claude-security-audit.yml.
• Re-audit on every version bump (as skill-format already requires) — a new version re-earns its verdict.

D. Reporting

• Structured findings artifact (severity, category, file:line, remediation) attached to the publish/PR.
• Security findings handled per repo policy (private channel for exploitable details, not a public dump) — consistent with the weekly-audit pattern.

Dependencies

• Feeds / gates feat(hub,skills): skills marketplace — publish/install skills as a first-class hub catalog lane #2467 (marketplace publish path) — this is the gate on that path.
• Builds on Skill marketplace: format spec, security tiers, and AMD Verified seeding #647 (security-tier design), claude-security-audit.yml (CI pattern), governance schemas, Code-sign & notarize all Agent Hub artifacts (installers + agent binaries) #1710 (signing).
• Resolves the body-injection open question in agent-skills / skill-format.
• Consumes the format/permission grammar from feat(skills): Phase 1 runtime — SKILL.md loader, validator, discovery + gaia skill CLI core #888 / skill-format (needs metadata.gaia.permissions to run the permission-truth check).

Non-goals

• Not the AMD-Verified audit policy/process (program/governance decision) — this delivers the automated gate + verdict that the human tier consumes.
• Not a general repository security audit (that is claude-security-audit.yml / claude-weekly-audit.yml); this is skill-artifact-scoped.
• No sandboxed runtime execution enforcement here — that is the Phase 2 permission sandbox; this is a pre-publish, static gate.

Acceptance criteria

• gaia skill audit ./skill/ runs code + instruction analyzers and prints an ALLOW/REVIEW/BLOCK verdict with file:line findings.
• A skill whose tools.py uses a domain not in its declared permissions is flagged (declared-vs-actual mismatch).
• A skill body containing known prompt-injection patterns is flagged.
• Publish is gated: BLOCK rejects, REVIEW quarantines, ALLOW writes the catalog entry with the recorded verdict + tier; a skill cannot claim a tier it did not clear.
• CI workflow runs the audit on skill-submission PRs; re-audit fires on version bump.
• Security findings follow the repo's private-disclosure policy; specs' body-injection open question marked resolved.

[github-actions]

This is in scope and well-formed — a pre-publish static gate is the right shape for the marketplace, and every anchor you cite is accurate, so the "reuse what exists" plan holds. The governance GovernanceDecision shape already gives you the exact ALLOW/REVIEW/BLOCK verdict type, the security-audit workflow is a clean CI template to mirror, and both specs genuinely still flag the body-injection question as open — so this really does close it.

One design flag worth deciding up front: the permission-truth check and dangerous-call detection should be deterministic (AST-based), not LLM-only. An LLM scan is the right tool for the fuzzy instruction-body / prompt-injection analysis, but "does this code touch network/shell/filesystem/env it didn't declare?" is a hard finding that a publisher shouldn't be able to talk the model out of. Pairing a deterministic pass (Python ast for tools.py/scripts/) for the declared-vs-actual check with the LLM pass for the injection scan makes the gate much harder to bypass and keeps BLOCK decisions defensible.

Tagging @kovtcharov-amd — the security-tier gating policy (which findings force REVIEW vs BLOCK, and how the verified human hook consumes this) is an architecture/security decision that should get maintainer sign-off before implementation starts.

🔍 Technical details

Verified against the tree:

• src/gaia/governance/schemas.py:17 — DecisionType = Literal["ALLOW", "REVIEW", "BLOCK"]; GovernanceDecision (decision/reason/policy_version/rule_ids/metadata) is a good carrier for the findings artifact via metadata.
• .github/workflows/claude-security-audit.yml exists and is the pattern to mirror for the PR-submission CI stage.
• workers/agent-hub/src/publish.ts exists — the publish hook point for the BLOCK/REVIEW/ALLOW gate.
• Body-injection open question is real in both specs: docs/spec/agent-skills.mdx:375 and docs/plans/skill-format.mdx:802 (advisory for experimental, required for community+ — matches your tier-scaling).
• No gaia skill subcommand exists yet, so gaia skill audit is net-new CLI surface (src/gaia/cli.py); the existing --skill references at cli.py:3176 are unrelated. hub/ currently has only agents/, so the skills/ catalog lane lands with feat(hub,skills): skills marketplace — publish/install skills as a first-class hub catalog lane #2467.

On the AST point: ast.walk over tools.py catching subprocess/os.system, eval/exec, __import__/importlib, socket, and open(..., 'w') outside declared scope gives you deterministic, reproducible hard findings for the permission-truth check; reserve the LLM stage for the instruction-body injection scan where fuzzy matching is actually required.

[kovtcharov]

Coordination from the #2467-A catalog lane (PR #2668), which owns the enforcement half of this gate in workers/agent-hub/src/audit.ts.

The three replay holes raised against assertAuditGate() are real — confirmed, not theoretical. The function reads report.verdict and nothing else; securityTier is used only as a presence predicate for whether a report is required, never compared against anything in the report. So today an ALLOW earned by an experimental audit publishes as verified, a 1.0.0 report publishes 1.1.0, and skill A's report publishes skill B.

Accepting the proposed binding fields verbatim — skill, version, security_tier, cleared_tiers, content_digest — with these checks in assertAuditGate():

Check	Failure
manifest security_tier ∈ report.cleared_tiers	403 audit_tier_not_cleared
report.skill == publishing name	400 audit_skill_mismatch
report.version == publishing version	428 audit_stale
report.content_digest == digest recomputed over uploaded bytes	400 audit_digest_mismatch

verdict / engine / audited_at / findings keep their current shape, so the existing parser is unaffected.

One correction worth pinning down: this closes replay, not forgery

content_digest was described as "the one that actually makes the gate non-bypassable." It does not, and the difference matters for what gets built next.

The report arrives as a publisher-supplied form part with no attestation — parseAuditReport() validates shape only, and there is no signature verification anywhere in the Worker. A hostile publisher therefore fabricates the whole JSON, computes a correct content_digest over their own files, sets cleared_tiers: ["experimental","community","verified"], and every one of the four checks passes.

So the four checks are worth doing — they close replay and accident (stale reports, forgotten re-audits, careless tier bumps), which is most of the real-world risk. But they are defense in depth, not a security boundary. Making a verdict unforgeable requires an attestation the publisher cannot mint: signing the report with a key held by CI (#1710), or having the Worker run the audit itself. Until that exists, neither this gate nor its docs should claim a skill's tier is trustworthy — only that it is consistent.

Suggest the recorded SkillAuditRecord eventually distinguishes "verdict asserted by the publisher" from "verdict attested by a trusted signer", so consumers can tell which they have.

Ownership

Implement the four checks in a stacked PR on top of #2668 rather than inside it. The digest is a byte-for-byte cross-language contract (Python engine ↔ TS Worker); one author should write both sides with shared test vectors, otherwise the TS half is written against an unwritten function and verified only by a mock — which proves the call happened, not that it interoperates. Please ship a fixture (a small skill tree → expected digest) that both suites assert against.

Sequencing constraint: the hardening must land before gaia skill publish (#2467 B) makes the gate load-bearing. Nothing can publish a skill today, so there is no live exposure — but the CLI must not ship ahead of these checks.

The gap is now documented at the top of workers/agent-hub/src/audit.ts in #2668 so it is not invisible to the next reader.

The REVIEW quarantine lane is split out to #2675 — the storage is trivial, but the maintainer sign-off workflow is the real design work and belongs with it.