-
Notifications
You must be signed in to change notification settings - Fork 0
Validation
Every new or changed skill must pass the official skills-ref validate command identified by the Agent Skills specification. The local helper supplements this with licenses, references, public hygiene, optional metadata, adapter checks, and artifact integrity. Neither structural layer replaces behavioral evaluation.
Use Node.js 24+ and run npm ci explicitly for the pinned CLI and type-checking dependencies. Subsequent local checks need no Python, network access or credentials. Standalone skill helpers still use built-in modules. Commands and official tool pins are centralized in package.json.
From the repository root:
npm run check
git diff --checknpm run check runs type checking, collection validation and disposable tests. Source lives in src/: singular command, service, repository, validator, migration and transport layers. The creator's standalone helper stays inside its distributable package and is reused by a repository.
Run against an owned checkout that remains stable throughout validation. Unsafe entries and detected changes are rejected; these checks are not a sandbox against concurrent adversarial mutation. Do not run candidate scripts or allow other writers to mutate the selected package while checking it.
The pinned official validator checks its implemented frontmatter and naming rules. It does not audit executable behavior, local resource completeness, icons, source rights, private data, or task performance. The collection's stricter checks remain required in addition to official conformance; neither check replaces a recorded behavioral evaluation on the actual candidate.
These checks maintain this source repository. They are not installed with a skill, and their commands must not be inferred in a consumer's project.
npm run validate discovers catalog packages, checks local links across repository documents, validates committed example runs named run.json, and checks upstreams.lock.json if present. Its public hygiene scan recognizes a small set of high-confidence credential, private-key, authenticated-URL, and local-user-path patterns without echoing matched values. It is a basic guard; binary content and less recognizable secrets still need review.
The byte scan also checks UTF-16LE and UTF-16BE text with or without a BOM.
It does not decrypt archives or establish that arbitrary data is safe to publish.
SVG checks require a valid namespace and inactive local geometry. PNG decoding
is capped at 16 MiB before inflation. Every package's assets/icon.render.json
binds its SVG and PNG hashes, raster dimensions and renderer invocation; changing
either image requires a reviewed rerender and receipt update. Validation checks
that recorded pair offline, not the artistic quality or honesty of a forged
receipt. The existing PNGs were rendered with rsvg-convert 2.63.2; the renderer
is a maintenance tool, not a dependency for validation or installed skills.
Material-based packages retain their exact pinned source path, SHA-256, reuse
decision and notice in assets/icon.source.json.
Duplicate-icon checks ignore SVG comments and compare PNG dimensions and decoded
RGBA16 pixels. Palette, grayscale, RGB, alpha and bit-depth representations are
normalized, including padding bits and fully transparent samples. Compression,
row filters, background hints (bKGD), text and pixel density (pHYs) cannot
make identical stored pixels distinct. This is not a renderer or a perceptual
similarity assessment; distinct rasterizations of similar artwork still require
visual review.
Color-managed PNGs are outside this bounded comparison profile. Validation
rejects cHRM, gAMA, iCCP, sRGB, cICP, mDCV and cLLI with an explicit
normalization diagnostic. These chunks can affect color interpretation or tone
mapping, as defined by the PNG color-space and mastering rules.
Rerender the reviewed SVG, or use a color-aware renderer to convert the image
to the collection's untagged sRGB pixel convention before exporting without
these chunks. Do not merely strip a profile from unchanged samples. Review the
result and refresh the paired render receipt. No ICC conversion or profile
decompression runs during validation.
Markdown character references and HTML URL whitespace are normalized before
scheme and path checks. Numeric references follow the HTML replacement rules:
for example, € resolves to €, while zero, surrogate and out-of-range
values resolve to U+FFFD. SVG remains subject to separate XML rules.
Package paths reject Windows device names, including the
console names CONIN$ and CONOUT$,
forbidden punctuation and trailing dots/spaces even on Unix hosts. Apache recognition
accepts the canonical terms plus only the recognized optional appendix or
application notice; extra clauses require review instead of silently passing.
This structural check does not establish legal rights.
The visual guide uses a pinned variable font with its full OFL notice. Its
font source receipt identifies the exact
upstream font and license bytes. The publication verifier requires every HTML
guide under docs/assets/ to appear in visual-guides.lock.json; only the
navigation page index.html is exempt from guide receipts.
Review dates must be real calendar dates in canonical YYYY-MM-DD form.
Canonical repository packages live in .agents/skills/. Collection validation permits only these exact aliases: CLAUDE.md to AGENTS.md, .claude/skills to ../.agents/skills, and .github/skills to ../.agents/skills. It verifies their targets, does not traverse them during inventory, and counts the canonical packages once. This collection-only exception never permits symlinks inside a package or run.
Root .work/, tmp/, .beads/, and .codex/environments/ are local operational state and are never read or traversed by collection validation. Other .codex/ files remain in the checked publication corpus. When Git metadata is available, a bounded read-only index check rejects tracked scratch without returning its names or contents. Git is required for that check; exported trees cannot establish what the publication index contains. Nested directories with those names remain part of the checked publication corpus. Root Git metadata is excluded. Cache-like names do not create additional exemptions; keep local environments and generated scratch under the designated root scratch directories.
Lock verification is offline. It checks source identity, immutable revisions, safe file paths, known consumers, digest formatting, and the aggregate hash described in source research. It cannot verify that upstream bytes, licensing, ownership, or adoption claims match the recorded source. The lock is an audit input for future evolution, not an installed runtime dependency.
The shared provenance-host policy rejects non-global literal IP ranges, including documentation and reserved ranges, using the IANA special-purpose registries recorded in the helper. It performs no DNS or network requests; accepting a hostname is not a reachability check or an SSRF protection boundary.
The distribution regressions copy only package directories into disposable layouts outside this checkout. They exercise the creator helper from an unrelated working directory, using read-only installed resources, separate output/run directories, and installer-style directory aliases. They also validate every package's copied resources without this repository's catalog, root instructions, src/, or package configuration. No consumer home or actual installation is modified.
This is a reproducible package-relocation check, not an invocation of npx skills or proof of model behavior in every host. Official validation remains required separately.
Python is supplied only by the GitHub workflow because the official upstream tool is implemented in Python. CI sets up Python 3.12, creates ignored .work/validation-env, and runs npm run ci:official. This Node use case checks the collection, installs the pinned external tool in that isolated environment, verifies its version, and invokes skills-ref validate for every canonical package and a disposable package generated by the current authoring scaffold. The generated trial is removed after validation; its rejection fails the same check. Local contributors do not prepare a Python environment.
Accept official evidence only for the exact PR head and package bytes. A standalone skill can return a handoff to a trusted workflow or another authorized environment with the official tool; it need not force Python into the user's local environment. If no matching official result exists, preserve the package but leave official validation pending and readiness blocked. Custom-parser or manual passes do not substitute.
The validator is built from agentskills/agentskills at 69ef37e9424c0a7ea9dd2293b559e43ec8176379, whose package declares version 0.1.0 and Python >=3.11. The exact source archive has SHA-256 0c9eabbe602095c4f4d771ee55bf74f6bc7e1c770f25d4fe29ce9802981daa20. Its Apache-2.0 license and source stay in the isolated tool installation; the validator is not vendored into a skill.
package.json records the official archive, version, and full runtime/build dependency hashes under config.officialSkillValidator. Pure policy checks immutable source identity, exact versions, unique dependency names, and hashes. The process adapter derives two temporary installation inputs and removes them afterward. It verifies an isolated Python 3.11+ environment, installs only hashed wheels, then builds the official source with dependency resolution and build isolation disabled. A failure stops later phases. No committed requirements file or duplicate version declaration is maintained in the implementation. This configuration is separate from the benchmark source lock.
The similarly named PyPI skills-ref distribution did not match the official source/executable identity during inspection, so this setup uses the official repository archive. A source pin and hashes establish identity, not a complete security certification. Update pins only after reviewing the source and repeating the checks.
The GitHub workflow runs on every pull request and every push to main, without a path filter. It validates the exact PR head checkout and every canonical package, which includes all added or modified skills and catches cross-package regressions. The catalog check rejects an omitted or unexpected package. Aliases are not discovered as duplicates, and private scratch is not part of the publication corpus.
The Node process adapter passes package paths as argument arrays to the environment's exact skills-ref executable, never through a shell. It applies a per-package execution bound, reports all results, and fails if the tool is missing, its version is wrong, or any package fails. It coordinates the actual official validator rather than reimplementing it. The stricter repository check runs first on the same stable checkout.
CI uses pinned actions, a read-only token, no persisted checkout credentials, no secrets, and a job timeout. It checks whitespace across the actual base/head diff. GitHub branch protection is not changed by this PR; maintainers can separately make the workflow required.