[scaffolds] Serve scaffolds from another repo via per-repo sources + an upstream index

[Adi-ty]

Why we need this

A scaffold whose templates are owned by another repo (e.g. the ci/* callers that wrap rtCamp/wp-shared-workflows) should be maintainable from a single PR in that repo — adding a workflow, changing an input, fixing a template should not require a coordinated edit in wp-tooling. This adds a per-repo source list plus an upstream index: wp-tooling lists only the repos that host scaffolds, and each owning repo publishes an index of the scaffolds it offers. Adding, changing, or registering a scaffold is then a single upstream PR; wp-tooling changes only when a new repo is onboarded.

---

Before you start

You'll need

• Write access to rtCamp/wp-tooling
• Node 22 LTS (node -v), npm 10 (npm -v)
• Network access for the live fetch smoke (reaches raw.githubusercontent.com)

Worth reading first

• CLAUDE.md → "Architecture patterns", "Non-negotiables" (zero runtime deps; non-TTY fallback)
• docs/authoring-scaffolds.md → "Remote scaffolds via per-repo sources + an upstream index"
• docs/ai-orchestration.md → EFETCHFAIL section + list --json origin/counts/warnings
• src/scaffolds/registry.js, src/scaffolds/sources.js, src/scaffolds/fetch.js

---

What you're building

In Scope

• A per-repo source list (scaffolds/sources.json) mapping nothing more than { github, ref, path } per source repo.
• An upstream index (<path>/index.json in each owning repo) enumerating the scaffolds it offers (id, path, name, description, optional checksum).
• Registry discovery that fetches each source's index, builds thin remote records (origin: "remote"), with lazy hydration on add (fetch + validate the remote scaffold.json, then fetch its templates).
• A zero-dependency remote-file fetcher with ETag-validated on-disk caching (If-None-Match → 304 serves cache; movable tags refresh when they move; SHA pins never refetch) and a cache clear CLI.
• list support for remote entries — online-preferred with a cache fallback; unreachable sources surface as warnings, never a hard failure.
• validate parity: default offline (sources-shape + remote ids from a cached index); opt-in validate --remote fetches each index + manifest and schema-validates them.
• New error code EFETCHFAIL (network/HTTP) distinct from EBADSCAFFOLD (bad index/manifest).
• Docs + the scaffold skill updated.
• Engine + fixture proof only — deterministic mocked integration tests plus a live fetch-substrate smoke.

Out of scope

• Migrating the real ci/* scaffolds — they stay local and untouched until wp-shared-workflows hosts + indexes + tags them (a per-scaffold follow-up).
• Shipping a default scaffolds/sources.json — the feature stays dormant until the first repo is onboarded.
• setup/*, lint/*, wp/* ever going remote — they have no upstream owner.

---

How to build it

File Layout

src/scaffolds/
  sources.js       NEW  — read/validate sources.json + index.json; entry -> thin record
  fetch.js         NEW  — fetchRemoteFile(repo, relPath, opts); ETag cache; readCached
  cache.js         NEW  — cache dir resolution + clear
  errors.js        NEW  — ScaffoldError (extracted to avoid a fetch<->registry cycle)
  registry.js      MOD  — scan() discovers via sources -> index; execute() hydrates remote first
  schema.js        MOD  — ALLOWED_SOURCES is ['template','package']; GITHUB_REPO_PATTERN reused
  validate.js      MOD  — validate sources.json shape offline; --remote fetches index + manifests
  list.js          MOD  — remote rows: kind 'template', origin 'remote', counts null; warnings
  add.js           MOD  — --refresh / --cache-dir; thread fetchOpts into scan()
src/cli/commands/
  cache.js         NEW  — `wp-tooling cache clear`
tests/scaffolds/
  sources.test.js / fetch.test.js / cache.test.js   NEW
  registry.test.js / validate.test.js               MOD (remote blocks)
tests/fixtures/scaffolds-sources/sources.json       NEW
docs/authoring-scaffolds.md, docs/ai-orchestration.md MOD
skills/scaffold/SKILL.md (+ synced claude-skills template)  MOD

The two file shapes

scaffolds/sources.json (wp-tooling side; also honoured at <proj>/bin/scaffolds/sources.json):

{
  "sources": [
    { "github": "rtCamp/wp-shared-workflows", "ref": "v1", "path": "scaffolds" }
  ]
}

ref is a literal pin the team controls (tag or SHA). path is the dir in the owning repo holding the index + scaffolds.

<path>/index.json (owning-repo side, fetched):

{
  "scaffolds": [
    {
      "id": "ci/test-php",
      "path": "ci/test-php",
      "name": "CI: PHPUnit",
      "description": "GitHub Actions workflow calling wp-shared-workflows ci-test-php.yml.",
      "checksum": "sha256:…"
    }
  ]
}

name/description live upstream so offline list has no local copy to drift; path is the scaffold dir relative to the source path; checksum is optional. The index is the remote substitute for the local recursive scaffold.json walk (impossible over raw.githubusercontent).

No change to scaffold.json or the folder layout. A remote repo's scaffolds/ folder is structurally identical to wp-tooling's own — the same <category>/<slug>/scaffold.json + templates, the same category/slug identity. scaffold.json is unchanged (no id, no remoteness field); the id in index.json is just the existing category/slug, listed only because a remote directory can't be readdir-walked. Local and remote scaffolds run through the same code path.

Implementation

Discovery. scan() walks local scaffolds, then for each sources.json entry fetches <github>/<ref>/<path>/index.json (cached, offline-tolerant) and builds a thin record per listed scaffold: { slug, category, name, description, origin:'remote', _repository:{github,ref,path:<source.path>/<entry.path>}, _checksum }. A scaffold id must be unique across local scaffolds and every source — a collision is a hard EBADSCAFFOLD at scan. An index that is unreachable + uncached is skipped with a warning (local list/add keep working); a reachable but invalid index is EBADSCAFFOLD.

Hydration. execute() hydrates a remote record before anything else (fetch + JSON.parse + validate() the scaffold.json, memoised), then prefetches template bodies for missing dests (offline-safe re-run; --dry-run fetches the manifest only, no bodies). The hydrated record is shape-identical to a local one — source stays template/package.

Fetcher (fetch.js, node built-ins only). fetchRemoteFile(repo, relPath, opts) composes the raw URL, caches the body + an ETag sidecar under ${XDG_CACHE_HOME:-$HOME/.cache}/wp-tooling/remote/ keyed by sha256(url), and validates with If-None-Match: a 304 serves the cache, a 200 updates it; a network failure with a cached copy serves cache + warns, otherwise throws. --refresh skips the conditional. Token from opts.token or WP_TOOLING_GITHUB_TOKEN; 403 + "rate limit" → rateLimited: true. readCached() reads cache only (offline id recognition).

Errors. EFETCHFAIL = network/HTTP. EBADSCAFFOLD = invalid index/manifest (bad JSON or schema). ScaffoldError attaches payload fields (url, statusCode, rateLimited, …) at the top level.

Conventions to follow

• Zero runtime dependencies (node built-ins only). Every command works in non-TTY/CI.
• JSDoc with @param/@returns/@throws on public functions; async/await, no .then() chains.
• Validators return string[] of field-path-prefixed messages.
• British English in prose; camelCase identifiers; kebab-case filenames.

Things to avoid

• Do not add a source: "repository" (or any remoteness field) to the manifest schema.
• Do not auto-run composer require / npm install / gh secret set — surface as developer actions.
• Do not ship a default scaffolds/sources.json — keep the feature dormant.

---

How to verify your work

$ cd node-packages/wp-tooling
$ npm run check            # ESLint + full Jest suite (lint clean, all suites pass)

Quick smoke

# 1. Backwards-compat: dormant default, local-only
$ node bin/wp-tooling.js list --json | grep -c '"origin":"remote"'    # -> 0

# 2. A project with a source: list reads the repo index (cached), or warns if unreachable
$ PROJ="$(mktemp -d)"; mkdir -p "$PROJ/bin/scaffolds"
$ cat > "$PROJ/bin/scaffolds/sources.json" <<'JSON'
{ "sources": [ { "github": "rtCamp/wp-shared-workflows", "ref": "v1", "path": "scaffolds" } ] }
JSON
$ node bin/wp-tooling.js list --cwd "$PROJ" --cache-dir "$PROJ/.cache"

# 3. cache clear
$ node bin/wp-tooling.js cache clear --cache-dir "$PROJ/.cache"

Runtime behavior

• sources.json + a reachable index.json surface scaffolds in list with origin:"remote", kind:"template", counts:null; an unreachable + uncached source is skipped with a warnings entry, not a failure.
• add <remote-id> hydrates (fetch + validate scaffold.json), then fetches + renders templates, writing files identically to a local scaffold.
• A remote id colliding with a local scaffold, or offered by two sources, throws EBADSCAFFOLD at scan.
• A missing/invalid index or manifest → EBADSCAFFOLD; a network/404/timeout on a manifest/template → EFETCHFAIL carrying url + statusCode (+ rateLimited on a throttled 403).
• The cache validates with ETag/If-None-Match: 304 serves the cache, 200 updates it; --refresh re-fetches; cache clear wipes the dir.
• --dry-run fetches the manifest but no template bodies and writes nothing; re-running add on an existing dest does no template I/O.
• validate default is offline (sources shape + remote ids from a cached index); validate --remote fetches index + manifests and schema-validates; collisions and EFETCHFAIL/schema errors report as rows.
• Backwards-compat: with no sources.json, local scaffolds behave exactly as before.

Code quality

• npm run check green: ESLint clean, full Jest suite passes.
• No runtime dependencies added; fetch.js uses node built-ins only.
• Mocked integration tests cover: index discovery, collisions (local + cross-source), hydrate+write, input render, ETag/304 caching across fresh registries, malformed/schema-invalid index + manifest → EBADSCAFFOLD, 404 → EFETCHFAIL, dry-run, parallel prefetch, offline-safe re-run, unreachable-index warning.
• Public functions carry JSDoc with @param/@returns/@throws.

Housekeeping

• CHANGELOG.md entry under ## Unreleased describing the sources/index feature, EFETCHFAIL, ETag caching, and cache clear.
• docs/authoring-scaffolds.md + docs/ai-orchestration.md + skills/scaffold/SKILL.md (and the synced scaffolds/setup/claude-skills/templates/scaffold-SKILL.md) updated and kept byte-identical.

---

Reviewer checklist

• No source: "repository" or other remoteness field in the manifest schema.
• fetch.js imports only node built-ins; no banned deps anywhere.
• Cache is keyed by sha256(url), written atomically, and validated by ETag (If-None-Match/304).
• EFETCHFAIL vs EBADSCAFFOLD boundary is correct (network vs bad index/manifest); a failed discovery index fetch with no cache is a warning, not a throw.
• list reads the cached index (online-preferred, cache fallback); --dry-run fetches no template bodies.
• Live smoke passes against raw.githubusercontent.com (ETag captured, 304 serves cache, --refresh, cache clear).

---

Submitting your work

Base your branch on	release/v1.0.0
Branch name	v1.0.0/task/remote-scaffold-inventory
PR target	release/v1.0.0
PR title	[v1.0.0] Serve scaffolds from another repo via per-repo sources + an index

PR description starts with Closes #29.

---

What this unblocks

• Per-scaffold migration of ci/* into rtCamp/wp-shared-workflows (move scaffold.json + templates, add it to that repo's index.json, tag, add the repo to sources.json here, delete the local copy) — one scaffold at a time, lockstep with an upstream tag.