[repository-quality] 🎯 Repository Quality Improvement Report - Skill Documentation Drift (Stale JavaScript Architecture References)

[github-actions[bot]]

Analysis Date: 2026-08-31
Focus Area: Skill Documentation Drift — Stale JavaScript Architecture References
Strategy Type: Custom
Custom Area: Yes — the repo's own .github/skills/*/SKILL.md files are load-bearing operational documentation consumed directly by Copilot agents (per AGENTS.md's lazy-skill-loading policy). A scripted cross-reference audit of all 52 skill directories found that two of the highest-traffic skills (javascript-refactoring, messages) still document a JavaScript embedding/bundling architecture (pkg/workflow/js.go with real (go/redacted):embed directives, pkg/workflow/scripts.go, pkg/workflow/bundler.go, GetJavaScriptSources() returning real script content) that has been fully replaced by a custom-actions architecture (actions/setup/js/*.cjs, 827 files) where js.go's embedded-script functions now return empty strings/placeholders by design. Agents following these skills verbatim would edit files that no longer do anything, silently producing broken workflows.

Executive Summary

A repo-wide scan of every .github/skills/*/SKILL.md for referenced file paths turned up 89 candidate broken references. After filtering false positives (code identifiers mistaken for filenames, e.g. helpers.go/utils.go used as anti-pattern examples, or generated docs like github-mcp-tools-report.md), two skills contain genuinely obsolete, actionable architecture documentation: javascript-refactoring/SKILL.md and messages/SKILL.md. Both instruct contributors to add (go/redacted):embed directives and script-source variables to pkg/workflow/js.go and a nonexistent pkg/workflow/scripts.go/bundler.go. In the current codebase, pkg/workflow/js.go is a 30-line stub whose GetJavaScriptSources() returns an empty map and whose getter functions return "" — all real script embedding and bundling now happens per-action under actions/setup/js/*.cjs (827 files, described in actions/README.md's "Generating Actions from JavaScript Modules" section). A handful of smaller stale references were also found in developer-internals/SKILL.md (validation_strict_mode.go, strict_mode.go, expression_safety.go — files that were split/renamed into strict_mode_*.go and expression_safety_validation.go/expression_safety_test.go) and error-pattern-safety/SKILL.md (test file names that no longer exist verbatim).

The risk is concrete and near-term: any Copilot agent invoking the javascript-refactoring or messages skill today to add a new safe-output message type would follow Step 3/Step 8 instructions to add embed directives to js.go, which has zero effect on the compiled binary, and the change would silently fail to ship the new script. Recommended immediate action is to rewrite these two skills against the current actions/-based architecture (mirroring actions/README.md's already-accurate "Creating a New Action" walkthrough), followed by fixing the smaller path staleness in developer-internals and error-pattern-safety, and adding a lightweight CI check that greps skill files for path-like backtick references and fails if the referenced file doesn't exist anywhere in the repo.

Full Analysis Report

Focus Area: Skill Documentation Drift — Stale JavaScript Architecture References

Current State Assessment

The repository ships 52 skills under .github/skills/ that are lazily loaded by agents per AGENTS.md. These are treated as authoritative, load-bearing instructions — not advisory docs — so factual drift directly causes incorrect agent behavior. A custom audit script extracted every backtick-quoted path ending in .go, .md, .yml, .yaml, or .json from each SKILL.md and checked existence relative to repo root and the skill's own directory.

Metrics Collected:

Metric	Value	Status
Total skills scanned	52	✅
Raw candidate broken references	89	⚠️
Skills with genuinely obsolete architecture docs	2 (javascript-refactoring, messages)	❌
Skills with minor stale filenames (renamed/split files)	2 (developer-internals, error-pattern-safety)	⚠️
pkg/workflow/js.go real vs. stub	Stub only (GetJavaScriptSources() returns map[string]string{})	❌
Current script architecture location	actions/setup/js/*.cjs (827 files)	✅ (correct, just undocumented in the two skills above)
actions/README.md accuracy vs. actual dir listing	Documents 6 sub-actions (setup, setup-cli, noop, minimize_comment, close_issue, close_pull_request, close_discussion); only setup/ and setup-cli/ exist as top-level dirs — the others are files/handlers within actions/setup/js/, not separate action directories	⚠️ mild wording ambiguity, not a hard break

Findings

Strengths

• actions/README.md accurately documents the current custom-actions architecture, including a correct "Generating Actions from JavaScript Modules" and "Creating a New Action" workflow — this is the right source of truth to port into the stale skills.
• Most skills (48/52) had zero broken references — the lazy-loading skill system is generally well-maintained.
• The obsolete-reference pattern is narrowly confined to two skills that both concern the same subsystem (JavaScript script management), suggesting a single coordinated architecture migration was not fully reflected back into skill docs.

Areas for Improvement

• [High] javascript-refactoring/SKILL.md (485 lines) instructs adding (go/redacted):embed directives to pkg/workflow/js.go and creating/editing pkg/workflow/scripts.go and pkg/workflow/bundler.go — none of which reflect the real embed/bundle path (actions/setup/js/*.cjs, bundled per actions/README.md). Following this skill literally produces a no-op change.
• [High] messages/SKILL.md (312 lines) Step 8 ("Register in Go Embeddings") and its File Summary table both point at pkg/workflow/safe_outputs.go and js.go for adding new message JS modules — same stale-architecture issue as above, in the skill most likely to be invoked when adding a new safe-output type.
• [Medium] developer-internals/SKILL.md references pkg/workflow/strict_mode.go, pkg/workflow/validation_strict_mode.go, and pkg/workflow/expression_safety.go as single files; the codebase has since split strict-mode logic into strict_mode_validation.go, strict_mode_steps_validation.go, strict_mode_network_validation.go, etc., and expression safety into expression_safety_validation.go / expression_safety_test.go.
• [Medium] error-pattern-safety/SKILL.md cites pkg/workflow/error_pattern_tuning_test.go and pkg/workflow/engine_error_patterns_infinite_loop_test.go as exact file paths in a bulleted "test files to run" list; neither exists under those exact names in the current tree, so a copy-pasted test command from this skill will fail with "no such file".
• [Low] No automated check exists to catch this class of drift — the same class of issue (skill references a deleted/renamed file) will recur silently after every internal refactor unless a CI guard is added.

Detailed Analysis

The root cause is architectural: gh-aw previously embedded and bundled JavaScript directly into the Go binary via (go/redacted):embed (documented faithfully by javascript-refactoring and messages), then migrated to a model where JavaScript lives as standalone .cjs files under actions/{action-name}/js/, packaged per-action and referenced by workflows via actions/setup (827 files under actions/setup/js/). The migration left pkg/workflow/js.go in place as a stub (GetJavaScriptSources() returns an empty map; individual getters return "" or "EXTERNAL_SCRIPT" placeholders), presumably for backward-compatible call sites, but the two skills teaching contributors how to add new JS never got updated to match. Because these skills are the canonical guidance a Copilot agent loads before making JS-related changes (per the "Lazy Skill Loading Policy" in AGENTS.md), an agent has no signal that the documented workflow is defunct — it will successfully compile Go code that adds a dead embed variable while never touching the real actions/setup/js/*.cjs files the workflow runtime actually loads. The fix requires rewriting Steps 3 and 8 of javascript-refactoring and messages to follow actions/README.md's already-correct "Creating a New Action" / "Generating Actions from JavaScript Modules" pattern, plus lower-priority path-name touchups in developer-internals and error-pattern-safety, plus a CI guard to prevent recurrence.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: Split the following tasks into individual work items.

Improvement Tasks

Task 1: Rewrite javascript-refactoring skill for the actions/ architecture

Priority: High
Estimated Effort: Medium
Focus Area: Skill Documentation Drift

Description: Update .github/skills/javascript-refactoring/SKILL.md to remove all instructions referencing pkg/workflow/js.go as a live embed target, pkg/workflow/scripts.go, and pkg/workflow/bundler.go. Replace with accurate steps for adding/refactoring JavaScript under actions/{action-name}/js/*.cjs, following the pattern already documented in actions/README.md ("Generating Actions from JavaScript Modules" and "Creating a New Action" sections). Preserve any still-valid general refactoring guidance (splitting large .cjs files, require() usage, testing conventions).

Acceptance Criteria:

• No remaining references to pkg/workflow/scripts.go or pkg/workflow/bundler.go as real files to edit
• pkg/workflow/js.go references, if kept, are annotated as legacy/stub, or removed entirely
• New steps reference actions/setup/js/*.cjs and the action-based bundling process from actions/README.md
• Skill still validates against the repository's skill schema/lint (if any)

Code Region: .github/skills/javascript-refactoring/SKILL.md

Update .github/skills/javascript-refactoring/SKILL.md. The skill currently documents an obsolete architecture where JavaScript is embedded via (go/redacted):embed directives in pkg/workflow/js.go and bundled via pkg/workflow/scripts.go and pkg/workflow/bundler.go. That architecture no longer exists: pkg/workflow/js.go is now a stub whose GetJavaScriptSources() returns an empty map, and all real JavaScript for workflows now lives under actions/setup/js/*.cjs (packaged as part of the actions/setup custom action). Rewrite the "Step 3: Add Embedded Variable in Go" section (and any other section referencing js.go/scripts.go/bundler.go as live files) to instead describe the current process for adding or refactoring a .cjs script under actions/setup/js/, cross-referencing the existing accurate documentation in actions/README.md (sections "Generating Actions from JavaScript Modules" and "Creating a New Action"). Keep general refactoring advice (splitting large files, require() patterns, testing with *.test.cjs) that remains valid. Verify by grepping the updated file for "pkg/workflow/js.go", "scripts.go", and "bundler.go" and confirming any remaining mentions are clearly marked as historical/legacy context, not actionable instructions.

---

Task 2: Fix messages skill's obsolete Go-embedding steps

Priority: High
Estimated Effort: Small
Focus Area: Skill Documentation Drift

Description: Update .github/skills/messages/SKILL.md, specifically "Step 8: Register in Go Embeddings", the "Verification Checklist" item "Go embed directive added in js.go", and the "File Summary" table row for pkg/workflow/js.go. These currently instruct adding real (go/redacted):embed directives, which have no effect since js.go's GetJavaScriptSources() is hardcoded to return an empty map. Replace with the correct current step for wiring a new message's JavaScript module into actions/setup/js/ so it is actually shipped in compiled workflows.

Acceptance Criteria:

• Step 8 describes the real current mechanism for registering a new message's JS module (via actions/setup/js/), not the defunct js.go embed step
• Verification checklist and File Summary table updated to match
• Cross-check against a recent real PR or the safe_output_handlers.go registration flow to confirm accuracy before finalizing wording

Code Region: .github/skills/messages/SKILL.md

Update .github/skills/messages/SKILL.md. "Step 8: Register in Go Embeddings" (and the related Verification Checklist item "Go embed directive added in js.go" plus the File Summary table row referencing pkg/workflow/js.go) instruct contributors to add a (go/redacted):embed directive to pkg/workflow/js.go when adding a new safe-output message type. This is now a no-op: pkg/workflow/js.go is a stub file whose GetJavaScriptSources() function returns an empty map, and no JavaScript is actually embedded there anymore. Real safe-output message scripts live under actions/setup/js/*.cjs (see e.g. actions/setup/js/noop.cjs, actions/setup/js/close_issue.cjs as working examples of already-migrated message handlers), and registration/wiring happens through pkg/workflow/safe_output_handlers.go plus the action's own file listing, not through Go embed directives. Investigate the current registration flow by reading pkg/workflow/safe_output_handlers.go and an existing actions/setup/js/*.cjs message handler, then rewrite Step 8, the checklist item, and the File Summary table to describe the accurate current steps. Keep Steps 1-7 as-is if they remain accurate (JSON schema, Go struct, Go parser changes are likely still valid) — only fix the JS registration step.

---

Task 3: Correct renamed/split file references in developer-internals skill

Priority: Medium
Estimated Effort: Small
Focus Area: Skill Documentation Drift

Description: .github/skills/developer-internals/SKILL.md references pkg/workflow/strict_mode.go, pkg/workflow/validation_strict_mode.go, and pkg/workflow/expression_safety.go as single files (lines ~30, 33, 68, 92, 101). These have since been split into multiple files: strict-mode logic now spans strict_mode_validation.go, strict_mode_steps_validation.go, strict_mode_network_validation.go, strict_mode_permissions_validation.go, strict_mode_sandbox_validation.go, strict_mode_env_validation.go, strict_mode_update_check_validation.go, and strict_mode_llm_gateway_test.go/strict_mode_zizmor_test.go (plus corresponding _test.go files); expression safety now lives in expression_safety_validation.go and expression_safety_test.go. Update the file references and any accompanying diagrams to reflect the actual current file layout.

Acceptance Criteria:

• All pkg/workflow/strict_mode.go / validation_strict_mode.go / expression_safety.go mentions replaced with the correct current file name(s) for the context they appear in
• Any Mermaid/flow diagrams referencing these files updated to match
• grep -n "strict_mode.go\|validation_strict_mode.go\|expression_safety.go" .github/skills/developer-internals/SKILL.md returns no matches (or only matches clearly marked historical)

Code Region: .github/skills/developer-internals/SKILL.md (lines ~30, 33, 68, 92, 101)

Update .github/skills/developer-internals/SKILL.md. Around lines 30, 33, 68, 92, and 101, the skill references pkg/workflow/strict_mode.go, pkg/workflow/validation_strict_mode.go, and pkg/workflow/expression_safety.go as if they are single existing files (including in a Mermaid diagram around line 101). Run `ls pkg/workflow/ | grep -E "strict_mode|expression_safety"` to see the actual current files — strict-mode logic is now split across strict_mode_validation.go, strict_mode_steps_validation.go, strict_mode_network_validation.go, strict_mode_permissions_validation.go, strict_mode_sandbox_validation.go, strict_mode_env_validation.go, strict_mode_update_check_validation.go (plus _test.go counterparts), and expression safety is now expression_safety_validation.go and expression_safety_test.go. Update all references and diagram nodes in the skill to point at the correct, currently-existing file(s) for each described responsibility (read each candidate file briefly to confirm which one covers the specific behavior the skill text is describing before substituting).

---

Task 4: Add a skill-reference integrity CI check

Priority: Medium
Estimated Effort: Medium
Focus Area: Skill Documentation Drift (prevention)

Description: Add a lightweight Makefile target and/or CI step that scans all .github/skills/*/SKILL.md files for backtick-quoted paths that look like source files (.go, .cjs, .md under known doc/action dirs) and verifies they exist somewhere in the repo, failing (or warning) when a referenced file is missing. This turns today's ad-hoc audit into a repeatable check that prevents this class of drift from silently recurring after future refactors.

Acceptance Criteria:

• New Makefile target (e.g. make lint-skill-references) or script under pkg/ or scripts/ that performs the audit
• False-positive filtering for code-identifier-style mentions (e.g. anti-pattern examples like utils.go, helpers.go used illustratively) via an allowlist/exception mechanism, so the check is precise enough to be enabled by default
• Documented in .github/skills/developer/SKILL.md or equivalent as part of the standard validation playbook
• Initial run against current skills passes cleanly after Tasks 1-3 are merged (or flags only pre-approved exceptions)

Code Region: New file, e.g. pkg/cli/lint_skill_references.go or a scripts/check-skill-references.sh, plus Makefile

Add a repeatable check that audits .github/skills/*/SKILL.md files for stale file-path references, to prevent recurrence of the documentation drift found in this analysis (javascript-refactoring and messages skills referencing deleted/repurposed files like pkg/workflow/scripts.go, pkg/workflow/bundler.go). Implement as either a Go-based checker (consistent with other pkg/linters/ style checks in this repo) or a bash script, that: (1) extracts backtick-quoted strings matching common source file patterns (*.go, *.cjs, *.md under docs/ or .github/, *.json, *.yml/*.yaml) from every .github/skills/*/SKILL.md, (2) resolves each candidate path relative to repo root and to the skill's own directory, (3) reports any that don't exist anywhere in the repo, (4) supports an allowlist/exception list (e.g. YAML or JSON file) for known illustrative/anti-pattern filenames that are intentionally not real files (such as `utils.go`, `helpers.go`, `misc.go` used as naming anti-pattern examples in developer-code-organization/SKILL.md). Wire it into the Makefile as a new target (e.g. lint-skill-references) following the existing pattern of other lint-* targets in this repo's Makefile. Do not make it a blocking CI gate on first pass — start as a reporting-only target like the existing `make lint-error-messages-report`, consistent with how other new audits have been rolled out in this repo, and document the target in the developer skill's standard command list.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-08-24	Large File Decomposition Debt	Custom	Y	32 non-test files over 800 lines; 4 tasks
2026-08-25	Dependency Health & Vulnerability Posture	Standard	N	116 deps, 100 outdated, 0 reachable vulns; 4 tasks
2026-08-26	Error Message Actionability Debt	Custom	Y	2,631 non-actionable error messages via repo's own linter; 3 tasks
2026-08-28	Custom-Linter Auto-Fix Coverage Gap	Custom	Y	46/67 linters lack SuggestedFixes; 3 tasks
2026-08-31	Skill Documentation Drift (Stale JS Architecture Refs)	Custom	Y	2 skills with defunct-architecture instructions, 2 with renamed-file staleness; 4 tasks

---

🎯 Recommendations

Immediate Actions (This Week)

1. Rewrite javascript-refactoring/SKILL.md and messages/SKILL.md Step 8 to match the actions/ architecture — Priority: High

Short-term Actions (This Month)

1. Fix renamed-file references in developer-internals/SKILL.md and error-pattern-safety/SKILL.md — Priority: Medium
2. Stand up a reporting-only lint-skill-references CI check — Priority: Medium

Long-term Actions (This Quarter)

1. Consider adding a "skill freshness" review step to the release checklist whenever a subsystem undergoes an architectural migration (embed→actions being the exemplar case here) — Priority: Low

---

📈 Success Metrics

• Skills with broken file references: 4 (2 high-severity, 2 medium) → 0
• Automated staleness detection: None → make lint-skill-references reporting target in place

---

Next Steps

1. Review and prioritise the tasks above
2. Assign tasks to Copilot coding agent via planner agent
3. Track progress on improvement items
4. Re-evaluate this focus area in 1 quarter, or immediately after any future JS/embedding architecture change

---

Generated by Repository Quality Improvement Agent
Next analysis: 2026-09-01 — Focus area selected by diversity algorithm

Generated by ⚡ Repository Quality Improvement Agent · copilot · auto · 53.4 AIC · ⌖ 13.3 AIC · ⊞ 9.9K · ◷

• expires on Sep 1, 2026, 5:03 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-01T13:03:47.019Z.

Closed by Workflow