Skip to content

fix: encode compliance-fix learnings into standards and Claude prompt#86

Merged
don-petry merged 7 commits intomainfrom
fix/standards-from-compliance-learnings
Apr 8, 2026
Merged

fix: encode compliance-fix learnings into standards and Claude prompt#86
don-petry merged 7 commits intomainfrom
fix/standards-from-compliance-learnings

Conversation

@don-petry
Copy link
Copy Markdown
Contributor

@don-petry don-petry commented Apr 8, 2026

Why

After running the bulk compliance-issue automation across all 7 repos, a careful review of the resulting 12 PRs surfaced systemic problems in how Claude was approaching standards-conformance fixes:

Critical finding Affected PRs Root cause
Hallucinated SHAs when pinning actions broodly#91, markets#73 (now closed) Claude has no network in the runner; when asked to "pin to SHA" it fabricated plausible-looking commits. The runs only "passed CI" because the cargo-audit job was skipped (no Cargo.toml in the repo) so the bad SHA was never executed.
Stale SHAs TalkTerm#75 (now closed) Same root cause: SHA was real but not the current stable.
Generated workflow files from scratch instead of using standards/workflows/ templates TalkTerm#74, ContentTwin#50 (now closed) Claude wrote a 92-line homegrown agent-shield (vs the standard 102-rule ecc-agentshield scan) and a 25-line dependency-review-action (vs the 218-line multi-ecosystem audit standard).
Missed required ecosystem matrix in CodeQL TalkTerm#76 (now closed) Standard requires repos with .github/workflows/*.yml to scan the actions ecosystem; Claude only configured javascript-typescript.
Wrong label colors and missing labels TalkTerm#64 (now closed) Claude generated default GitHub labels instead of reading the org's required-label table.
administration: write not granted to claude-issue job bmad-bgreat-suite#77 (now closed) The thin caller can't grant a permission the called workflow doesn't request — needs to land in the reusable workflow itself.
Stale-context skip ("blocked by previous attempts") markets#46, TalkTerm#41, TalkTerm#42 (eventually retried successfully after manual prompt) Claude saw old "blocked" comments and gave up without retrying after the underlying issue was fixed.

What this PR changes

1. .github/workflows/claude-code-reusable.yml

  • Adds administration: write to the claude-issue job permissions so Claude can autonomously create labels, rulesets, and enable Discussions
  • Adds Bash(gh api:*) and Bash(gh label create:*) to --allowedTools so Claude can fetch templates and look up SHAs
  • Adds five new prompt rules covering: template-first workflow creation, org-standards lookup, real SHA verification, CodeQL ecosystem matrix, and stale-context behavior

2. standards/ci-standards.md

  • New top-level section "Using Templates from standards/workflows/" establishing the rule and listing all available templates
  • New subsection "Looking Up the Correct SHA" under Action Pinning Policy with gh api recipes for tags, branches, and SHA verification
  • Updates Permissions Policy table to add administration: write to the Claude Code row, with an explanatory note about how it propagates from the reusable workflow

3. AGENTS.md

  • New top-level section "Organization Standards" linking the standards root and each per-topic standard file (CI, workflows, agent-config, settings, dependabot)
  • Closes the gap that AGENTS.md previously only referenced petry-projects/.github once in passing (line 694), failing the spirit of the AgentShield cross-reference check

Test plan

  • CI on this PR passes
  • After merge: re-toggle the claude label on one closed compliance issue (e.g. TalkTerm#74 agent-shield) as a canary
  • Verify Claude now (a) fetches standards/workflows/agent-shield.yml via gh api instead of generating from scratch, (b) produces a PR matching the template byte-for-byte
  • If canary passes, bulk re-toggle the other 7 compliance issues that need rework

Summary by CodeRabbit

  • Documentation

    • Added Organization Standards guidance with topic-to-standard mapping and explicit remediation rules (read standard, copy canonical template, file an issue if missing).
    • Expanded CI standards with template-copying guidance, action-pinning lookup expectations, and standards-conformance rules for fixes and reporting.
  • Chores

    • Broadened automation workflow permissions and expanded allowed tool operations to improve automated remediation and orchestration, and clarified scanning requirements and handling of blocked prior comments.

Copilot AI review requested due to automatic review settings April 8, 2026 01:24
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Apr 8, 2026

Warning

Rate limit exceeded

@don-petry has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 8 minutes and 27 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 8 minutes and 27 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: dce1aded-ef4d-4941-98db-f2a7f28da510

📥 Commits

Reviewing files that changed from the base of the PR and between 57dae28 and e94809e.

📒 Files selected for processing (2)
  • .github/workflows/claude-code-reusable.yml
  • standards/ci-standards.md
📝 Walkthrough

Walkthrough

Updated the Claude Code reusable workflow to expand administration token handling, enlarge the claude_args allowlist, and extend the agent prompt with standards/conformance rules; added an "Organization Standards" section to AGENTS.md and expanded CI standards with template-copy and SHA-lookup guidance.

Changes

Cohort / File(s) Summary
Claude Code Workflow Configuration
​.github/workflows/claude-code-reusable.yml
Added documentation about using GH_PAT_WORKFLOWS for administration operations; added/clarified administration token usage; expanded claude_args allowlist with gh api:* and gh label create:*; extended Claude prompt with "Standards-conformance rules" (template-copy requirement, SHA lookup/pinning behavior, CodeQL guidance, blocked-comment handling).
Organization Standards Documentation
AGENTS.md
Inserted "Organization Standards" section pointing to petry-projects/.github/.../standards/ with a topic→standard map and a remediation rule: read standard first, copy canonical template when fixing compliance issues.
CI Standards Documentation
standards/ci-standards.md
Added requirement to copy workflow templates from standards/workflows/ (including gh api/base64 example); added "Looking Up the Correct SHA" subsection with GitHub API examples and failure behavior (don’t pin on lookup/network failure); clarified administration-token usage for workflows that need admin operations.

Sequence Diagram(s)

sequenceDiagram
  participant Repo as Repository (GitHub)
  participant Workflow as GH Actions Workflow (claude-issue)
  participant Claude as anthropics/claude-code-action
  participant GHAPI as GitHub API / gh CLI

  Workflow->>Claude: invoke standards check with claude_args
  Claude->>GHAPI: call `gh` ops (e.g., `gh api:*`, `gh label create:*`, resolve refs)
  GHAPI-->>Claude: return template contents / label results / commit SHAs
  Claude-->>Workflow: produce findings and remediation actions
  alt missing template or labels / fixes required
    Workflow->>GHAPI: create labels / fetch templates / apply changes (uses GH_PAT_WORKFLOWS admin token)
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~40 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately and specifically summarizes the primary change: encoding compliance-fix learnings into standards documentation and Claude prompts to prevent systemic workflow generation failures.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/standards-from-compliance-learnings

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Encodes lessons learned from org-wide compliance automation into the org standards docs and the reusable Claude workflow prompt, aiming to prevent workflow/template drift and to avoid incorrect GitHub Action SHA pinning.

Changes:

  • Add a “template-first” policy and SHA lookup recipes to standards/ci-standards.md, plus update the Claude Code permissions guidance.
  • Add an “Organization Standards” cross-reference section to AGENTS.md.
  • Extend the reusable Claude workflow (claude-code-reusable.yml) with administration: write, expanded --allowedTools, and additional prompt rules.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File Description
standards/ci-standards.md Adds template-first guidance + SHA lookup section; updates Claude permissions table and adds explanatory note.
AGENTS.md Adds a top-level “Organization Standards” section with canonical links to standards.
.github/workflows/claude-code-reusable.yml Adds admin permission, expands tool allowlist, and strengthens the automation prompt rules.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scratch. Available templates: `agent-shield.yml`, `claude.yml`,
`dependabot-automerge.yml`, `dependabot-rebase.yml`,
`dependency-audit.yml`, `feature-ideation.yml`. Fetch via:
`gh api repos/petry-projects/.github/contents/standards/workflows/<file>.yml --jq '.content' | base64 -d`
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new prompt suggests fetching workflow templates via the contents API and piping through base64 -d, but base64 isn't included in --allowedTools (only gh api is). Either allow Bash(base64:*) (and likely redirection) or change the suggested fetch command to use gh api -H "Accept: application/vnd.github.raw" ... > .github/workflows/<file>.yml so it works with the current tool allowlist.

Suggested change
`gh api repos/petry-projects/.github/contents/standards/workflows/<file>.yml --jq '.content' | base64 -d`
`gh api -H "Accept: application/vnd.github.raw" repos/petry-projects/.github/contents/standards/workflows/<file>.yml > .github/workflows/<file>.yml`

Copilot uses AI. Check for mistakes.
Comment on lines +645 to +648
> labels) require manual human action. The grant only takes effect on the
> `claude-issue` job in the reusable workflow — repo-level callers don't
> need to set it themselves because they use `secrets: inherit` and inherit
> the called workflow's permission shape via the intersection rule.
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This note describes the reusable-workflow permissions model incorrectly. For workflow_call, the called workflow’s GITHUB_TOKEN permissions are the intersection of (a) the caller job’s permissions and (b) the called workflow job’s permissions — so adding administration: write only in the reusable workflow will NOT take effect unless caller workflows also grant administration: write on the uses: job. secrets: inherit does not impact token permissions. Recommend updating the note and ensuring the standards/workflows/claude.yml template (and this repo’s .github/workflows/claude.yml) include administration: write when the automation job needs it.

Suggested change
> labels) require manual human action. The grant only takes effect on the
> `claude-issue` job in the reusable workflow — repo-level callers don't
> need to set it themselves because they use `secrets: inherit` and inherit
> the called workflow's permission shape via the intersection rule.
> labels) require manual human action. For reusable workflows invoked via
> `workflow_call`, the effective `GITHUB_TOKEN` permissions are the
> intersection of the caller job's `permissions` and the called workflow
> job's `permissions`, so adding `administration: write` only in the reusable
> workflow is not sufficient. Caller workflows must also grant
> `administration: write` on the `uses:` job when this automation needs
> repo-level administration access. `secrets: inherit` passes secrets only
> and does not affect token permissions.

Copilot uses AI. Check for mistakes.
Comment on lines +19 to +26
| Template | Purpose |
|----------|---------|
| [`agent-shield.yml`](workflows/agent-shield.yml) | Deep agent-config security scan via `ecc-agentshield` |
| [`claude.yml`](workflows/claude.yml) | Thin caller delegating to the org-level reusable Claude Code workflow |
| [`dependabot-automerge.yml`](workflows/dependabot-automerge.yml) | Auto-approve and squash-merge eligible Dependabot PRs |
| [`dependabot-rebase.yml`](workflows/dependabot-rebase.yml) | Rebase Dependabot PRs on demand |
| [`dependency-audit.yml`](workflows/dependency-audit.yml) | Multi-ecosystem audit (npm, pnpm, gomod, cargo, pip) |
| [`feature-ideation.yml`](workflows/feature-ideation.yml) | BMAD Method ideation pipeline (BMAD-enabled repos only) |
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude.yml is described here as a “thin caller delegating to the org-level reusable Claude Code workflow”, but the actual standards/workflows/claude.yml file currently contains the full workflow (it runs anthropics/claude-code-action directly rather than uses: .../claude-code-reusable.yml). This mismatch will confuse consumers and undermines the new “copy templates verbatim” rule; either update the template to match the thin-caller model or adjust this description to match reality.

Copilot uses AI. Check for mistakes.
Comment on lines +95 to +100
you're adding, **copy it verbatim** rather than writing from
scratch. Available templates: `agent-shield.yml`, `claude.yml`,
`dependabot-automerge.yml`, `dependabot-rebase.yml`,
`dependency-audit.yml`, `feature-ideation.yml`. Fetch via:
`gh api repos/petry-projects/.github/contents/standards/workflows/<file>.yml --jq '.content' | base64 -d`
Adapt only when the file genuinely needs repo-specific content.
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, this prompt lists claude.yml as one of the standards/workflows/ templates to copy verbatim, but standards/workflows/claude.yml is not currently a thin caller (it runs the Claude action directly). That makes the instruction ambiguous for repos trying to follow the new “template-first” rule; align the prompt with the actual template, or update the template to match the reusable-workflow delegation model.

Suggested change
you're adding, **copy it verbatim** rather than writing from
scratch. Available templates: `agent-shield.yml`, `claude.yml`,
`dependabot-automerge.yml`, `dependabot-rebase.yml`,
`dependency-audit.yml`, `feature-ideation.yml`. Fetch via:
`gh api repos/petry-projects/.github/contents/standards/workflows/<file>.yml --jq '.content' | base64 -d`
Adapt only when the file genuinely needs repo-specific content.
you're adding, use that template instead of writing from scratch.
Available templates: `agent-shield.yml`, `claude.yml`,
`dependabot-automerge.yml`, `dependabot-rebase.yml`,
`dependency-audit.yml`, `feature-ideation.yml`. Fetch via:
`gh api repos/petry-projects/.github/contents/standards/workflows/<file>.yml --jq '.content' | base64 -d`
For `claude.yml`, copy the standard thin caller template and keep
it delegating to the reusable Claude workflow rather than running
the Claude action inline. Adapt templates only when the file
genuinely needs repo-specific content.

Copilot uses AI. Check for mistakes.
Comment on lines 66 to 70
issues: write
actions: read
checks: read
administration: write
steps:
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding administration: write here will only affect the job-scoped GITHUB_TOKEN. In this job the action later receives github_token: ${{ secrets.GH_PAT_WORKFLOWS }} (a separate PAT), so unless that PAT is guaranteed to be present and to have equivalent admin capabilities, the new permission may not actually enable label/ruleset/discussions changes. Consider having the action default to ${{ secrets.GH_PAT_WORKFLOWS || github.token }} (or documenting that the PAT must include admin-level scopes) so the permission grant is effective.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/claude-code-reusable.yml:
- Line 69: The reusable workflow requests administration: write but the caller
job doesn't grant it; update the caller workflow (.github/workflows/claude.yml)
to add administration: write to the permissions block for the claude-code job so
the intersection of permissions includes admin scope; locate the claude-code job
in claude.yml and add the administration: write permission alongside the
existing contents: write and id-token: write entries to ensure admin-scoped
operations in claude-code can execute.

In `@AGENTS.md`:
- Line 24: The AGENTS.md line violating MD013 exceeds the configured max line
length; edit AGENTS.md and reflow/wrap the long sentence on Line 24 into shorter
lines (preserving the exact wording and punctuation) so each line is within the
repo's markdown max-length limit to satisfy MD013 — do not rewrite content or
change the template semantics, only insert line breaks to wrap the existing
sentence.

In `@standards/ci-standards.md`:
- Around line 641-648: Update the note to correctly state GitHub Actions
permission behavior: clarify that the reusable workflow's job (e.g., the
claude-issue job) requesting administration: write is necessary but not
sufficient — the calling job must also explicitly grant administration: write on
its GITHUB_TOKEN (callers cannot be elevated by the callee). Replace the
assertion about "secrets: inherit" and the "intersection rule" granting the
scope with guidance that both the reusable workflow and any caller must declare
administration: write on their respective jobs to allow repo-level changes
(labels, rulesets, Discussions).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 4fcee896-7f9c-4fcc-8960-a15b4743307f

📥 Commits

Reviewing files that changed from the base of the PR and between 92c3a85 and 80e54a5.

📒 Files selected for processing (3)
  • .github/workflows/claude-code-reusable.yml
  • AGENTS.md
  • standards/ci-standards.md

issues: write
actions: read
checks: read
administration: write
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Checking Claude workflow permission scopes..."
for f in .github/workflows/claude-code-reusable.yml .github/workflows/claude.yml standards/workflows/claude.yml; do
  if [ -f "$f" ]; then
    echo
    echo "=== $f ==="
    rg -n -C2 "permissions:|administration:" "$f" || true
  fi
done

echo
echo "Expected:"
echo "- administration: write present in reusable workflow job"
echo "- administration: write ALSO present in caller/template workflow job permissions"

Repository: petry-projects/.github

Length of output: 2068


administration: write on line 69 will not be effective unless the caller workflow also grants it.

The reusable workflow declares administration: write, but the caller (.github/workflows/claude.yml) still only grants contents: write, id-token: write, and related scopes. GitHub Actions enforces the intersection of caller-granted and callee-requested permissions, so any admin-scoped operations in claude-code will be constrained.

Update .github/workflows/claude.yml to include administration: write in the permissions section of the claude-code job.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/claude-code-reusable.yml at line 69, The reusable workflow
requests administration: write but the caller job doesn't grant it; update the
caller workflow (.github/workflows/claude.yml) to add administration: write to
the permissions block for the claude-code job so the intersection of permissions
includes admin scope; locate the claude-code job in claude.yml and add the
administration: write permission alongside the existing contents: write and
id-token: write entries to ensure admin-scoped operations in claude-code can
execute.

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Apr 8, 2026

@don-petry don-petry merged commit 2272db6 into main Apr 8, 2026
17 checks passed
@don-petry don-petry deleted the fix/standards-from-compliance-learnings branch April 8, 2026 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants