Lockdown check fails on private cross-repo workflow_call: Contents API uses caller's GITHUB_TOKEN, not callee-accessible token

[norrietaylor]

Version: gh-aw v0.72.1

Symptom

Every consumer workflow_call dispatch against a private gh-aw callee fails at the lock.yml's activation step with:

Cross-repo invocation detected: workflow source is "<callee>",
  current repo is "<caller>"
GET /repos/<callee>/contents/.github/workflows/<chore>.lock.yml
    ?ref=<callee-ref-sha> — 404 Not Found
Could not fetch content for .github/workflows/<chore>.lock.yml: Not Found
Unable to fetch lock file content for hash comparison via API,
  trying local filesystem fallback
Local lock file not found: …/work/<caller>/<caller>/.github/workflows/<chore>.lock.yml
##[warning]Could not compare frontmatter hashes — assuming lock file is outdated
##[error]ERR_CONFIG: Lock file '.github/workflows/<chore>.lock.yml' is outdated
  or unverifiable! Could not verify frontmatter hash for
  '.github/workflows/<chore>.md'. Run 'gh aw compile' to regenerate the lock file.

The check_workflow_timestamp_api.cjs activation step then sets stale_lock_file_failed=true, and the conclusion job marks the whole run failed before the agent ever runs.

Concrete reproducer: gominimal/min-ctl run 25906428806 calls gominimal/min-aw/.github/workflows/worker-fix.lock.yml@v0.6.3. Both repos private, same org. Fails as above on every dispatch.

Root cause

check_workflow_timestamp_api.cjs issues the Contents API call against the callee repo using GITHUB_TOKEN from the caller's runner. For a private callee:

• GITHUB_TOKEN on a workflow run is repo-scoped to the running repo (the caller, e.g. gominimal/min-ctl).
• The Contents API on a private repo requires read access to that specific repo.
• gominimal/min-ctl's GITHUB_TOKEN has no scope on gominimal/min-aw → 404, even though both repos are in the same org and workflow_call itself was already authorized via repo-actions-access settings.

This works fine when the callee is public (elastic/ai-github-actions is public; their consumers don't see this) — public Contents API doesn't require auth. The gap is private-callee-private-caller.

Why the existing fallbacks don't help

1. API path uses GITHUB_TOKEN → 404 for private callees.
2. GH_AW_GITHUB_TOKEN fallback secret slot is declared optional in the lock.yml's workflow_call.secrets. Consumers can provide a token here, but the documentation/templates don't make this clear, and there's no practical way to mint a short-lived cross-repo token without long-lived PAT storage (security cost).
3. Local filesystem fallback looks for the lock.yml on the caller's checkout. Consumers using the wrapper distribution pattern don't ship .lock.yml files locally (that's the point — thin wrappers).

Proposed fix

The lock.yml already receives APP_PRIVATE_KEY via workflow_call.secrets (used for safe-output writes via actions/create-github-app-token). If the corresponding GitHub App is installed on the callee repo as well, an App-minted installation token has cross-repo Contents read access.

check_workflow_timestamp_api.cjs should:

1. Detect cross-repo invocation (it already does — emits "Cross-repo invocation detected" log).
2. Detect that the initial Contents API call returned 404 / 401 / 403.
3. Mint a fresh installation token using APP_PRIVATE_KEY + vars.APP_ID (both already available to the activation step), scoped to the callee repo.
4. Retry the Contents API call with the minted token.
5. If the mint fails (no APP_PRIVATE_KEY in secrets, App not installed on callee, etc.), fall through to today's behavior with a clearer error message: "Configure GH_AW_GITHUB_TOKEN with cross-repo Contents read access, or install the gh-aw App on the callee repo."

Alternative if internal mint is too invasive: emit a clearer error directing consumers to set up GH_AW_GITHUB_TOKEN. The current message ("outdated or unverifiable") sends users toward gh aw compile, which doesn't address the auth gap.

Workaround for current users

Ship .lock.yml mirrors on the consumer side alongside .md mirrors. The filesystem fallback then succeeds. Trade-off: ~80–90 KB per chore on the consumer (defeats the thin-wrapper goal). See gominimal/min-ctl#66 for the workaround in practice.

Related

• Bundle-apply race: safe_outputs fetch-depth=1 loses ancestor commits between agent and safe_outputs jobs #32310 — bundle-apply race in safe_outputs (separate root cause, same setup)

[kkruel8100]

Experiencing same issue. Proposed workaround is not viable on central ops repo and large scale quantity of target repos. Please increase priority level and mark bug status.

[pelikhan]

/scout do a deep research the mentioned workflows and repos and provide a summary with snippets of the current situation

[github-actions]

🏕️ Lost in the wilderness! Scout failed to deliver outputs. Sending search party...

[github-actions]

🔍 Scout Research Report

Triggered by @pelikhan via /scout comment

Executive Summary

The bug in #32312 is real, reproducible from the code in this repo, and the proposed fix path (mint an App-installation token in the activation step when cross-repo + 404/401/403) is sound. The root cause confirmed in code: the compiled Check workflow lock file step in every *.lock.yml runs actions/github-script without specifying a github-token, so it inherits the caller's repo-scoped GITHUB_TOKEN — which has no read access to the private callee. Prior fixes (#23935, #24422) taught the script which file to look for cross-repo but did not change which token it uses to fetch it. The same reporter (@norrietaylor) filed a separate cross-repo race in #32310, suggesting the wrapper-distribution pattern is exercising a class of gaps.

Click to expand detailed findings

Research Findings

1. Root cause confirmed in scout.lock.yml (and every compiled lock.yml)

The activation step that calls check_workflow_timestamp_api.cjs is generated without a github-token: override, so github inside github-script is authenticated with the default caller GITHUB_TOKEN:

# .github/workflows/scout.lock.yml:223-234
- name: Check workflow lock file
  id: check-lock-file
  uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
  env:
    GH_AW_WORKFLOW_FILE: "scout.lock.yml"
    GH_AW_CONTEXT_WORKFLOW_REF: "${{ github.workflow_ref }}"
  with:
    script: |
      const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs');
      setupGlobals(core, github, context, exec, io, getOctokit);
      const { main } = require('${{ runner.temp }}/gh-aw/actions/check_workflow_timestamp_api.cjs');
      await main();

Compare to safe-output steps in the same file that do use the fallback chain (scout.lock.yml:1347+):

github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}

The activation step is missing this. getFileContent (actions/setup/js/github_api_helpers.cjs:69-101) then calls github.rest.repos.getContent on the callee owner/repo with the caller's token, producing the 404 the issue describes.

2. Cross-repo detection already exists — only the token slot is missing

check_workflow_timestamp_api.cjs already does the right resolution (actions/setup/js/check_workflow_timestamp_api.cjs:86-160): it calls actions.getWorkflowRun to read referenced_workflows[], picks sha over ref, and logs Cross-repo invocation detected: workflow source is "<callee>", current repo is "<caller>" — that's the exact line in the issue's error block. So the script already knows it's cross-repo. It just doesn't switch tokens when it does.

The inline comments name this gap explicitly:

// Fallback: compare frontmatter hashes using local filesystem files.
// Used when the GitHub API is inaccessible (e.g., cross-org reusable workflow where
// the caller's GITHUB_TOKEN cannot read the source repo).

That fallback is the local checkout, which doesn't exist on wrapper consumers — the issue is correct that filesystem fallback is structurally useless for the thin-wrapper distribution pattern.

3. Prior fixes addressed the file, not the auth

• Integrity check fails for cross-repo workflow_call: GITHUB_WORKFLOW_REF points to caller, not callee #23935 (closed 2026-04-03, v0.65.4): fixed GITHUB_WORKFLOW_REF pointing to caller instead of callee for workflow_call. Same script, same symptom name ("outdated or unverifiable") but a different root cause — wrong path lookup.
• Remote workflow call still broken #24422 (closed 2026-04-04): bundled fix of two interacting bugs in cross-repo reusable workflows; introduced the referenced_workflows API lookup the script uses today.

Both fixes ran on public callees where Contents API needs no auth. The private-callee case was never reached by those fixes because the test fixtures didn't exercise it.

4. Related bug in same setup: #32310 (closed)

Same reporter, same gominimal/min-aw ↔ gominimal/min-ctl repos. Bundle-apply race in safe_outputs because checkout uses fetch-depth: 1 and the bundle's base SHA falls out of reach when main advances between the agent and safe-outputs jobs. Different root cause, but it confirms the wrapper-distribution + private-callee pattern is being actively used and surfacing edge cases the existing test coverage misses.

5. The lock.yml secrets contract already names the right slot

Every compiled workflow_call: block already declares the optional slot (e.g. dependabot-worker.lock.yml):

workflow_call:
  secrets:
    GH_AW_GITHUB_TOKEN:
      required: false
    GH_AW_GITHUB_MCP_SERVER_TOKEN:
      required: false

So the consumer-side mechanism to pass a cross-repo-capable token already exists. The activation step just doesn't read it. Adding github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} to the compiled Check workflow lock file step (in compiler_activation_job_builder.go / compiler_activation_job.go) would let consumers who provision the secret pass it through without any new mechanism.

6. App-token mint path is feasible and idiomatic

The reporter's preferred fix (use APP_PRIVATE_KEY + vars.APP_ID already in scope to mint a fresh installation token) is the GitHub-recommended pattern for cross-repo private reads — see the [Cloud Chronicles guide]((cloudchronicles.blog/redacted) and GitHub's own authentication docs: "If you need a token that requires permissions that aren't available in the GITHUB_TOKEN, create a GitHub App and generate an installation access token within your workflow." actions/create-github-app-token accepts an explicit owner: parameter so the minted token can be scoped to the callee org/repo. The same action is already used elsewhere in safe-output jobs in this codebase, so introducing it into the activation job is consistent.

Key constraint to verify before implementing: the App must be installed on the callee repo too, not just the caller. The fallback error message should make this explicit.

7. The error message points users in the wrong direction

From check_workflow_timestamp_api.cjs:343:

Lock file '...' is outdated or unverifiable! Could not verify frontmatter hash for '...'.
Run 'gh aw compile' to regenerate the lock file.

When the actual failure is an auth 404 against the callee, gh aw compile does nothing useful — the lock file is fine, the script just can't read it. Even without changing the auth behavior, branching the message on the cross-repo + API-failure case would save users the wild-goose chase.

Recommendations

In priority order:

1. Minimal fix (low risk, ships value immediately): In compiler_activation_job_builder.go, emit github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} on the Check workflow lock file step. Consumers who already provision GH_AW_GITHUB_TOKEN (PAT or App-minted) get the fix for free; no new contract.
2. Sharper error message (independent of fix rejig docs #1): When workflowRepo !== currentRepo AND the API call returns 401/403/404 AND the local filesystem fallback finds no lock file, fail with: "Cross-repo lock check failed: the caller's GITHUB_TOKEN cannot read <callee>. Provision GH_AW_GITHUB_TOKEN (PAT or App installation token with Contents:read on the callee), or install the gh-aw GitHub App on the callee repo." Suppress the "Run gh aw compile" suggestion in this branch — it's misleading.
3. Deeper fix (App-mint, what the issue proposes): Detect cross-repo + Contents-API-failure, then if APP_PRIVATE_KEY + APP_ID are present, mint an installation token scoped to the callee owner and retry. Falls back to message in (2) when the mint fails. This removes the consumer's burden of provisioning a separate PAT but adds App-install-on-callee as a prerequisite — document that clearly.
4. Test coverage: Add a cross-repo + private-callee integration test fixture. The existing pkg/workflow/call_workflow_secrets_test.go validates the secrets list shape but doesn't exercise the activation-step auth path. A test that builds a fake referenced_workflows API response and asserts the step uses the fallback token (not GITHUB_TOKEN) would have caught this and would catch regressions in (1)/(3).
5. Docs: Update docs/src/content/docs/reference/cross-repository.md with a "Private callee" section. Today it covers "reading from other private repositories" for MCP tooling but not the lock-file activation path.

The minimum to unblock @norrietaylor and @kkruel8100 right now is #1 + #2. Those are a few-line change in the compiler and a conditional in the script; both are independent of the App-mint design and can ship in a patch release without breaking any existing consumer.

Key Sources

• This repo: actions/setup/js/check_workflow_timestamp_api.cjs — lines 86–160 (cross-repo detection), 246–282 (compareFrontmatterHashes), 343 (misleading error)
• This repo: actions/setup/js/github_api_helpers.cjs:69-101 — getFileContent using caller-scoped github client
• Compiled example: .github/workflows/scout.lock.yml:223-234 — activation step with no github-token:
• Prior fixes: #23935, #24422
• Related open bug, same setup: #32310 (bundle-apply race)
• GitHub Community discussion confirming GITHUB_TOKEN is repo-scoped by design: community/discussions/46566
• GitHub Docs on App-token mint pattern: Authentication in a workflow
• Cross-org App-token recipe: [Cloud Chronicles guide]((cloudchronicles.blog/redacted)

Suggested Next Steps

1. Open a small PR adding github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} to the activation step generator and regenerating affected lock files (recommendation 1).
2. In the same PR or a follow-up, branch the warning message on cross-repo + API-failure (recommendation 2).
3. Track the App-mint path (recommendation 3) as a separate issue/PR — it has a larger surface (new dependency on the App being installed on callees, secret naming convention) and shouldn't block the quick wins.
4. Add a compiler test ensuring the activation step's actions/github-script invocation carries the fallback github-token (recommendation 4).
5. Note for the gominimal consumers: until the fix lands, the documented workaround in gominimal/min-ctl#66 (ship .lock.yml mirrors) works, but provisioning GH_AW_GITHUB_TOKEN as a workflow_call secret with Contents:read on the callee repo will work today without any gh-aw change — the secret slot is already declared in every compiled lock.yml.

🔭 Intelligence gathered by Scout · ● 11.3M · ◷

[pelikhan]

@copilot follow scout plan, ignore app mint for now.

[norrietaylor]

Thanks Peli!

I will try this out once it lands and report back.