Skip to content

fix(codex): resolve codex: command not found on GPU runners inside AWF container#29467

Draft
Copilot wants to merge 3 commits intomainfrom
copilot/aw-fix-codex-command-error
Draft

fix(codex): resolve codex: command not found on GPU runners inside AWF container#29467
Copilot wants to merge 3 commits intomainfrom
copilot/aw-fix-codex-command-error

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 1, 2026

On aw-gpu-runner-T4, RUNNER_TOOL_CACHE=/home/runner/work/_tool — not /opt/hostedtoolcache. The AWF chroot container mounts /opt and /tmp but not /home/runner/work/_tool, so the codex binary installed there is invisible inside the container, causing daily failures.

Changes

  • Installation (GetInstallationSteps): When AWF/firewall is enabled, adds a post-install step that copies the codex binary to /tmp/gh-aw/npm-bins/ on the host. /tmp is mounted read-write in the AWF container, making codex accessible on any runner type.

  • Execution (GetExecutionSteps): Prepends export PATH="/tmp/gh-aw/npm-bins:$PATH" before the standard GetNpmBinPathSetup() call inside the AWF container command, so the copied binary is found first on GPU runners where the hostedtoolcache find yields nothing.

# New installation step (firewall-enabled workflows only)
- name: Copy Codex binary to AWF-accessible location
  run: |
    CODEX_BIN="$(command -v codex)" || { echo '::error::codex binary not found after npm install' >&2; exit 1; }
    mkdir -p /tmp/gh-aw/npm-bins && cp "$CODEX_BIN" /tmp/gh-aw/npm-bins/codex

# Execution inside AWF container (before existing PATH setup)
export PATH="/tmp/gh-aw/npm-bins:$PATH" && export PATH="$(find /opt/hostedtoolcache ...)$PATH" && codex exec ...
  • Tests: 3 new unit tests — copy step present with firewall, absent without firewall, /tmp/gh-aw/npm-bins in AWF PATH.
  • 11 codex-engine workflow lock files recompiled.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE 4262739/b387/impGO111MODULE -c k/gh-aw/gh-aw/cmGOINSECURE k/gh-aw/gh-aw/cmGOMOD 64/bin/go GOSUMDB GOWORK 64/bin/go /opt/hostedtoolcGO111MODULE (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE 4262739/b400/impGO111MODULE -c 4262739/b400/embGOINSECURE GOPROXY 64/bin/go GOSUMDB GOWORK 64/bin/go /opt/hostedtoolcGO111MODULE (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE 4262739/b403/impGO111MODULE -c k/gh-aw/gh-aw/pkGOINSECURE GOPROXY 64/bin/go GOSUMDB GOWORK run-script/lib/n-json /opt/hostedtoolcGO111MODULE (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE node /opt�� prettier --check 64/bin/go **/*.ts **/*.json --ignore-path git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv ignore-path ../../../.prettierignore config /usr/bin/git remote.origin.ursh GOWORK 64/bin/go git -C /workflows/daily-fact.lock.yml config /usr/bin/git l GOPROXY 64/bin/go git (http block)
  • https://api.github.com/repos/actions/setup-node/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE node /opt�� prettier --check 64/bin/go **/*.ts **/*.json --ignore-path git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv ignore-path ../../../.prettierignore TOKEN"; }; f get nt-scoped-approved.lock.yml l owner=github ed } } /usr/bin/gh api /workflows/daily-fact.lock.yml -f modules/@npmcli/run-script/lib/node-gyp-bin/node -f owner=github -f infocmp (http block)
  • https://api.github.com/repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv --check **/*.cjs 64/bin/go **/*.json --ignore-path run-script/lib/n-t go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv prettier --check 64/bin/go --ignore-path .prettierignore 64/bin/go go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv /workflows/grumpy-reviewer.lock.yml config deql remote.origin.urnode bash 64/bin/go deql -C w/js/**/*.json' --ignore-path show (http block)
  • https://api.github.com/repos/github/gh-aw
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .default_branch -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .default_branch 7365980c12808757474442ea:.github-f go bash -json GO111MODULE ache/go/1.25.8/xxterm-color git push�� *.json' '!../../../pkg/workflow/js/**/*.json' --ignore-path ../../../.prettierignore origin repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -json GO111MODULE ache/go/1.25.8/x/home/REDACTED/work/gh-aw/gh-aw/.github/workflows git (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI linked an issue May 1, 2026 that may be closed by this pull request
Copilot AI and others added 2 commits May 1, 2026 08:41
…WF container access

Fixes codex: command not found on GPU runners (aw-gpu-runner-T4) where
RUNNER_TOOL_CACHE=/home/runner/work/_tool is not mounted in the AWF
chroot container.

- Add installation step to copy codex binary to /tmp/gh-aw/npm-bins/
  (only when firewall/AWF is enabled) so it's accessible inside the
  AWF container via the /tmp mount
- Add /tmp/gh-aw/npm-bins to PATH in AWF container execution command
  as a fallback before the standard hostedtoolcache PATH setup
- Add 3 tests to verify the new behavior
- Recompile 205 workflows (daily-fact.lock.yml and others updated)

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/68631831-5458-4c19-8d4c-aaf3d022e6ed

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
- Use `command -v codex` instead of `which codex` for better portability
- Add explicit error message and exit 1 if codex binary is not found
  after npm install, failing fast with a clear error message
- Update test to check for `command -v codex` pattern
- Recompile 11 codex-engine workflows with improved step

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/68631831-5458-4c19-8d4c-aaf3d022e6ed

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix codex command not found error in workflow fix(codex): resolve codex: command not found on GPU runners inside AWF container May 1, 2026
Copilot AI requested a review from gh-aw-bot May 1, 2026 08:49
@pelikhan pelikhan added the awf label May 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aw] Workflow Health Dashboard — 2026-04-30

3 participants