fix(codex): resolve codex: command not found on GPU runners inside AWF container#29467
Draft
fix(codex): resolve codex: command not found on GPU runners inside AWF container#29467
codex: command not found on GPU runners inside AWF container#29467Conversation
…WF container access Fixes codex: command not found on GPU runners (aw-gpu-runner-T4) where RUNNER_TOOL_CACHE=/home/runner/work/_tool is not mounted in the AWF chroot container. - Add installation step to copy codex binary to /tmp/gh-aw/npm-bins/ (only when firewall/AWF is enabled) so it's accessible inside the AWF container via the /tmp mount - Add /tmp/gh-aw/npm-bins to PATH in AWF container execution command as a fallback before the standard hostedtoolcache PATH setup - Add 3 tests to verify the new behavior - Recompile 205 workflows (daily-fact.lock.yml and others updated) Agent-Logs-Url: https://github.com/github/gh-aw/sessions/68631831-5458-4c19-8d4c-aaf3d022e6ed Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
- Use `command -v codex` instead of `which codex` for better portability - Add explicit error message and exit 1 if codex binary is not found after npm install, failing fast with a clear error message - Update test to check for `command -v codex` pattern - Recompile 11 codex-engine workflows with improved step Agent-Logs-Url: https://github.com/github/gh-aw/sessions/68631831-5458-4c19-8d4c-aaf3d022e6ed Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Fix codex command not found error in workflow
fix(codex): resolve May 1, 2026
codex: command not found on GPU runners inside AWF container
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On
aw-gpu-runner-T4,RUNNER_TOOL_CACHE=/home/runner/work/_tool— not/opt/hostedtoolcache. The AWF chroot container mounts/optand/tmpbut not/home/runner/work/_tool, so the codex binary installed there is invisible inside the container, causing daily failures.Changes
Installation (
GetInstallationSteps): When AWF/firewall is enabled, adds a post-install step that copies the codex binary to/tmp/gh-aw/npm-bins/on the host./tmpis mounted read-write in the AWF container, making codex accessible on any runner type.Execution (
GetExecutionSteps): Prependsexport PATH="/tmp/gh-aw/npm-bins:$PATH"before the standardGetNpmBinPathSetup()call inside the AWF container command, so the copied binary is found first on GPU runners where the hostedtoolcachefindyields nothing./tmp/gh-aw/npm-binsin AWF PATH.Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
https://api.github.com/graphql/usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE 4262739/b387/impGO111MODULE -c k/gh-aw/gh-aw/cmGOINSECURE k/gh-aw/gh-aw/cmGOMOD 64/bin/go GOSUMDB GOWORK 64/bin/go /opt/hostedtoolcGO111MODULE(http block)/usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE 4262739/b400/impGO111MODULE -c 4262739/b400/embGOINSECURE GOPROXY 64/bin/go GOSUMDB GOWORK 64/bin/go /opt/hostedtoolcGO111MODULE(http block)/usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE 4262739/b403/impGO111MODULE -c k/gh-aw/gh-aw/pkGOINSECURE GOPROXY 64/bin/go GOSUMDB GOWORK run-script/lib/n-json /opt/hostedtoolcGO111MODULE(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v6/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE node /opt�� prettier --check 64/bin/go **/*.ts **/*.json --ignore-path git(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv ignore-path ../../../.prettierignore config /usr/bin/git remote.origin.ursh GOWORK 64/bin/go git -C /workflows/daily-fact.lock.yml config /usr/bin/git l GOPROXY 64/bin/go git(http block)https://api.github.com/repos/actions/setup-node/git/ref/tags/v6/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE node /opt�� prettier --check 64/bin/go **/*.ts **/*.json --ignore-path git(http block)/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv ignore-path ../../../.prettierignore TOKEN"; }; f get nt-scoped-approved.lock.yml l owner=github ed } } /usr/bin/gh api /workflows/daily-fact.lock.yml -f modules/@npmcli/run-script/lib/node-gyp-bin/node -f owner=github -f infocmp(http block)https://api.github.com/repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b/usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv --check **/*.cjs 64/bin/go **/*.json --ignore-path run-script/lib/n-t go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv prettier --check 64/bin/go --ignore-path .prettierignore 64/bin/go go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv /workflows/grumpy-reviewer.lock.yml config deql remote.origin.urnode bash 64/bin/go deql -C w/js/**/*.json' --ignore-path show(http block)https://api.github.com/repos/github/gh-aw/usr/bin/gh gh api /repos/github/gh-aw --jq .default_branch -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/github/gh-aw --jq .default_branch 7365980c12808757474442ea:.github-f go bash -json GO111MODULE ache/go/1.25.8/xxterm-color git push�� *.json' '!../../../pkg/workflow/js/**/*.json' --ignore-path ../../../.prettierignore origin repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -json GO111MODULE ache/go/1.25.8/x/home/REDACTED/work/gh-aw/gh-aw/.github/workflows git(http block)If you need me to access, download, or install something from one of these locations, you can either: