Skip to content

Fix stale $INSTRUCTION assertion in TestEngineArgsIntegrationCodex#30100

Merged
pelikhan merged 3 commits intomainfrom
copilot/aw-fix-workflow-health-issues
May 4, 2026
Merged

Fix stale $INSTRUCTION assertion in TestEngineArgsIntegrationCodex#30100
pelikhan merged 3 commits intomainfrom
copilot/aw-fix-workflow-health-issues

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 4, 2026

TestEngineArgsIntegrationCodex was failing in CI because it asserted that "$INSTRUCTION" appeared in the compiled YAML — a pattern from before the codex harness (codex_harness.cjs) was introduced. Since CodexEngine.GetHarnessScriptName() always returns "codex_harness.cjs", the prompt is now passed via --prompt-file and $INSTRUCTION is never emitted.

Change

  • Updated the test to verify --custom-flag value appears before --prompt-file instead of before "$INSTRUCTION":
// Before
instructionIdx := strings.Index(result, "\"$INSTRUCTION\"")
if customFlagIdx == -1 || instructionIdx == -1 {
    t.Fatal("Could not find both --custom-flag and $INSTRUCTION in compiled YAML")
}

// After
promptFileIdx := strings.Index(result, "--prompt-file")
if customFlagIdx == -1 || promptFileIdx == -1 {
    t.Fatal("Could not find both --custom-flag and --prompt-file in compiled YAML")
}

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh gh repo view --json owner,name --jq .owner.login + "/" + .name x_amd64/link GOINSECURE GOMOD GOMODCACHE x_amd64/link env /ref/tags/v9.0.0 GO111MODULE sv GOINSECURE GOMOD GOMODCACHE J_/CWrYu2czG7Ca7ylQP4Z8/vCNYLdc7D8RXanEmFBss (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw util GOMODCACHE go env t_NmpAeXU GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE rtcfg (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go sRem�� -json GO111MODULE util.test GOINSECURE GOMOD GOMODCACHE util.test (http block)
  • https://api.github.com/orgs/test-owner/actions/secrets
    • Triggering command: /usr/bin/gh gh api /orgs/test-owner/actions/secrets --jq .secrets[].name /tmp/go-build100GOINSECURE -trimpath 64/bin/go -d github.com/ayman-atomic -lang=go1.24 go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/actions/ai-inference/git/ref/tags/v1
    • Triggering command: /usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq [.object.sha, .object.type] | @tsv /ref/tags/v9 go sv ithub/workflows/git GO111MODULE ache/node/24.14.--show-toplevel git rev-�� --show-toplevel /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linux_amd64/link /usr/bin/git bility_SameInputgh -importcfg /usr/lib/git-cor/repos/actions/github-script/git/ref/tags/v9 git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v3
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq [.object.sha, .object.type] | @tsv add resolved$ /usr/bin/git ty-test.md GO111MODULE 64/bin/go git rev-�� --show-toplevel go /usr/bin/git vent, headBranchgh GO111MODULE DiscussionsEnabl/repos/actions/github-script/git/ref/tags/v9 git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v5
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv runs/20260504-075927-35634/test-source-field-variant-63854068/.github/workflows GO111MODULE /opt/hostedtoolcache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json =my-default ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv --show-toplevel /usr/lib/git-cor--jq /usr/bin/git run --auto /opt/hostedtoolc--show-toplevel git rev-�� ithub-script/git/ref/tags/v9 node bject.type] | @tsv ons-test20829191git go ps git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv --show-toplevel git /usr/bin/infocmp --show-toplevel go /usr/bin/infocmp--show-toplevel infocmp -1 xterm-color infocmp /opt/hostedtoolcache/go/1.25.8/x64/bin/node xterm-color 620722/b413/consrev-parse /usr/bin/git node (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv runs/20260504-075927-35634/test--f node /usr/lib/git-core/git l --check 64/bin/go /usr/lib/git-core/git unpa�� --pack_header=2,3 -q /opt/hostedtoolcache/node/24.14.1/x64/bin/node -json GOMOD 64/bin/go node (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv --show-toplevel 620722/b452/_testmain.go /tmp/go-build145620722/b450/styl-f -json GO111MODULE 64/bin/go /tmp/go-build145620722/b450/styles.test -tes�� -test.paniconexit0 s/4/artifacts /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linux_amd64/vet -test.timeout=10git -test.run=^Test -test.short=true--show-toplevel /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linux_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv thImports871149745/001 -importcfg /usr/bin/git -s -w -buildmode=exe git conf�� --get remote.origin.url /usr/bin/git -json GO111MODULE 64/bin/go git (http block)
  • https://api.github.com/repos/actions/github-script/git/ref/tags/v8
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq [.object.sha, .object.type] | @tsv /repos/actions/setup-node/git/ref/tags/v4 --jq (http block)
  • https://api.github.com/repos/actions/github-script/git/ref/tags/v9
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv ntic-workflows.agent.md -specification-writer.agent.md ate-safe-output-type.agent.md mpy-reviewer.age/opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linux_amd64/vet hnical-doc-write-unsafeptr=false eloper.instructi-unreachable=false tribution-checke/tmp/go-build131722364/b082/vet.cfg erac�� tom-engine-implementation.agent.OUTPUT -writer.agent.md 64/bin/grep (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv enerate-needs-ex-errorsas pdate-actions-gi-ifaceassert rgo/bin/git (http block)
  • https://api.github.com/repos/actions/github-script/git/ref/tags/v9.0.0
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9.0.0 --jq [.object.sha, .object.type] | @tsv GOSUMDB GOWORK 64/bin/go GOSUMDB GOWORK 64/bin/go sh -c "prettier" --cheGOINSECURE go 64/bin/go tierignore GO111MODULE 64/bin/go go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9.0.0 --jq [.object.sha, .object.type] | @tsv prettier --check 64/bin/go **/*.ts **/*.json --ignore-path git cat-�� --end-of-optionsGOINSECURE blob 64/bin/go -json GO111MODULE 64/bin/go go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9.0.0 --jq [.object.sha, .object.type] | @tsv che/go-build/3f/GOINSECURE GOPROXY 64/bin/go GOSUMDB GOWORK 64/bin/go /opt/hostedtoolcGO111MODULE -o /tmp/go-build100GOINSECURE -trimpath 64/bin/go -p main -lang=go1.25 go (http block)
  • https://api.github.com/repos/actions/setup-go/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv /repos/actions/github-script/git/ref/tags/v9 --jq /usr/bin/gh -json GO111MODULE cfg gh api /repos/actions/github-script/git/ref/tags/v9 --jq /usr/bin/git ithub/workflows GO111MODULE (http block)
  • https://api.github.com/repos/actions/setup-node/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --get remote.origin.url /usr/bin/git -json GO111MODULE 64/bin/go git add test.txt resolved$ /usr/bin/git -json GO111MODULE 64/bin/go /usr/bin/git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv xterm-color go /usr/bin/git SameOutput276308gh (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv ithub-script/git/ref/tags/v9 git bject.type] | @tsv ons-test20829191git remote /usr/bin/infocmp--show-toplevel git rev-�� /ref/tags/v9 infocmp sv s/test.md go _id":222}] infocmp (http block)
  • https://api.github.com/repos/actions/setup-node/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv runs/20260504-075927-35634/test-2097910701 node /usr/lib/git-core/git prettier --check 64/bin/go /usr/lib/git-core/git --gi�� 3721681324 --format=%(objectname) /usr/bin/git -json GOCACHE 64/bin/go git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv /tmp/gh-aw-test-runs/20260504-075927-35634/test-2652720492/.github/workflows (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv /ref/tags/v9 go sv -json GO111MODULE 64/bin/go git rev-�� --show-toplevel go /usr/bin/git -json GO111MODULE 64/bin/go git (http block)
  • https://api.github.com/repos/actions/upload-artifact/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/upload-artifact/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv GOMODCACHE go /usr/bin/git -json GO111MODULE 64/bin/go git ent.�� .md md (http block)
  • https://api.github.com/repos/aws-actions/configure-aws-credentials/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/aws-actions/configure-aws-credentials/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel git /usr/bin/git /tmp/TestGuardPogit rev-parse /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git --show-toplevel go om/owner/repo.gistatus git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/aws-actions/configure-aws-credentials/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel git /usr/bin/git /tmp/TestGuardPogh config /usr/bin/git git rev-�� --show-toplevel git /usr/bin/infocmp --get remote.origin.urrev-parse /usr/bin/git infocmp (http block)
  • https://api.github.com/repos/azure/login/git/ref/tags/v2
    • Triggering command: /usr/bin/gh gh api /repos/azure/login/git/ref/tags/v2 --jq [.object.sha, .object.type] | @tsv --show-toplevel git /usr/bin/git ons-test20829191git rev-parse /opt/hostedtoolc--show-toplevel git rev-�� --show-toplevel node /usr/bin/git /tmp/TestHashConinfocmp go om/myorg/repo.gixterm-color git (http block)
  • https://api.github.com/repos/docker/login-action/git/ref/tags/v3
    • Triggering command: /usr/bin/gh gh api /repos/docker/login-action/git/ref/tags/v3 --jq [.object.sha, .object.type] | @tsv ithub-script/git/ref/tags/v9 node bject.type] | @tsv ons-test20829191git go ps git rev-�� --show-toplevel ps /usr/bin/git git url pts.test git (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v0.1.2
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0.1.2 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/go/1.25.8/x64/pkg/tool/linux_amd64/link /usr/bin/git 620722/b413/consgh GO111MODULE 64/pkg/tool/linu/repos/actions/github-script/git/ref/tags/v9 git rev-�� --show-toplevel 64/pkg/tool/linux_amd64/compile /usr/bin/git _.a GO111MODULE 620722/b413/_pkgxterm-color git (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv --show-toplevel /tmp/go-build145620722/b450/_testmain.go /usr/bin/infocmp -json GO111MODULE 64/bin/go infocmp -1 xterm-color go /usr/bin/git -json GO111MODULE x_amd64/link git (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.2.3
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-04-27 GOMOD GOMODCACHE go env LsRemoteWithRealGitcustom_branch3586427057/001' LsRemoteWithRealGitcustom_branch3586427057/001' 1/x64/bin/bash GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-04-04 GOMOD GOMODCACHE go env -json GO111MODULE ndor/bin/bash GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-02-03 GOMOD GOMODCACHE go eWit�� -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/1/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/1/artifacts --jq .artifacts[].name GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet env -json GO111MODULE not-exist-xyzzy.txt GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run download 1 --dir test-logs/run-1 GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env 4/001/inlined-a.md GO111MODULE cfg GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/12345/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12345/artifacts --jq .artifacts[].name GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run download 12345 --dir test-logs/run-12345 GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env 8390/001/stability-test.md GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/1234567890
    • Triggering command: /usr/bin/gh gh api repos/{owner}/{repo}/actions/runs/1234567890 --jq {databaseId: .id, number: .run_number, url: .html_url, status: .status, conclusion: .conclusion, workflowName: .name, workflowPath: .path, createdAt: .created_at, startedAt: .run_started_at, updatedAt: .updated_at, event: .event, headBranch: .head_branch, /tmp/go-build100GOINSECURE -trimpath 64/bin/go -p main -lang=go1.25 go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/12346/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12346/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/link GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/link env 8390/001/stability-test.md GO111MODULE rtcfg.link GOINSECURE GOMOD GOMODCACHE 3JL8ddMPoqaiAM618n/W7nDXlF-2wrJG-dwarf=false (http block)
    • Triggering command: /usr/bin/gh gh run download 12346 --dir test-logs/run-12346 GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/2/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/2/artifacts --jq .artifacts[].name GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run download 2 --dir test-logs/run-2 GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env 2444354449 GO111MODULE cfg GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/3/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/3/artifacts --jq .artifacts[].name GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env 2097910701 GO111MODULE ache/go/1.25.8/x--created GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run download 3 --dir test-logs/run-3 cfg 64/pkg/tool/linux_amd64/compile GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/compile env 4269648244 GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE util GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/4/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/4/artifacts --jq .artifacts[].name GO111MODULE x_amd64/compile GOINSECURE GOMOD GOMODCACHE x_amd64/compile env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run download 4 --dir test-logs/run-4 GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env 01/test1.md GO111MODULE k GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/5/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/5/artifacts --jq .artifacts[].name GO111MODULE x_amd64/link GOINSECURE GOMOD GOMODCACHE x_amd64/link env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE Bh/hKJC44cVKMHVntest@example.com (http block)
    • Triggering command: /usr/bin/gh gh run download 5 --dir test-logs/run-5 GO111MODULE er.test GOINSECURE GOMOD GOMODCACHE er.test 4562�� -json GO111MODULE cfg GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/workflows
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path /tmp/go-build100GOINSECURE -trimpath 64/bin/go -p github.com/charm-atomic -lang=go1.24 go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 100 GOMOD GOMODCACHE go env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 6 GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/contents/.github/workflows/shared/reporting.md
    • Triggering command: /tmp/go-build145620722/b404/cli.test /tmp/go-build145620722/b404/cli.test -test.testlogfile=/tmp/go-build145620722/b404/testlog.txt -test.paniconexit0 -test.v=true -test.parallel=4 -test.timeout=10m0s -test.run=^Test -test.short=true GOINSECURE GOMOD GOMODCACHE node /opt�� prettier --check 64/bin/go --ignore-path .prettierignore 64/bin/go go (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v0.47.4
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v0.47.4 --jq [.object.sha, .object.type] | @tsv --show-toplevel e/git /usr/bin/git xterm-color go /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git --get remote.origin.urrev-parse /usr/bin/docker git (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv pC4FS5t0C GO111MODULE tartedAt,updatedAt,event,headBranch,headSha,displayTitle GOINSECURE GOMOD GOMODCACHE rtcfg env g/gitutil/gitutil.go g/gitutil/gitutil_test.go /opt/hostedtoolcache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.2.3
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env b/workflows GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v2.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env b/workflows GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env b/workflows GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE x_amd64/link GOINSECURE GOMOD GOMODCACHE x_amd64/link (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v3.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v3.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env b/workflows GO111MODULE x_amd64/compile GOINSECURE GOMOD GOMODCACHE x_amd64/compile (http block)
  • https://api.github.com/repos/google-github-actions/auth/git/ref/tags/v2
    • Triggering command: /usr/bin/gh gh api /repos/google-github-actions/auth/git/ref/tags/v2 --jq [.object.sha, .object.type] | @tsv --show-toplevel /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linu^remote\..*\.gh-resolved$ /usr/bin/git /tmp/go-build145git -trimpath ps git rev-�� --show-toplevel ps /usr/bin/gh git -goversion 64/pkg/tool/linu--show-toplevel gh (http block)
  • https://api.github.com/repos/nonexistent/action/git/ref/tags/v999.999.999
    • Triggering command: /usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq [.object.sha, .object.type] | @tsv DefaultBranchFromLsRemoteWithRealGitmain_branch2279005788/001' DefaultBranchFromLsRemoteWithRealGitmain_branch2279005788/001' ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE es.test GOINSECURE GOMOD GOMODCACHE es.test (http block)
  • https://api.github.com/repos/nonexistent/repo/actions/runs/12345
    • Triggering command: /usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/vet env -json GO111MODULE cfg GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/owner/repo/actions/workflows
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/bin/go -d github.com/githu-atomic -lang=go1.25 go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/bin/go -p main -lang=go1.25 go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh workflow list --repo owner/repo --json name,path,state ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env agent-performance-analyzer.md GO111MODULE cfg GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet (http block)
  • https://api.github.com/repos/test-owner/test-repo/actions/secrets
    • Triggering command: /usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name /tmp/go-build100GOINSECURE -trimpath 64/bin/go -d main -lang=go1.25 go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/test/repo
    • Triggering command: /usr/bin/gh gh api /repos/test/repo --jq .default_branch -json GO111MODULE cfg GOINSECURE GOMOD GOMODCACHE go env 5927-35634/test-952720695/.github/workflows GO111MODULE 1/x64/bin/node GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/xREDACTED (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI linked an issue May 4, 2026 that may be closed by this pull request
Copilot AI and others added 2 commits May 4, 2026 07:56
Agent-Logs-Url: https://github.com/github/gh-aw/sessions/0d85d61d-8890-4bb1-998c-a93dca798c47

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
…TRUCTION

The codex engine now uses the codex_harness.cjs which passes the prompt via
--prompt-file instead of the $INSTRUCTION shell variable. Update the integration
test to match the current behavior.

Fixes CI failure in 'Integration: Workflow Features' job.

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/0d85d61d-8890-4bb1-998c-a93dca798c47

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix critical issues in Workflow Health Dashboard Fix stale $INSTRUCTION assertion in TestEngineArgsIntegrationCodex May 4, 2026
Copilot AI requested a review from gh-aw-bot May 4, 2026 08:17
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 4, 2026

Hey @copilot-swe-agent 👋 — great work tracking down the stale $INSTRUCTION assertion and updating the test to match the current --prompt-file harness behaviour. The fix to engine_args_integration_test.go is clear, well-described, and directly addresses the CI failure.

One thing stands out that would make this easier to review:

  • Unfocused diff — the PR bundles two action_pins.json data file updates (pkg/actionpins/data/action_pins.json and pkg/workflow/data/action_pins.json, +15 lines each) alongside the test fix. These appear unrelated to the stale-assertion problem. Separating them into their own PR (or explaining their connection in the body) would help reviewers validate each change independently.

If you'd like a hand splitting this up, here's a ready-to-use agent prompt:

Split PR #30100 in the github/gh-aw repository into two focused PRs:
1. The test fix only: update pkg/workflow/engine_args_integration_test.go to replace the stale $INSTRUCTION assertion with the --prompt-file assertion (the change already present in the branch).
2. The data file updates only: the additions to pkg/actionpins/data/action_pins.json and pkg/workflow/data/action_pins.json — include a description explaining what triggered these pin updates and why they are safe to merge.
Ensure each PR has a clear title and body describing its specific purpose.

Generated by Contribution Check · ● 1.5M ·

@pelikhan pelikhan marked this pull request as ready for review May 4, 2026 13:46
Copilot AI review requested due to automatic review settings May 4, 2026 13:46
@pelikhan pelikhan merged commit d9702c0 into main May 4, 2026
19 checks passed
@pelikhan pelikhan deleted the copilot/aw-fix-workflow-health-issues branch May 4, 2026 13:46
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a failing Codex engine args integration test by updating the assertion to match the current Codex harness behavior (prompt passed via --prompt-file rather than "$INSTRUCTION"), and refreshes embedded action pin data with additional major-version pins.

Changes:

  • Update TestEngineArgsIntegrationCodex to assert --custom-flag value appears before --prompt-file in the compiled lock YAML.
  • Add embedded action pin entries for common major-version action refs (actions/cache/save@v4, actions/checkout@v4, actions/setup-python@v5) in both pin datasets.
Show a summary per file
File Description
pkg/workflow/engine_args_integration_test.go Updates Codex integration test to assert args ordering relative to --prompt-file (current harness behavior).
pkg/workflow/data/action_pins.json Adds additional embedded pins for major-version action references used by workflows/compilation.
pkg/actionpins/data/action_pins.json Mirrors the embedded pin updates for the actionpins package’s resolver dataset.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 3/3 changed files
  • Comments generated: 1

Comment on lines +41 to +45
"actions/cache/save@v4": {
"repo": "actions/cache/save",
"version": "v4",
"sha": "0057852bfaa89a56745cba8c7296529d2fc39830"
},
@github-actions github-actions Bot mentioned this pull request May 4, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 4, 2026

🧪 Test Quality Sentinel Report

Test Quality Score: 100/100

Excellent test quality

Metric Value
New/modified tests analyzed 1
✅ Design tests (behavioral contracts) 1 (100%)
⚠️ Implementation tests (low value) 0 (0%)
Tests with error/edge cases 1 (100%)
Duplicate test clusters 0
Test inflation detected No
🚨 Coding-guideline violations None

Test Classification Details

Test File Classification Issues Detected
TestEngineArgsIntegrationCodex pkg/workflow/engine_args_integration_test.go ✅ Design None

Analysis

This PR fixes a stale assertion in TestEngineArgsIntegrationCodex. The old code checked for a $INSTRUCTION string that no longer appears in the codex engine compiled output; the fix replaces it with assertions that verify --custom-flag value appears before --prompt-file — the correct behavioral contract for codex engine arg injection.

Build tag: ✅ File starts with //go:build integration on line 1.

Design invariant enforced: The compiled workflow YAML for codex engine must include user-supplied args in the correct position (before --prompt-file). This is an observable output contract that users of the compiler depend on.

What would break if deleted?: A regression in codex engine arg positioning would go undetected.

No red flags detected: no mock libraries, no missing build tags, no assertion-message violations, no test inflation.


Language Support

Tests analyzed:

  • 🐹 Go (*_test.go): 1 test — integration (//go:build integration)
  • 🟨 JavaScript (*.test.cjs, *.test.js): 0 tests

Verdict

Check passed. 0% of new/modified tests are implementation tests (threshold: 30%).

📖 Understanding Test Classifications

Design Tests (High Value) verify what the system does:

  • Assert on observable outputs, return values, or state changes
  • Cover error paths and boundary conditions
  • Would catch a behavioral regression if deleted
  • Remain valid even after internal refactoring

Implementation Tests (Low Value) verify how the system does it:

  • Assert on internal function calls (mocking internals)
  • Only test the happy path with typical inputs
  • Break during legitimate refactoring even when behavior is correct
  • Give false assurance: they pass even when the system is wrong

Goal: Shift toward tests that describe the system's behavioral contract — the promises it makes to its users and collaborators.

References: §25322680678

🧪 Test quality analysis by Test Quality Sentinel · ● 572.2K ·

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Test Quality Sentinel: 100/100. Test quality is excellent — 0% of new/modified tests are implementation tests (threshold: 30%). The fix correctly replaces a stale $INSTRUCTION assertion with a proper behavioral contract check for codex engine arg positioning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aw] Workflow Health Dashboard — 2026-05-04

4 participants