Skip to content

fix: surface ReadDir error in detectFirewallAuditArtifacts instead of silently returning partial results#29724

Merged
pelikhan merged 2 commits intomainfrom
copilot/fix-detect-firewall-audit-artifacts
May 2, 2026
Merged

fix: surface ReadDir error in detectFirewallAuditArtifacts instead of silently returning partial results#29724
pelikhan merged 2 commits intomainfrom
copilot/fix-detect-firewall-audit-artifacts

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 2, 2026

detectFirewallAuditArtifacts swallowed os.ReadDir failures in the legacy artifact scan (step 4), silently returning whatever manifest/audit paths were found in steps 1–3. For runs where the legacy artifact was the only artifact, this produced empty results with no indication of why — masking the root cause from analyzeFirewallPolicy callers.

Changes

  • Signature changedetectFirewallAuditArtifacts now returns (manifestPath, auditJSONLPath string, err error); the ReadDir failure is wrapped with path context and propagated
  • Caller updateanalyzeFirewallPolicy propagates the new error rather than treating a detection failure as "no artifacts found"
  • Tests — all existing call sites updated; new "unreadable run directory returns error" sub-test added to TestDetectFirewallAuditArtifacts
// Before
func detectFirewallAuditArtifacts(runDir string) (manifestPath, auditJSONLPath string) {
    entries, err := os.ReadDir(runDir)
    if err != nil {
        return // silent partial return
    }
    ...
}

// After
func detectFirewallAuditArtifacts(runDir string) (manifestPath, auditJSONLPath string, err error) {
    entries, readErr := os.ReadDir(runDir)
    if readErr != nil {
        err = fmt.Errorf("detectFirewallAuditArtifacts: reading run dir %s: %w", runDir, readErr)
        return
    }
    ...
}

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh gh repo view --json owner,name --jq .owner.login + "/" + .name ache/go/1.25.8/x64/bin/go **/*.ts **/*.json --ignore-path go list�� -mod=readonly -f ache/go/1.25.8/x64/bin/go -- unsafe 64/bin/go go (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOWORK 64/bin/go ger.test 6263�� -x c ache/go/1.25.8/x64/bin/go - GO111MODULE 64/bin/go go (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOWORK 64/bin/go /opt/hostedtoolcache/go/1.25.8/x-goversion -o /tmp/go-build2850543849/b404/_pk-c=4 -trimpath .cfg -p main -lang=go1.25 go (http block)
  • https://api.github.com/orgs/test-owner/actions/secrets
    • Triggering command: /usr/bin/gh gh api /orgs/test-owner/actions/secrets --jq .secrets[].name -c=4 -nolocalimports -importcfg /tmp/go-build1626397822/b434/importcfg -pack /home/REDACTED/work/gh-aw/gh-aw/pkg/repoutil/repoutil.go /home/REDACTED/work/gh-aw/gh-aw/pkg/repoutil/repoutil_test.go env -json GO111MODULE tions/setup/js/node_modules/.binGOMODCACHE GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/actions/ai-inference/git/ref/tags/v1
    • Triggering command: /usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq [.object.sha, .object.type] | @tsv view owner/repo /usr/bin/git 74/001/test-emptgit **/*.cjs ck git rev-�� --show-toplevel UB4Z85HNraJd7B2VBq/aMwv_nfK75JUC-test.v=true /usr/bin/infocmp --format=%T /sys/fs/cgroup g_.a infocmp (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v3
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq [.object.sha, .object.type] | @tsv --show-toplevel go /usr/bin/gh -json GO111MODULE 64/bin/go gh api ithub-script/git/ref/tags/v9 --jq bject.type] | @tsv Gitbranch_with_hgit Gitbranch_with_hrev-parse 64/bin/go git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/go/1.25.8/xTest User /usr/bin/git -json GO111MODULE 1/x64/bin/node git rev-�� --show-toplevel go /usr/bin/git SameOutput367079sed (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/go/1.25.8/x--name-only /usr/bin/git 2555888094 GO111MODULE 64/pkg/tool/linu--show-toplevel git rev-�� --show-toplevel 64/pkg/tool/linux_amd64/compile /usr/bin/git g_.a GO111MODULE 6397822/b398/vet.cfg git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v5
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv 4551-56854/test-.artifacts[].name GO111MODULE 1/x64/bin/node GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/xowner/repo t-ha�� vaScript1242755574/001/test-empty-frontmatter.md GO111MODULE e/git-receive-pack GOINSECURE GOMOD GOMODCACHE (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv --show-toplevel go /usr/bin/git Onlymin-integritgit GO111MODULE .cfg git rev-�� --show-toplevel ache/go/1.25.8/x64/pkg/tool/linux_amd64/link /usr/bin/git 6397822/b420/filgit GO111MODULE e/git git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv --show-toplevel gh /usr/bin/git runs/20260502-09git s/5/artifacts /usr/bin/git git rev-�� ref/tags/v1 git sv GOMODCACHE go /tmp/go-build162--show-toplevel git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE /usr/local/sbin/bash GOINSECURE GOMOD GOMODCACHE bash --no�� licyTrustedUsersExpressionCompilnonexistent/repo GOPROXY /opt/hostedtoolcache/go/1.25.8/xstatus,conclusion GOSUMDB GOWORK 64/bin/go /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linutest@example.com (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv /tmp/go-build1626397822/b423/_pkg_.a l ache/go/1.25.8/x64/pkg/tool/linux_amd64/link -p github.com/githurev-parse -lang=go1.25 ache/go/1.25.8/x64/pkg/tool/linux_amd64/link /tmp�� 6397822/b459/tty.test -goversion 1/x64/bin/node -c=4 -nolocalimports -importcfg 1/x64/bin/node (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv 6397822/b466/_pkg_.a config 6397822/b466=> remote.origin.urgit GO111MODULE 64/bin/go git -C ZFWo/sMr_w3qRcZY0wv0fZFWo remote /opt/hostedtoolcache/node/24.14.1/x64/bin/node -json GO111MODULE ache/go/1.25.8/x--show-toplevel node (http block)
  • https://api.github.com/repos/actions/github-script/git/ref/tags/v8
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq [.object.sha, .object.type] | @tsv --show-toplevel go /usr/bin/git SameOutput367079head =main /opt/hostedtoolc/tmp/gh-aw/aw-feature-branch.patch git rev-�� --show-toplevel go /usr/bin/git /ref/tags/v9 6397822/b413/_terev-parse sv git (http block)
  • https://api.github.com/repos/actions/github-script/git/ref/tags/v9
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv 738118f635fe1b35GOINSECURE GO111MODULE x_amd64/compile GOINSECURE GOMOD GOMODCACHE x_amd64/compile env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/actions/setup-go/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel NKQxPfr/feDwSjXStImIOC-NY5wN /usr/bin/git -json GO111MODULE 1/x64/bin/bash git rev-�� --show-toplevel node /usr/bin/git prettier --check ache/go/1.25.8/x--show-toplevel git (http block)
  • https://api.github.com/repos/actions/setup-node/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv runs/20260502-094551-56854/test-2555888094 remote.origin.url 6397822/b466/_pkg_.a -json GO111MODULE ache/go/1.25.8/x--show-toplevel node /tmp�� /tmp/TestHashStability_SameInputSameOutput3670792216/001/stability-test.md go ache/node/24.14.1/x64/bin/node l && debian-sa1 git GO111MODULE ache/go/1.25.8/x--show-toplevel ache/node/24.14.1/x64/bin/node (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv GOMODCACHE o fix."; \ exit/tmp/go-build1626397822/b115/vet.cfg /usr/bin/git -json GO111MODULE ache/go/1.25.8/x--show-toplevel git rev-�� --show-toplevel node /usr/bin/gh e-analyzer.md --check ache/go/1.25.8/x--show-toplevel gh (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/go/1.25.8/xremote /usr/bin/git -json GO111MODULE 1/x64/bin/node git rev-�� --show-toplevel go /usr/bin/git vaScript12427555git GO111MODULE 64/pkg/tool/linuHEAD git (http block)
  • https://api.github.com/repos/actions/setup-node/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv ring655294105/001/test1.md '/tmp/TestParseDefaultBranchFromLsRemoteWithRealGitbranch_with_hyphen673776607/001' /usr/sbin/bash GOINSECURE GOMOD GOMODCACHE bash --no�� licyTrustedUsersExpressionCompiledOutput3835800866/001 GOPROXY /usr/bin/git GOSUMDB GOWORK 64/bin/go git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv --show-toplevel GOPROXY 6397822/b456/timeutil.test GOSUMDB GOWORK 64/bin/go 6397822/b456/timeutil.test e=/t�� t0 /tmp/TestParseDefaultBranchFromLsRemoteWithRealGitmaster_branch3-ifaceassert (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv sistency_InlinedImports2752181805/001/noflag-a.m--workflow -trimpath /usr/bin/git -p main -lang=go1.25 git rev-�� runs/20260502-094551-56854/test-2555888094 -dwarf=false ache/node/24.14.1/x64/bin/node s/test.md -c=4 -nolocalimports infocmp (http block)
  • https://api.github.com/repos/actions/upload-artifact/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/upload-artifact/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv /tmp/TestGuardPolicyBlockedUsersApprovalLabelsCompiledOutput3105096800/001 remote /opt/hostedtoolcache/node/24.14.1/x64/bin/node -json GO111MODULE 1/x64/lib/node_m--show-toplevel node /tmp�� /tmp/TestHashConsistency_GoAndJavaScript1242755574/001/test-frontmatter-with-nested-objects.md go /usr/bin/git 7738a5bf9e8d9cbcgit GO111MODULE 64/pkg/tool/linu--show-toplevel git (http block)
  • https://api.github.com/repos/aws-actions/configure-aws-credentials/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/aws-actions/configure-aws-credentials/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet /usr/bin/git 2555888094 GO111MODULE /opt/hostedtoolc--show-toplevel git rev-�� --show-toplevel go /usr/bin/git -json GO111MODULE 64/pkg/tool/linu-m git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/aws-actions/configure-aws-credentials/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/go/1.25.8/xrev-parse /usr/bin/git 4551-56854/test-git GO111MODULE /opt/hostedtoolc--show-toplevel git rev-�� --show-toplevel go /usr/bin/git /v3.0.0 GO111MODULE sv git (http block)
  • https://api.github.com/repos/azure/login/git/ref/tags/v2
    • Triggering command: /usr/bin/gh gh api /repos/azure/login/git/ref/tags/v2 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/go/1.25.8/x64/pkg/tool/linux_amd64/compile /usr/bin/git 4551-56854/test-git stmain.go 64/pkg/tool/linu--show-toplevel git rev-�� --show-toplevel 64/pkg/tool/linux_amd64/vet /usr/bin/git 6397822/b438/semsed GO111MODULE 6397822/b438/imps/.*"branch"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p git (http block)
  • https://api.github.com/repos/docker/login-action/git/ref/tags/v3
    • Triggering command: /usr/bin/gh gh api /repos/docker/login-action/git/ref/tags/v3 --jq [.object.sha, .object.type] | @tsv f/tags/v4 Wo/bJFsyISU_LIYu-f sv 2555888094 GO111MODULE 1/x64/bin/node git rev-�� --show-toplevel go /usr/bin/git SameOutput367079git GO111MODULE 64/pkg/tool/linux_amd64/vet git (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v0.1.2
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0.1.2 --jq [.object.sha, .object.type] | @tsv xterm-color go /usr/bin/git -json GO111MODULE ache/go/1.25.8/x--show-toplevel git rev-�� --show-toplevel sh /usr/bin/git 92762821/001 GOPROXY At,event,headBra--show-toplevel git (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv /tmp/TestGuardPolicyTrustedUsersRequiresMinIntegrity2668252880/001 rev-parse /usr/bin/git -json GO111MODULE ache/go/1.25.8/x--show-toplevel git rev-�� --show-toplevel resolved$ /usr/bin/git 0b8b1dfd13b51876git GO111MODULE 64/pkg/tool/linu--show-toplevel git (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.2.3
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv /tmp/TestGuardPolicyTrustedUsersExpressionCompiledOutput3835800866/001 remote (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-04-25 GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linu/tmp/TestParseDefaultBranchFromLsRemoteWithRealGitbranch_with_hyphen673776607/002/work env add-source-path-3031667958/.github/workflows GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-04-02 GOMOD GOMODCACHE x_amd64/compile env 0543849/b423/_pkg_.a GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-02-01 GOMOD GOMODCACHE BW/Vpa8Q5oQtl-xBremote.origin.url env 0543849/b436/_pkg_.a GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/1/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/1/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/link GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/link /hom�� util.test **/*.cjs ortcfg.link **/*.json --ignore-path ../../../.pretti--show-toplevel 6lOae4WsPt2nvzZ6yM/kanFXMcb9Ib9j3Zv3h0y/rLzSlIqW0GXogzJW_IN_ (http block)
    • Triggering command: /usr/bin/gh gh run download 1 --dir test-logs/run-1 GOPROXY 64/pkg/tool/linux_amd64/vet GOSUMDB GOWORK 64/bin/go 64/pkg/tool/linux_amd64/vet -c "prettier" --check 'scripts/**/*.js' --ignore-path .prettierignoGOINSECURE bash ache/go/1.25.8/x64/bin/go --noprofile go 64/bin/go go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/12345/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12345/artifacts --jq .artifacts[].name GO111MODULE ntdrain.test GOINSECURE GOMOD GOMODCACHE ntdrain.test estl�� "prettier" --check '**/*.cjs' '*-p GOPROXY ache/go/1.25.8/x64/bin/go GOSUMDB GOWORK 64/bin/go go (http block)
    • Triggering command: /usr/bin/gh gh run download 12345 --dir test-logs/run-12345 ne_constants.go x_amd64/compile GOINSECURE GOMOD GOMODCACHE x_amd64/compile /hom�� plorer.md **/*.cjs ache/go/1.25.8/x64/bin/go **/*.json --ignore-path ../../../.pretti--show-toplevel go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/1234567890
    • Triggering command: /usr/bin/gh gh api repos/{owner}/{repo}/actions/runs/1234567890 --jq {databaseId: .id, number: .run_number, url: .html_url, status: .status, conclusion: .conclusion, workflowName: .name, workflowPath: .path, createdAt: .created_at, startedAt: .run_started_at, updatedAt: .updated_at, event: .event, headBranch: .head_branch, -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE tions/setup/node_modules/.bin/noGOMODCACHE GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/12346/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12346/artifacts --jq .artifacts[].name GO111MODULE aw.test GOINSECURE GOMOD GOMODCACHE aw.test 6263�� 416668117 GOPROXY x_amd64/compile GOSUMDB GOWORK 64/bin/go x_amd64/compile (http block)
    • Triggering command: /usr/bin/gh gh run download 12346 --dir test-logs/run-12346 GO111MODULE 64/pkg/tool/linux_amd64/compile GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/compile /hom�� g_.a **/*.cjs ache/go/1.25.8/x64/bin/go **/*.json tants ../../../.pretti--get go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/2/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/2/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/vet /hom�� 74/001/test-empty-frontmatter.md **/*.cjs x_amd64/vet **/*.json --ignore-path ../../../.pretti--show-toplevel x_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh run download 2 --dir test-logs/run-2 GOPROXY 64/pkg/tool/linux_amd64/link GOSUMDB GOWORK 64/bin/go 64/pkg/tool/linux_amd64/link -c 74/001/test-complex-frontmatter--errorsas bash 64/pkg/tool/linux_amd64/vet --noprofile go 64/bin/go 64/pkg/tool/linux_amd64/vet (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/3/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/3/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/compile GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/compile /hom�� g_.a **/*.cjs ache/go/1.25.8/x64/bin/go **/*.json --ignore-path ../../../.pretti--show-toplevel stat (http block)
    • Triggering command: /usr/bin/gh gh run download 3 --dir test-logs/run-3 lint:cjs 64/pkg/tool/linux_amd64/vet GOSUMDB GOWORK run-script/lib/node-gyp-bin/node/tmp/go-build1626397822/b452/testutil.test 64/pkg/tool/linux_amd64/vet -c "prettier" --check 'scripts/**/*-s bash ache/go/1.25.8/x64/bin/go --noprofile go 64/bin/go go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/4/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/4/artifacts --jq .artifacts[].name GO111MODULE x_amd64/compile GOINSECURE GOMOD GOMODCACHE x_amd64/compile /hom�� --check **/*.cjs ache/go/1.25.8/x64/bin/go **/*.json --ignore-path ../../../.pretti--show-toplevel stat (http block)
    • Triggering command: /usr/bin/gh gh run download 4 --dir test-logs/run-4 GOPROXY 64/pkg/tool/linux_amd64/link GOSUMDB GOWORK 64/bin/go 64/pkg/tool/linux_amd64/link -c eutil.test bash ortcfg.link --noprofile go 64/bin/go odp3IL84tZKq5R-8Dk/Exs2fuW_hbdM5nZZzdgr/Win8VWOQaRiefE3LJ-80 (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/5/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/5/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/link GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/link /hom�� 74/001/test-empty-frontmatter.md **/*.cjs ck **/*.json --ignore-path ../../../.prettierignore UB4Z85HNraJd7B2VBq/aMwv_nfK75JUC-test.v=true (http block)
    • Triggering command: /usr/bin/gh gh run download 5 --dir test-logs/run-5 GO111MODULE 64/pkg/tool/linux_amd64/compile GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/compile /opt�� 74/001/test-complex-frontmatter--p --check ck --ignore-path .prettierignore 64/bin/go go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/workflows
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE tions/setup/js/node_modules/.binGOMODCACHE GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 100 GOMOD GOMODCACHE 64/pkg/tool/linugit-upload-pack '/tmp/TestParseDefaultBranchFromLsRemoteWithRealGitbranch_with_hyphen673776607/001' env g_.a GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 6 GOMOD GOMODCACHE 64/pkg/tool/linu--json -c 3737334175 GOPROXY ache/go/1.25.8/x--created GOSUMDB GOWORK 64/bin/go /opt/hostedtoolcTest User (http block)
  • https://api.github.com/repos/github/gh-aw/contents/.github/workflows/shared/reporting.md
    • Triggering command: /tmp/go-build1626397822/b404/cli.test /tmp/go-build1626397822/b404/cli.test -test.testlogfile=/tmp/go-build1626397822/b404/testlog.txt -test.paniconexit0 -test.v=true -test.parallel=4 -test.timeout=10m0s -test.run=^Test -test.short=true GOINSECURE GOMOD GOMODCACHE go ules�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v0.47.4
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v0.47.4 --jq [.object.sha, .object.type] | @tsv etup-node/git/ref/tags/v6 go bject.type] | @tsv -json GO111MODULE eutil.test git rev-�� --show-toplevel eutil.test /usr/bin/git faultBranchFromLinfocmp faultBranchFromL-1 /opt/hostedtoolcxterm-color git (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv g_.a go ache/go/1.25.8/x64/bin/go -d er 64/bin/go go env _FNmUncPS GO111MODULE cal/bin/bash GOINSECURE GOMOD GOMODCACHE ortcfg (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.2.3
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go 9340�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v2.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE ache/go/1.25.8/x-lang=go1.25 GOINSECURE GOMOD GOMODCACHE go env Gitmaster_branchremote.origin.url Gitmaster_branch3928620234/001' 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv -c=4 -nolocalimports -importcfg /tmp/go-build1626397822/b440/importcfg -pack /home/REDACTED/work/gh-aw/gh-aw/pkg/sliceutil/sliceutil.go /home/REDACTED/work/gh-aw/gh-aw/pkg/sliceutil/sliceutil_test.go env 713404163/001 713404163/002/work 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv .js' --ignore-pa-c=4 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v3.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v3.0.0 --jq [.object.sha, .object.type] | @tsv .js' --ignore-path .prettierignoGOINSECURE GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE nZZzdgr/Win8VWOQaRiefE3LJ-80 env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/google-github-actions/auth/git/ref/tags/v2
    • Triggering command: /usr/bin/gh gh api /repos/google-github-actions/auth/git/ref/tags/v2 --jq [.object.sha, .object.type] | @tsv tags/v6 ache/go/1.25.8/xrev-parse sv 4551-56854/test-git GO111MODULE 1/x64/bin/node git rev-�� --show-toplevel go (http block)
  • https://api.github.com/repos/nonexistent/action/git/ref/tags/v999.999.999
    • Triggering command: /usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq [.object.sha, .object.type] | @tsv ./cmd/... ./pkg/... Name,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle -d GO111MODULE 64/bin/go go env efaultBranchFromLsRemoteWithRealGitbranch_with_hyphen673776607/001' efaultBranchFromLsRemoteWithRealGitbranch_with_hyphen673776607/001' ache/go/1.25.8/x64/bin/bash GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/nonexistent/repo/actions/runs/12345
    • Triggering command: /usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion GOSUMDB GOWORK 64/bin/go 64/pkg/tool/linutest@example.com /hom�� 494387744/001 scripts/**/*.js tartedAt,updatedAt,event,headBranch,headSha,displayTitle -d GO111MODULE 64/bin/go go (http block)
  • https://api.github.com/repos/owner/repo/actions/workflows
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ules/.bin/node GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ode GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh workflow list --repo owner/repo --json name,path,state ortcfg.link **/*.json --ignore-path ../../../.pretti--git-dir w5Iacm3KDPUrwDoD1g/r0m2lN7CnCbulsA8NlEQ/bZswKJCyremote.origin.url env sRemoteWithRealGitmain_branch4189340583/001 sRemoteWithRealGitmain_branch4189340583/002/work g_.a GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/test-owner/test-repo/actions/secrets
    • Triggering command: /usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/test/repo
    • Triggering command: /usr/bin/gh gh api /repos/test/repo --jq .default_branch 74/001/test-frontmatter-with-env-template-expressions.md make k tierignore go 64/bin/go go sRem�� -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Change function signature to (manifestPath, auditJSONLPath string, err error)
so that os.ReadDir failures in the legacy artifact scan are propagated
instead of silently returning partial results.

Update analyzeFirewallPolicy to propagate the new error, update all
existing test call sites, and add a test for unreadable run directories.

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/9977db3b-ab44-4e94-b414-fdc09bf1c366

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix silent return on ReadDir error in detectFirewallAuditArtifacts fix: surface ReadDir error in detectFirewallAuditArtifacts instead of silently returning partial results May 2, 2026
Copilot AI requested a review from gh-aw-bot May 2, 2026 09:47
@pelikhan pelikhan marked this pull request as ready for review May 2, 2026 10:00
Copilot AI review requested due to automatic review settings May 2, 2026 10:00
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR ensures unreadable run directories don’t silently yield empty/partial firewall policy artifact detection results by propagating os.ReadDir failures up to callers.

Changes:

  • Updated detectFirewallAuditArtifacts to return (manifestPath, auditJSONLPath, err) and propagate ReadDir failures with additional context.
  • Updated analyzeFirewallPolicy to return an error when artifact detection fails (instead of treating it as “no artifacts”).
  • Updated tests to handle the new signature and added a subtest covering unreadable run directories.
Show a summary per file
File Description
pkg/cli/firewall_policy.go Adds an error return to artifact detection and propagates detection failures from analyzeFirewallPolicy.
pkg/cli/firewall_policy_test.go Updates call sites for the new signature and adds a permission/error-path test.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 2

Comment on lines 410 to +414
// 1. sandbox/firewall/audit/ — primary path after flattenUnifiedArtifact strips the /tmp/gh-aw/ prefix
// 2. agent/sandbox/firewall/audit/ — non-flattened unified agent artifact (new structure)
// 3. agent/tmp/gh-aw/sandbox/firewall/audit/ — non-flattened unified agent artifact (old structure)
// 4. firewall-audit*/ — legacy separate firewall-audit-logs artifact (backward compat)
func detectFirewallAuditArtifacts(runDir string) (manifestPath, auditJSONLPath string) {
func detectFirewallAuditArtifacts(runDir string) (manifestPath, auditJSONLPath string, err error) {
Comment on lines +678 to +687
t.Run("unreadable run directory returns error", func(t *testing.T) {
if os.Getuid() == 0 {
t.Skip("root can read any directory; skipping permission test")
}
dir := t.TempDir()
require.NoError(t, os.Chmod(dir, 0000), "Should be able to remove read permission")
t.Cleanup(func() { _ = os.Chmod(dir, 0755) })

_, _, err := detectFirewallAuditArtifacts(dir)
require.Error(t, err, "Should return an error when run dir is unreadable")
@github-actions github-actions Bot mentioned this pull request May 2, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 2, 2026

🧪 Test Quality Sentinel Report

Test Quality Score: 82/100

Excellent test quality

Metric Value
New/modified tests analyzed 8 sub-tests in TestDetectFirewallAuditArtifacts
✅ Design tests (behavioral contracts) 8 (100%)
⚠️ Implementation tests (low value) 0 (0%)
Tests with error/edge cases 8 (100%)
Duplicate test clusters 0
Test inflation detected ⚠️ Yes (29 test lines / 10 production lines = 2.9:1)
🚨 Coding-guideline violations None

Test Classification Details

View per-test breakdown (8 sub-tests)
Sub-test File Classification Notes
agent/sandbox/firewall/audit (happy path) pkg/cli/firewall_policy_test.go ✅ Design Verifies no error + correct paths
agent/sandbox/firewall/audit (non-flattened new) pkg/cli/firewall_policy_test.go ✅ Design Verifies no error + correct paths
agent/tmp/gh-aw/sandbox/... (old structure) pkg/cli/firewall_policy_test.go ✅ Design Verifies no error + correct paths
agent-artifacts/sandbox/... pkg/cli/firewall_policy_test.go ✅ Design Verifies no error + correct paths
prefixed-agent/sandbox/... pkg/cli/firewall_policy_test.go ✅ Design Verifies no error + correct paths
firewall-audit-logs (legacy) pkg/cli/firewall_policy_test.go ✅ Design Verifies no error + correct paths
no artifacts pkg/cli/firewall_policy_test.go ✅ Design Verifies no error + empty results
unreadable run directory returns error (new) pkg/cli/firewall_policy_test.go:669 ✅ Design Error-path test; asserts error is returned and identifies function

Flagged Tests — Requires Review

⚠️ Test Inflation: firewall_policy_test.go

Classification: Minor concern (not a guideline violation)
Issue: 29 lines added to the test file vs. 10 lines added to the production file (2.9:1 ratio, threshold: 2:1). The inflation is largely mechanical — 7 existing sub-tests each gained one err variable capture and one require.NoError assertion to match the new 3-value return signature. This is inherent to the change and does not reflect low-quality padding.
Verdict: Acceptable in context; no action required.


Language Support

Tests analyzed:

  • 🐹 Go (*_test.go): 8 sub-tests — unit (//go:build !integration) ✅
  • 🟨 JavaScript (*.test.cjs, *.test.js): 0 tests changed

Verdict

Check passed. 0% of new/modified tests are implementation tests (threshold: 30%). The key addition is the new "unreadable run directory returns error" sub-test, which directly exercises the behavioral contract introduced by this PR — that ReadDir failures are surfaced as errors rather than silently swallowed. All assertion calls include descriptive messages. No mock libraries used. Build tag present.


📖 Understanding Test Classifications

Design Tests (High Value) verify what the system does:

  • Assert on observable outputs, return values, or state changes
  • Cover error paths and boundary conditions
  • Would catch a behavioral regression if deleted
  • Remain valid even after internal refactoring

Implementation Tests (Low Value) verify how the system does it:

  • Assert on internal function calls (mocking internals)
  • Only test the happy path with typical inputs
  • Break during legitimate refactoring even when behavior is correct
  • Give false assurance: they pass even when the system is wrong

Goal: Shift toward tests that describe the system's behavioral contract — the promises it makes to its users and collaborators.

References: §25249428087

🧪 Test quality analysis by Test Quality Sentinel · ● 608.7K ·

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Test Quality Sentinel: 82/100. Test quality is excellent — 0% of new tests are implementation tests (threshold: 30%). The new error-path sub-test directly verifies the behavioral contract introduced by this PR.

@pelikhan pelikhan merged commit cfec195 into main May 2, 2026
39 of 41 checks passed
@pelikhan pelikhan deleted the copilot/fix-detect-firewall-audit-artifacts branch May 2, 2026 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: detectFirewallAuditArtifacts silently returns partial results on ReadDir error

4 participants