Skip to content

Fix test failures after action pin updates#25745

Merged
pelikhan merged 1 commit intomainfrom
copilot/fix-tests-build-wasm
Apr 11, 2026
Merged

Fix test failures after action pin updates#25745
pelikhan merged 1 commit intomainfrom
copilot/fix-tests-build-wasm

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 11, 2026

Commit 66fb0c5 updated action pins (actions/github-script v9→v9.0.0, actions/upload-artifact v7→v7.0.1) but left tests with stale SHA expectations, plus exposed a latent bug in TestPreActivationJob.

TestPreActivationJob panic — slice bounds out of range

strings.Index(lockContentStr, "agent:") matched the container image ref gh-aw-firewall/agent:0.25.18 in the YAML header before the actual job key, causing activationIdx > agentIdx.

// Before: matches container image reference in header comment
agentIdx := strings.Index(lockContentStr, "agent:")

// After: matches only the YAML job key
agentIdx := strings.Index(lockContentStr, "\n  agent:\n")

Stale SHA expectations across 11 test files

  • actions/github-script: 3a2844b7e...d746ffe35...
  • actions/upload-artifact: bbbca2dda...043fb46d1...
  • TestGetActionPinWithData_V7ExactMatch updated for v7.0.1 pin

Wasm golden files regenerated

basic-copilot.golden and with-imports.golden updated via make update-wasm-golden.

TestLogsJSON* skip conditions

Added HTTP 403 to skip conditions — these tests hit the real GitHub API and fail in environments without API access.

… in logs tests

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/edcf2728-6284-4370-b05f-957d5021be23

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan pelikhan marked this pull request as ready for review April 11, 2026 02:36
Copilot AI review requested due to automatic review settings April 11, 2026 02:36
@pelikhan pelikhan merged commit 936d8ce into main Apr 11, 2026
50 of 54 checks passed
@pelikhan pelikhan deleted the copilot/fix-tests-build-wasm branch April 11, 2026 02:36
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates test fixtures and expectations to align with recent GitHub Action pin changes, and hardens TestPreActivationJob string matching to avoid a slice-bounds panic caused by accidentally matching container image references.

Changes:

  • Updated test expectations for actions/github-script and actions/upload-artifact pinned SHAs (including v7→v7.0.1 resolution).
  • Regenerated Wasm compile golden files to reflect updated pins (and now-digest-pinned container image refs in the generated workflow output).
  • Fixed TestPreActivationJob to locate YAML job keys more reliably by matching indented job headers (avoiding false matches like .../agent:0.25.18), and added missing “not found” guards in one test case.
Show a summary per file
File Description
pkg/workflow/threat_detection_test.go Updates expected upload-artifact pin SHA in detection log step test.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/with-imports.golden Regenerated golden output with updated action pins and container digests.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/basic-copilot.golden Regenerated golden output with updated action pins and container digests.
pkg/workflow/stop_time_check_job_test.go Prevents false agent: matches by searching for indented job keys; adds missing index-not-found guard in one slice operation.
pkg/workflow/safe_output_helpers_test.go Updates expected github-script pin SHA occurrences.
pkg/workflow/redact_secrets_test.go Updates expected github-script pin SHA in redaction step assertion.
pkg/workflow/pr_ready_for_review_checkout_test.go Updates expected github-script pin SHA in PR checkout assertions.
pkg/workflow/pr_checkout_test.go Updates expected github-script pin SHA in PR checkout assertions.
pkg/workflow/mcp_logs_upload_test.go Updates expected upload-artifact pin SHA in MCP logs upload tests.
pkg/workflow/compute_text_lazy_test.go Updates expected github-script pin SHA in compiled workflow assertion.
pkg/workflow/compile_outputs_pr_test.go Updates expected github-script pin SHA in safe_outputs job assertion.
pkg/workflow/compile_outputs_label_test.go Updates expected github-script pin SHA in safe_outputs job assertion.
pkg/workflow/compile_outputs_issue_test.go Updates expected github-script pin SHA in safe_outputs job assertion.
pkg/workflow/compile_outputs_comment_test.go Updates expected github-script pin SHA in safe_outputs job assertion.
pkg/workflow/action_pins_test.go Updates expected v7 SHA for upload-artifact (v7 resolves to v7.0.1 pin).
pkg/cli/logs_json_stderr_order_test.go Skips JSON/stderr ordering tests when GitHub API returns HTTP 403 (no API access).
pkg/cli/logs_ci_scenario_test.go Skips “no runs” JSON test when GitHub API returns HTTP 403 (no API access).

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 17/17 changed files
  • Comments generated: 0

@github-actions github-actions bot mentioned this pull request Apr 11, 2026
@github-actions
Copy link
Copy Markdown
Contributor

🧪 Test Quality Sentinel Report

Test Quality Score: 92/100

Excellent test quality

Metric Value
New/modified tests analyzed 16
✅ Design tests (behavioral contracts) 16 (100%)
⚠️ Implementation tests (low value) 0 (0%)
Tests with error/edge cases 16 (100%)
Duplicate test clusters 1 (pre-existing)
Test inflation detected No
🚨 Coding-guideline violations 0

Test Classification Details

📋 All 16 modified tests (click to expand)
Test File Classification Issues Detected
TestLogsJSONOutputWithNoRuns pkg/cli/logs_ci_scenario_test.go:67 ✅ Design Added HTTP 403 skip — expands error resilience
TestLogsJSONOutputBeforeStderr pkg/cli/logs_json_stderr_order_test.go:75 ✅ Design Added HTTP 403 skip — expands error resilience
TestLogsJSONAndStderrRedirected pkg/cli/logs_json_stderr_order_test.go:202 ✅ Design Added HTTP 403 skip — expands error resilience
TestGetActionPinWithData_V7ExactMatch pkg/workflow/action_pins_test.go:910 ✅ Design Updated SHA bbbca2d → 043fb46 (v7.0.1 pin)
TestCompileWithSafeOutputsComment pkg/workflow/compile_outputs_comment_test.go:277 ✅ Design Updated github-script SHA 3a2844b → d746ffe
TestCompileWithCreateIssue pkg/workflow/compile_outputs_issue_test.go:391 ✅ Design Updated github-script SHA 3a2844b → d746ffe
TestCompileWithSafeOutputsLabel pkg/workflow/compile_outputs_label_test.go:137 ✅ Design Updated github-script SHA 3a2844b → d746ffe
TestCompileWithSafeOutputsPR pkg/workflow/compile_outputs_pr_test.go ✅ Design Updated github-script SHA 3a2844b → d746ffe
(compute_text_lazy test) pkg/workflow/compute_text_lazy_test.go ✅ Design SHA pin update
(mcp_logs_upload tests) pkg/workflow/mcp_logs_upload_test.go ✅ Design SHA pin updates (4 additions)
(pr_checkout tests) pkg/workflow/pr_checkout_test.go ✅ Design SHA pin updates (integration build tag ✅)
(pr_ready_for_review tests) pkg/workflow/pr_ready_for_review_checkout_test.go ✅ Design SHA pin updates (integration build tag ✅)
(redact_secrets tests) pkg/workflow/redact_secrets_test.go ✅ Design SHA pin updates
(safe_output_helpers tests) pkg/workflow/safe_output_helpers_test.go ✅ Design SHA pin updates
TestPreActivationJob pkg/workflow/stop_time_check_job_test.go ✅ Design Multiple sub-tests updated (10 additions, 4 deletions)
TestBuildUploadDetectionLogStep pkg/workflow/threat_detection_test.go:660 ✅ Design Updated upload-artifact SHA 3a2844b → d746ffe

Flagged Tests — Requires Review

No tests require review. The one item worth noting (not a failure condition):

📎 Repeated SHA assertion pattern — compile_outputs_{comment,issue,label,pr}_test.go

Classification: Pre-existing pattern, not introduced by this PR
Observation: Four separate test files each contain an identical strings.Contains(lockContent, "uses: actions/github-script@<SHA>") assertion with trivially different function names. This is a minor duplicate test cluster but it was present before this PR; the PR merely updated all four to the new SHA.
Suggestion: Consider extracting the common SHA assertion into a shared test helper to reduce the churn when pins are updated in the future (e.g., assertGitHubScriptSHA(t, lockContent, expectedSHA)). This would mean a future pin update only requires changing the value in one place.


Language Support

Tests analyzed:

  • 🐹 Go (*_test.go): 16 test functions — 14 unit (//go:build !integration), 2 integration (//go:build integration)
  • 🟨 JavaScript (*.test.cjs, *.test.js): 0 tests changed in this PR

i️ Two golden data files (pkg/workflow/testdata/TestWasmGolden_CompileFixtures/*.golden) were also updated to reflect new action SHA values. These are not test files but serve as expected output fixtures.


Verdict

Check passed. 0% of new tests are implementation tests (threshold: 30%). This PR is purely a maintenance update — updating pinned SHA hashes after upstream action releases (github-script@v9v9.0.0, upload-artifact@v7v7.0.1) and hardening 3 integration tests to also skip on HTTP 403 responses. All build tags are correct, no mock libraries are used, and every modified test enforces a genuine behavioral contract (the compiler must emit the correct pinned SHA in generated workflows).


📖 Understanding Test Classifications

Design Tests (High Value) verify what the system does:

  • Assert on observable outputs, return values, or state changes
  • Cover error paths and boundary conditions
  • Would catch a behavioral regression if deleted
  • Remain valid even after internal refactoring

Implementation Tests (Low Value) verify how the system does it:

  • Assert on internal function calls (mocking internals)
  • Only test the happy path with typical inputs
  • Break during legitimate refactoring even when behavior is correct
  • Give false assurance: they pass even when the system is wrong

Goal: Shift toward tests that describe the system's behavioral contract — the promises it makes to its users and collaborators.

🧪 Test quality analysis by Test Quality Sentinel · ● 1.5M ·

Copy link
Copy Markdown
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Test Quality Sentinel: 92/100. Test quality is excellent — 0% of new tests are implementation tests (threshold: 30%). All 16 modified test functions enforce genuine behavioral contracts; all build tags are correct; no mock libraries are used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants