Fix static-analysis scan pipeline short-circuiting and add scanner output completeness assertion

[Copilot]

The static-analysis compilation pipeline aborted prematurely when an early scanner (e.g., zizmor) returned a finding error, silently skipping 6 of 8 security scanners (poutine, runner-guard, syft, grype, yamllint, shellcheck). This caused security scan runs to complete successfully while masking findings from 75% of the configured toolchain.

Changes

• Non-short-circuiting batch scanner pipeline
  Refactored compileSpecificFiles and compileAllFilesInDirectory in pkg/cli/compile_pipeline.go to use runBatchExternalTools. Batch scanners now execute sequentially to completion regardless of individual tool findings or errors, accumulating failures to return only after post-processing and results output complete.

• Guaranteed scanner invocation logging
  Updated actionlint.go, syft.go, grype.go, and shellcheck.go to ensure every enabled tool emits an execution log line to stderr even when operating on zero lock files, container images, or script steps.

• Output completeness assertion step
  Added an Assert static analysis output completeness step to .github/workflows/static-analysis-report.md that checks compile-output.txt for output from all 8 tools and triggers a hard failure if any scanner produces zero output.

Implementation

// pkg/cli/compile_pipeline.go: execute all enabled scanners sequentially without early return
func runBatchExternalTools(
	ctx context.Context,
	config CompileConfig,
	opts batchToolsOptions,
	stats *CompilationStats,
	validationResults *[]ValidationResult,
) (strictGrantErr error, batchToolErr error) {
	if err := runBatchLinters(ctx, config, opts); err != nil && batchToolErr == nil {
		batchToolErr = err
	}
	if err := runBatchDirScanners(ctx, config, opts); err != nil && batchToolErr == nil {
		batchToolErr = err
	}
	sGrantErr, containerErr := runBatchContainerScanners(ctx, config, opts, stats, validationResults)
	if sGrantErr != nil && strictGrantErr == nil {
		strictGrantErr = sGrantErr
	}
	if containerErr != nil && batchToolErr == nil {
		batchToolErr = containerErr
	}
	if err := runBatchScriptLinters(ctx, config, opts); err != nil && batchToolErr == nil {
		batchToolErr = err
	}
	return strictGrantErr, batchToolErr
}

• Fixes [uk-ai-resilience] Static-analysis scan pipeline instrumentation gap: 6/8 tools produced zero output (Tier C) #58562

[pelikhan]

@copilot resolve the merge conflicts on this branch.

[Copilot]

🟡 Changes recommended

Moderate issues leave scanner completeness detection, empty-input logging, and regression coverage unreliable.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Prevents scanner failures from short-circuiting later static-analysis tools and adds output-completeness monitoring.

Changes:

• Runs batch scanners sequentially while preserving errors.
• Adds execution logging for empty scan inputs.
• Adds workflow completeness checks and regression coverage.

File summaries

File	Description	Review notes
pkg/cli/syft.go	Logs zero-image scans.	No unresolved comments.
pkg/cli/shellcheck.go	Refactors helpers and logs zero-step scans.	Nit (2 votes): A new comment names the wrong function and contains an empty priority list.
pkg/cli/grype.go	Logs zero-image scans.	No unresolved comments.
pkg/cli/compile_pipeline.go	Adds non-short-circuiting scanner orchestration.	Moderate (1 vote): Actionlint still emits no marker with zero lock files. Nit (1 vote): A helper comment names the wrong function.
pkg/cli/compile_external_tools_test.go	Adds orchestration regression coverage.	Moderate (2 votes): Empty inputs do not test continued execution after an early scanner error.
pkg/cli/actionlint.go	Adds actionlint execution logging.	No unresolved comments.
.github/workflows/static-analysis-report.md	Adds scanner-output completeness assertions.	Moderate (2 votes): The shellcheck substring can be satisfied by actionlint output; use scanner-specific markers and regenerate the lock file.
.github/workflows/static-analysis-report.lock.yml	Regenerates the compiled workflow.	Must be regenerated after correcting the completeness assertion.

Review details

Suppressed comments (1)

pkg/cli/compile_pipeline.go:499

• This comment names runBatchExternalTools, but it is attached to runBatchLinters; the actual orchestration function is declared later at line 646. Document the helper that follows here so generated documentation and code navigation are accurate.

// runBatchExternalTools executes all enabled batch analysis tools sequentially without short-circuiting
// when individual tools report findings or errors.

• Files reviewed: 8/8 changed files
• Comments generated: 4
• Review effort level: Balanced

[Copilot]

Fixed in 32f97f7: replaced the bare grep -qi "$tool" check with a per-scanner marker map (e.g. Running actionlint (, Running shellcheck on) so actionlint's "with shellcheck/pyflakes" message can no longer satisfy the dedicated shellcheck check. Verified locally that a log containing actionlint's message but missing the shellcheck marker now fails the assertion.

[Copilot]

Correction: the commit hash is ffd49a9, not 32f97f7.

[Copilot]

Fixed in 32f97f7: added package-level function-variable seams for all 9 batch scanner entry points and rewrote the test to make actionlint return an error, then assert all scanners still run in pipeline order and the first error is preserved. Confirmed this test fails if the pipeline is reverted to short-circuit on the first error.

[Copilot]

Correction: the commit hash is ffd49a9, not 32f97f7.

[Copilot]

Fixed in 32f97f7: removed the len(lockFiles) > 0 guards for all lock-file batch tools (actionlint, zizmor, syft, grype, grant, yamllint) and for the directory scanners (poutine, runner-guard), so they're always invoked when enabled. runBatchLockFileTool and RunShellcheckOnLockFilesAndResources now emit an explicit zero-input stderr marker before returning, so the completeness check can tell "ran with zero input" apart from "never invoked".

[Copilot]

Correction: the commit hash is ffd49a9, not 32f97f7.

[Copilot]

Fixed in 32f97f7: rewrote the doc comment on extractRunStepsFromJob to describe its actual behavior (job-level step extraction with the resolved default shell), removing the mislabeled/duplicated reference to extractRunStepsFromLockFile.

[Copilot]

Correction: the commit hash is ffd49a9, not 32f97f7.

[Copilot]

@copilot resolve the merge conflicts on this branch.

Resolved in 73bb262 — merged main, resolved the lock-file conflict in static-analysis-report.lock.yml by regenerating it with gh aw compile, and verified all 299 workflows compile and pass validation.

[gh-aw-bot]

pr-sous-chef

@copilot Please take the next forward-progress pass on this PR.

• A Copilot review is still requesting follow-up on scanner-output completeness assertions and regression coverage for continued execution after an early scanner error.
• Please address the outstanding review feedback on current HEAD, then run the pr-finisher skill and summarize exactly what changed.
• If one or more review comments are already fixed on HEAD, say that explicitly and resolve the related thread(s).

Generated by PR Sous Chef: https://github.com/github/gh-aw/actions/runs/33946898762

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 26.1 AIC · ⌖ 8.68 AIC · ⊞ 9.2K · ◷
_{Comment /souschef to run again}

[Copilot]

pr-sous-chef
```...

Addressed all four outstanding review threads in 32f97f7:

1. static-analysis-report.md completeness check: replaced the bare grep -qi "$tool" substring match with a per-scanner marker map (e.g. Running actionlint (, Running shellcheck on) so actionlint's "with shellcheck/pyflakes" summary can no longer satisfy the dedicated shellcheck check. Verified locally that a log missing the shellcheck marker (but containing actionlint's message) now correctly fails the assertion.
2. compile_external_tools_test.go: rewrote the test to inject an error from the first scanner (actionlint) via new package-level function-variable seams, and assert all 9 scanners still run in pipeline order with the first error preserved. Confirmed this test fails if the pipeline is reverted to short-circuit on the first error.
3. compile_pipeline.go:503: removed the len(lockFiles) > 0 guards that skipped invoking a tool (and its stderr marker) entirely when zero lock files were produced. runBatchLockFileTool and RunShellcheckOnLockFilesAndResources now emit an explicit zero-input marker so the completeness check distinguishes "ran with zero input" from "never invoked".
4. shellcheck.go:146-149: fixed the doc comment on extractRunStepsFromJob (it was mislabeled as documenting extractRunStepsFromLockFile and left the priority list empty).

Full pkg/cli test suite passes and all 299 workflows recompile successfully.