Skip to content

[ci-coach] Fix critical test coverage gaps - add catch-all matrix groups #6561

@github-actions

Description

@github-actions

CI Test Coverage Fix

Summary

CRITICAL: Discovered that 19 integration tests (28.6% of CLI tests and 45.5% of Parser tests) were NOT being executed by any CI job due to missing catch-all matrix groups. This PR adds two catch-all groups to ensure 100% test coverage.

Problem Identified

The CI matrix strategy uses specific test patterns to split tests across parallel jobs. However, two packages lacked catch-all groups (empty pattern ""), causing tests that don't match any specific pattern to be silently skipped:

Missing CLI Tests (14 tests):

  • All completion tests: TestCompletion*Integration (9 tests)
  • Safe inputs MCP server tests: TestSafeInputsMCPServer* (4 tests)
  • TestMain

Missing Parser Tests (5 tests):

  • TestFrontmatterLocationIntegration
  • TestFrontmatterOffsetCalculation
  • TestImprovementComparison
  • TestLocateJSONPathInYAML_RealExample
  • TestValidateWithSchemaAndLocation_PreciseLocation

Changes Made

Added two new matrix groups with empty patterns (catch-all):

1. CLI Completion & Safe Inputs (line 91-93)

- name: "CLI Completion & Safe Inputs"
  packages: "./pkg/cli"
  pattern: ""  # Catch-all for tests not matched by other CLI patterns

2. Parser Location & Validation (line 121-123)

- name: "Parser Location & Validation"
  packages: "./pkg/parser"  
  pattern: ""  # Catch-all for tests not matched by other Parser patterns

Impact

Type: Test Coverage Fix
Impact: CRITICAL
Risk: LOW
Time Impact: +2-3 minutes per CI run (running previously skipped tests)

Benefits:

  • ✅ Ensures 100% integration test coverage for CLI package (was 71.4%, now 100%)
  • ✅ Ensures 100% integration test coverage for Parser package (was 54.5%, now 100%)
  • ✅ Prevents silent test skipping in the future
  • ✅ Improves confidence in code changes

Matrix Strategy

The workflow now properly implements the recommended pattern for test splitting:

Packages with ONLY catch-all (run all tests):

  • ./cmd/gh-aw - 1 group

Packages with specific patterns + catch-all (split tests, ensure none skipped):

  • ./pkg/cli - 6 groups (5 specific + 1 catch-all) ✅
  • ./pkg/workflow - 13 groups (12 specific + 1 catch-all) ✅
  • ./pkg/parser - 2 groups (1 specific + 1 catch-all) ✅

Total matrix groups: 21 (was 19)

Validation

✅ YAML syntax validated
✅ Catch-all groups confirmed present
✅ Matrix entries increased from 19 to 21
✅ No build/test failures expected (only adds missing tests)

Context

This issue was discovered during CI Coach workflow run #15 analysis. Previous runs (#13, #14) optimized parallelization and matrix balance but did not catch the coverage gaps because they focused on performance rather than correctness.

Testing Plan

After merge:

  • Monitor new "CLI Completion & Safe Inputs" job for passes/failures
  • Monitor new "Parser Location & Validation" job for passes/failures
  • Verify all 19 previously skipped tests now execute
  • Confirm no unexpected failures from previously untested code paths

References:

Proposed by CI Coach workflow run #15

AI generated by CI Optimization Coach


Note

This was originally intended as a pull request, but the git push operation failed.

Workflow Run: View run details and download patch artifact

The patch file is available as an artifact (aw.patch) in the workflow run linked above.
To apply the patch locally:

# Download the artifact from the workflow run https://github.com/githubnext/gh-aw/actions/runs/20246600152
# (Use GitHub MCP tools if gh CLI is not available)
gh run download 20246600152 -n aw.patch
# Apply the patch
git am aw.patch
Show patch preview (47 of 47 lines)
From f8d1ff08d5fa84756ce353010743e9e97fb9e3cc Mon Sep 17 00:00:00 2001
From: "github-actions[bot]" <github-actions[bot]@users.noreply.github.com>
Date: Mon, 15 Dec 2025 20:42:26 +0000
Subject: [PATCH] fix: Add catch-all matrix groups for complete test coverage

- Add 'CLI Completion & Safe Inputs' catch-all group for ./pkg/cli
- Add 'Parser Location & Validation' catch-all group for ./pkg/parser
- Ensures 19 previously skipped integration tests now execute
- Increases matrix from 19 to 21 groups

Without catch-all groups, tests not matching specific patterns were
silently skipped. This affected 14 CLI tests (28.6%) and 5 Parser
tests (45.5%), including all completion tests and safe inputs tests.

Fixes test coverage gaps discovered in CI Coach run #15.
---
 .github/workflows/ci.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index e9ba2bf..e51d219 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -87,6 +87,9 @@ jobs:
           - name: "CLI Logs & Firewall"
             packages: "./pkg/cli"
             pattern: "TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow"
+          - name: "CLI Completion & Safe Inputs"  # Catch-all for remaining CLI tests
+            packages: "./pkg/cli"
+            pattern: ""
           - name: "Workflow Compiler"
             packages: "./pkg/workflow"
             pattern: "TestCompile|TestWorkflow|TestGenerate|TestParse"
@@ -117,6 +120,9 @@ jobs:
           - name: "Parser Remote Fetch & Cache"
             packages: "./pkg/parser"
             pattern: "TestDownloadFileFromGitHub|TestResolveIncludePath|TestDownloadIncludeFromWorkflowSpec|TestImportCache"
+          - name: "Parser Location & Validation"  # Catch-all for remaining parser tests
+            packages: "./pkg/parser"
+            pattern: ""
           - name: "Workflow Permissions"
             packages: "./pkg/workflow"
             pattern: "TestPermissions|TestPackageExtr
... (truncated)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions