-
Notifications
You must be signed in to change notification settings - Fork 34
Description
CI Test Coverage Fix
Summary
CRITICAL: Discovered that 19 integration tests (28.6% of CLI tests and 45.5% of Parser tests) were NOT being executed by any CI job due to missing catch-all matrix groups. This PR adds two catch-all groups to ensure 100% test coverage.
Problem Identified
The CI matrix strategy uses specific test patterns to split tests across parallel jobs. However, two packages lacked catch-all groups (empty pattern ""), causing tests that don't match any specific pattern to be silently skipped:
Missing CLI Tests (14 tests):
- All completion tests:
TestCompletion*Integration(9 tests) - Safe inputs MCP server tests:
TestSafeInputsMCPServer*(4 tests) TestMain
Missing Parser Tests (5 tests):
TestFrontmatterLocationIntegrationTestFrontmatterOffsetCalculationTestImprovementComparisonTestLocateJSONPathInYAML_RealExampleTestValidateWithSchemaAndLocation_PreciseLocation
Changes Made
Added two new matrix groups with empty patterns (catch-all):
1. CLI Completion & Safe Inputs (line 91-93)
- name: "CLI Completion & Safe Inputs"
packages: "./pkg/cli"
pattern: "" # Catch-all for tests not matched by other CLI patterns2. Parser Location & Validation (line 121-123)
- name: "Parser Location & Validation"
packages: "./pkg/parser"
pattern: "" # Catch-all for tests not matched by other Parser patternsImpact
Type: Test Coverage Fix
Impact: CRITICAL
Risk: LOW
Time Impact: +2-3 minutes per CI run (running previously skipped tests)
Benefits:
- ✅ Ensures 100% integration test coverage for CLI package (was 71.4%, now 100%)
- ✅ Ensures 100% integration test coverage for Parser package (was 54.5%, now 100%)
- ✅ Prevents silent test skipping in the future
- ✅ Improves confidence in code changes
Matrix Strategy
The workflow now properly implements the recommended pattern for test splitting:
Packages with ONLY catch-all (run all tests):
./cmd/gh-aw- 1 group
Packages with specific patterns + catch-all (split tests, ensure none skipped):
./pkg/cli- 6 groups (5 specific + 1 catch-all) ✅./pkg/workflow- 13 groups (12 specific + 1 catch-all) ✅./pkg/parser- 2 groups (1 specific + 1 catch-all) ✅
Total matrix groups: 21 (was 19)
Validation
✅ YAML syntax validated
✅ Catch-all groups confirmed present
✅ Matrix entries increased from 19 to 21
✅ No build/test failures expected (only adds missing tests)
Context
This issue was discovered during CI Coach workflow run #15 analysis. Previous runs (#13, #14) optimized parallelization and matrix balance but did not catch the coverage gaps because they focused on performance rather than correctness.
Testing Plan
After merge:
- Monitor new "CLI Completion & Safe Inputs" job for passes/failures
- Monitor new "Parser Location & Validation" job for passes/failures
- Verify all 19 previously skipped tests now execute
- Confirm no unexpected failures from previously untested code paths
References:
- Analysis: §20246600152
Proposed by CI Coach workflow run #15
AI generated by CI Optimization Coach
Note
This was originally intended as a pull request, but the git push operation failed.
Workflow Run: View run details and download patch artifact
The patch file is available as an artifact (aw.patch) in the workflow run linked above.
To apply the patch locally:
# Download the artifact from the workflow run https://github.com/githubnext/gh-aw/actions/runs/20246600152
# (Use GitHub MCP tools if gh CLI is not available)
gh run download 20246600152 -n aw.patch
# Apply the patch
git am aw.patchShow patch preview (47 of 47 lines)
From f8d1ff08d5fa84756ce353010743e9e97fb9e3cc Mon Sep 17 00:00:00 2001
From: "github-actions[bot]" <github-actions[bot]@users.noreply.github.com>
Date: Mon, 15 Dec 2025 20:42:26 +0000
Subject: [PATCH] fix: Add catch-all matrix groups for complete test coverage
- Add 'CLI Completion & Safe Inputs' catch-all group for ./pkg/cli
- Add 'Parser Location & Validation' catch-all group for ./pkg/parser
- Ensures 19 previously skipped integration tests now execute
- Increases matrix from 19 to 21 groups
Without catch-all groups, tests not matching specific patterns were
silently skipped. This affected 14 CLI tests (28.6%) and 5 Parser
tests (45.5%), including all completion tests and safe inputs tests.
Fixes test coverage gaps discovered in CI Coach run #15.
---
.github/workflows/ci.yml | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index e9ba2bf..e51d219 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -87,6 +87,9 @@ jobs:
- name: "CLI Logs & Firewall"
packages: "./pkg/cli"
pattern: "TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow"
+ - name: "CLI Completion & Safe Inputs" # Catch-all for remaining CLI tests
+ packages: "./pkg/cli"
+ pattern: ""
- name: "Workflow Compiler"
packages: "./pkg/workflow"
pattern: "TestCompile|TestWorkflow|TestGenerate|TestParse"
@@ -117,6 +120,9 @@ jobs:
- name: "Parser Remote Fetch & Cache"
packages: "./pkg/parser"
pattern: "TestDownloadFileFromGitHub|TestResolveIncludePath|TestDownloadIncludeFromWorkflowSpec|TestImportCache"
+ - name: "Parser Location & Validation" # Catch-all for remaining parser tests
+ packages: "./pkg/parser"
+ pattern: ""
- name: "Workflow Permissions"
packages: "./pkg/workflow"
pattern: "TestPermissions|TestPackageExtr
... (truncated)