[CI Failure Doctor] CMD Tests fail because validateEngine list formatting is nondeterministic

[github-actions]

🏥 CI Failure Investigation - Run #34185

Summary

The Integration: CMD Tests job running go test ./cmd/gh-aw failed due to TestValidateEngine asserting that the validation error string matches a hard-coded list of engine IDs, but the CLI code formatted that list in a nondeterministic order that changed when the registry map iteration order shifted.

Failure Details

• Run: 21775107387
• Commit: 5c5b3b0
• Trigger: push

Root Cause Analysis

validateEngine called workflow.GetGlobalEngineRegistry().GetSupportedEngines() and printed the result via strings.Join without sorting or inserting an "or" before the final entry. Because GetSupportedEngines iterates a map, the engine names sometimes come back in a different order (the failed log shows codex, copilot, custom, claude), which no longer matched the string that TestValidateEngine was hard-coded to expect ('claude', 'codex', 'copilot', or 'custom'). The exact string comparison therefore started failing even though the underlying validation logic was still correct.

Failed Jobs and Errors

• Integration: CMD Tests (job 62830258415) – TestValidateEngine failed with invalid engine value 'gpt4'. Must be 'codex', 'copilot', 'custom', 'claude', but the test asserted the message had to say 'claude', 'codex', 'copilot', or 'custom'.

Investigation Findings

• Logs show TestValidateEngine failing in cmd/gh-aw/main_entry_test.go because err.Error() no longer matched the literal string the test constructs.
• The engine registry list is generated from a map, so iteration order is not stable. The error message also lacked the conjunction "or", so the test could not pass even when the names were the same but ordered differently.
• Manual go test ./cmd/gh-aw invocation currently hits proxy.golang.org and fails with 403 Forbidden while downloading modules, so the job cannot be rerun locally until the proxy access issue is resolved.

Recommended Actions

• Sort the supported engine list and humanize the joined string (including an "or" before the final entry) before embedding it in the error message so that the message is deterministic and matches what TestValidateEngine expects. (Fix applied in cmd/gh-aw/main.go.)
• Once the environment can reach proxy.golang.org, rerun go test ./cmd/gh-aw (or the full Integration: CMD Tests job) to confirm the fix and that no additional regressions occur.

Prevention Strategies

Format validation error messages that depend on registry contents with deterministic sorting and natural-language connectors instead of relying on map ordering; tests that compare exact strings can remain stable if the formatter is predictable.

AI Team Self-Improvement

When comparing error output to golden strings, always sort any slice derived from a map and render it with human-readable punctuation (e.g., add "or" before the final entry) so the text stays stable even if the map order changes.

Historical Context

TestValidateEngine has previously failed whenever the engine registry order shifted, so this failure matches a known pattern: the tests assert a hard-coded string while the CLI code reconstructs that string from a map. This investigation reinforces that the formatting must be deterministic rather than relying on go map iteration order.

AI generated by CI Failure Doctor

To add this workflow in your repository, run gh aw add githubnext/agentics/workflows/ci-doctor.md@ea350161ad5dcc9624cf510f134c6a9e39a6f94d. See usage guide.

• expires on Feb 8, 2026, 6:02 AM UTC

[github-actions]

🍪 Issue Monster has assigned this to Copilot!

I've identified this issue as a good candidate for automated resolution and assigned it to the Copilot agent.

The Copilot agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

AI generated by Issue Monster