Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Nov 15, 2025

Engine validation errors lacked actionable guidance. Users encountered messages like "invalid engine configuration in included file" without examples or valid options listed.

Changes

Error Message Enhancements

Multiple engine specifications (L93)

  • Added count: (%d engine specifications detected) for clarity on duplication extent

Parse failure (L104)

  • Labeled format examples: Example (string): and (object):
  • Added complete object example with optional model field
  • Used escaped newlines \\n for proper YAML display

Invalid configuration (L119)

  • Specified root cause: missing or invalid 'id' field
  • Clarified requirement: Expected string or object with 'id' field
  • Provided both format examples with proper YAML syntax

Test Coverage

Added TestValidateEngineErrorMessageQuality and TestValidateSingleEngineSpecificationErrorMessageQuality with subtests validating:

  • All valid engines listed (copilot, claude, codex, custom)
  • Example presence and format
  • Count inclusion for multiple engines
  • Both string and object format examples

Example

Before:

invalid engine configuration in included file

After:

invalid engine configuration in included file, missing or invalid 'id' field.
Expected string or object with 'id' field.
Example (string): engine: copilot or (object): engine:\n  id: copilot\n  model: gpt-4

Follows error message style guide template: [what's wrong] + [what's expected] + [example].

Original prompt

This section details on the original issue you should resolve

<issue_title>[task] Enhance engine validation error messages with examples and valid options</issue_title>
<issue_description>## Objective
Improve error messages in pkg/workflow/engine_validation.go to follow the error message style guide template: [what's wrong] + [what's expected] + [example].

Context

Part of discussion #4071 - Repository Quality Improvement focusing on error message quality. Currently only 8% of error messages follow the established style guide. Engine validation errors are high-priority as they're frequently encountered by users.

Approach

Review all error messages in pkg/workflow/engine_validation.go and enhance them according to .github/instructions/error-messages.instructions.md.

For each error message:

  1. State what's wrong clearly
  2. Explain what's expected (valid values, formats, types)
  3. Provide a concrete example using "Example:" prefix

Reference excellent examples in time_delta.go and manual_approval.go for the proper format.

Files to Modify

  • Update: pkg/workflow/engine_validation.go (improve all error messages)
  • Update: pkg/workflow/engine_validation_test.go (add test assertions for examples)

Example Transformations

Before:

fmt.Errorf("invalid engine: %s", engineID)

After:

fmt.Errorf("invalid engine: %s. Valid engines are: copilot, claude, codex, custom. Example: engine: copilot", engineID)

Before:

fmt.Errorf("engine must be string or object, got %T", val)

After:

fmt.Errorf("engine must be string or object, got %T. Example (string): engine: copilot or (object): engine:\\n  id: copilot\\n  model: gpt-5", val)

Acceptance Criteria

  • All engine validation errors include "Example:" in the message
  • Invalid engine errors list all valid options (copilot, claude, codex, custom)
  • Type mismatch errors show the received type using %T
  • Model validation errors provide example of correct format
  • All improved messages have corresponding test assertions in engine_validation_test.go
  • Error messages use proper YAML syntax in examples (quotes for strings, proper indentation)
  • Run make test-unit to verify all tests pass

AI generated by Plan Command for discussion #4071</issue_description>

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

- Improved error messages in engine_validation.go following style guide
- Added count to multiple engine specifications error for clarity
- Enhanced parse error with proper Example (string) and (object) labels
- Made invalid configuration error more specific about missing 'id' field
- Used escaped newlines (\\n) for proper YAML formatting in examples
- Added comprehensive test coverage in engine_validation_test.go
- All tests pass successfully

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Improve engine validation error messages with examples Enhance engine validation error messages with examples and valid options Nov 15, 2025
Copilot AI requested a review from pelikhan November 15, 2025 19:02
Copilot finished work on behalf of pelikhan November 15, 2025 19:02
@pelikhan pelikhan marked this pull request as ready for review November 15, 2025 19:38
Copilot AI review requested due to automatic review settings November 15, 2025 19:38
Copilot finished reviewing on behalf of pelikhan November 15, 2025 19:41
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR enhances engine validation error messages to follow the established style guide template: [what's wrong] + [what's expected] + [example]. The improvements make error messages more actionable by adding counts, format examples, and valid options.

Key Changes:

  • Added engine specification count to multiple engines error for better clarity
  • Enhanced parse failure errors with labeled format examples (string and object)
  • Improved invalid configuration errors by specifying root cause (missing/invalid 'id' field)
  • Added comprehensive test coverage to validate error message quality

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
pkg/workflow/engine_validation.go Enhanced three error messages with counts, examples, and valid options following style guide
pkg/workflow/engine_validation_test.go Added two new test functions with subtests to validate error message quality and completeness

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

}

// Error should include an example
if !strings.Contains(errorMsg, "Example:") && !strings.Contains(errorMsg, "engine: copilot") {
Copy link

Copilot AI Nov 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test assertion uses incorrect logical operator (AND &&). The condition should be !strings.Contains(errorMsg, "Example:") || !strings.Contains(errorMsg, "engine: copilot") instead, because the test should fail if the error message is missing EITHER "Example:" OR the example content. With AND, the test only fails if BOTH are missing, which would allow an error message with just "engine: copilot" but no "Example:" prefix to pass.

Suggested change
if !strings.Contains(errorMsg, "Example:") && !strings.Contains(errorMsg, "engine: copilot") {
if !strings.Contains(errorMsg, "Example:") || !strings.Contains(errorMsg, "engine: copilot") {

Copilot uses AI. Check for mistakes.
}

// Error should include example
if !strings.Contains(errorMsg, "Example:") && !strings.Contains(errorMsg, "engine: copilot") {
Copy link

Copilot AI Nov 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test assertion uses incorrect logical operator (AND &&). The condition should be !strings.Contains(errorMsg, "Example:") || !strings.Contains(errorMsg, "engine: copilot") instead, because the test should fail if the error message is missing EITHER "Example:" OR the example content. With AND, the test only fails if BOTH are missing, which would allow an error message with just "engine: copilot" but no "Example:" prefix to pass.

Suggested change
if !strings.Contains(errorMsg, "Example:") && !strings.Contains(errorMsg, "engine: copilot") {
if !strings.Contains(errorMsg, "Example:") || !strings.Contains(errorMsg, "engine: copilot") {

Copilot uses AI. Check for mistakes.
@pelikhan pelikhan merged commit 3637ed7 into main Nov 15, 2025
140 of 145 checks passed
@pelikhan pelikhan deleted the copilot/enhance-error-message-validation branch November 15, 2025 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[task] Enhance engine validation error messages with examples and valid options

2 participants