Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Nov 15, 2025

Schema validation errors now include field-specific inline examples following the error message pattern: [what's wrong] + [what's expected] + [example].

Changes

Enhanced error messages (pkg/workflow/schema_validation.go):

  • Added enhanceSchemaValidationError() to wrap ValidationError with field-specific examples
  • Added extractFieldPath() to extract field name from InstanceLocation
  • Added getFieldExample() with 22 common workflow field examples + generic type fallbacks

Test coverage (2 new test files):

  • Unit tests for field extraction, example generation, and error enhancement
  • Integration tests for workflow compilation and example quality

Example

Before:

GitHub Actions schema validation failed: type mismatch at /timeout-minutes

After:

GitHub Actions schema validation failed: type mismatch at /timeout-minutes. Example: timeout-minutes: 10

For enum fields:

GitHub Actions schema validation failed: invalid enum value. Valid engines are: copilot, claude, codex, custom. Example: engine: copilot

For complex fields:

GitHub Actions schema validation failed: type error. Example: permissions:
  contents: read
  issues: write

Generic type-based examples (string, boolean, object, array, integer) are provided for fields not in the predefined map.

Fixes #4071

Original prompt

This section details on the original issue you should resolve

<issue_title>[task] Add inline examples to schema validation errors</issue_title>
<issue_description>## Objective
Enhance schema validation error messages to provide inline examples rather than just referencing the schema. When validation fails, show the user exactly what correct YAML should look like.

Context

Part of discussion #4071 - Repository Quality Improvement. Schema validation errors currently return generic messages that reference the schema but don't provide actionable examples. This forces users to search documentation.

Approach

  1. Enhance pkg/workflow/schema_validation.go to parse schema validation errors and identify which field failed
  2. Create a helper function extractFieldFromSchemaError() to parse JSON schema validation errors
  3. Build a map of common field examples for quick lookup
  4. Provide field-specific examples based on the failed field

Files to Modify

  • Update: pkg/workflow/schema_validation.go
  • Update: pkg/workflow/schema_validation_test.go
  • Create: Helper function for field extraction and example mapping

Example Improvements

Current behavior:

return fmt.Errorf("GitHub Actions schema validation failed: %w", err)

Improved behavior for timeout-minutes:

GitHub Actions schema validation failed for 'timeout-minutes': must be an integer. Example: timeout-minutes: 10

Improved behavior for engine:

GitHub Actions schema validation failed for 'engine': invalid value. Valid engines are: copilot, claude, codex, custom. Example: engine: copilot

Improved behavior for permissions:

GitHub Actions schema validation failed for 'permissions': must be an object with permission levels. Example: permissions:\\n  contents: read\\n  issues: write

Implementation Steps

  1. Add helper function to extract field name from schema validation error message
  2. Create map of field-specific examples for common fields:
    • timeout-minutes, engine, permissions, on, tools, etc.
  3. Enhance validateGitHubActionsSchema() to parse the error and provide targeted examples
  4. Add comprehensive tests for different schema validation failure scenarios

Acceptance Criteria

  • Schema validation errors extract field name from validation error
  • Field-specific examples are provided based on the failed field
  • Common schema violations (type mismatches, missing fields) have tailored examples
  • Error messages reference both the schema issue and provide inline examples
  • Tests verify example inclusion in schema validation errors
  • Handle cases where field name cannot be extracted (fallback to generic message)
  • Run make test-unit to verify all tests pass

AI generated by Plan Command for discussion #4071</issue_description>

Comments on the Issue (you are @copilot in this section)


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 2 commits November 15, 2025 19:05
Enhance schema validation error messages to provide field-specific examples
for common workflow fields. When schema validation fails, users now receive
actionable inline examples showing correct YAML syntax.

- Add enhanceSchemaValidationError() to parse ValidationError and add examples
- Add extractFieldPath() to extract field name from InstanceLocation
- Add getFieldExample() with map of 22 common workflow field examples
- Add generic type-based examples for unknown fields (string, boolean, object, array, integer)
- Add comprehensive unit tests for field extraction, example generation, and error enhancement
- Add integration test for schema validation with examples
- Add example quality test to verify all field examples contain required parts

Example improvements:
- timeout-minutes: "Example: timeout-minutes: 10"
- engine: "Valid engines are: copilot, claude, codex, custom. Example: engine: copilot"
- permissions: "Example: permissions:\n  contents: read\n  issues: write"

Fixes #4071

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add inline examples to schema validation errors Add inline examples to schema validation errors Nov 15, 2025
Copilot AI requested a review from pelikhan November 15, 2025 19:10
Copilot finished work on behalf of pelikhan November 15, 2025 19:10
@pelikhan pelikhan marked this pull request as ready for review November 15, 2025 19:39
Copilot AI review requested due to automatic review settings November 15, 2025 19:39
Copilot finished reviewing on behalf of pelikhan November 15, 2025 19:41
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR enhances schema validation error messages to provide inline, field-specific examples rather than requiring users to reference external documentation. When a GitHub Actions schema validation fails, users now receive helpful examples showing correct YAML syntax for the field in question.

Key changes:

  • Added intelligent error enhancement to provide context-aware examples based on the failing field
  • Implemented a comprehensive field example map covering 22 common workflow fields with fallback to generic type-based examples
  • Added thorough test coverage including unit tests and integration tests for example quality

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
pkg/workflow/schema_validation.go Enhanced with enhanceSchemaValidationError(), extractFieldPath(), and getFieldExample() functions to wrap validation errors with examples
pkg/workflow/schema_validation_test.go Added comprehensive unit tests for field extraction, example generation, and error enhancement logic
pkg/workflow/schema_validation_integration_test.go Added integration tests to verify workflow compilation and validate example quality for common fields

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +162 to +181
"permissions": "Example: permissions:\\n contents: read\\n issues: write",
"on": "Example: on: push or on:\\n issues:\\n types: [opened]",
"runs-on": "Example: runs-on: ubuntu-latest",
"concurrency": "Example: concurrency: production or concurrency:\\n group: ${{ github.workflow }}\\n cancel-in-progress: true",
"env": "Example: env:\\n NODE_ENV: production",
"tools": "Example: tools:\\n github:\\n allowed: [list_issues]",
"steps": "Example: steps:\\n - name: Checkout\\n uses: actions/checkout@v4",
"jobs": "Example: jobs:\\n build:\\n runs-on: ubuntu-latest\\n steps:\\n - run: echo 'hello'",
"strategy": "Example: strategy:\\n matrix:\\n os: [ubuntu-latest, windows-latest]",
"container": "Example: container: node:20 or container:\\n image: node:20\\n options: --user root",
"services": "Example: services:\\n postgres:\\n image: postgres:15\\n env:\\n POSTGRES_PASSWORD: postgres",
"defaults": "Example: defaults:\\n run:\\n shell: bash",
"name": "Example: name: \"Build and Test\"",
"if": "Example: if: github.event_name == 'push'",
"environment": "Example: environment: production or environment:\\n name: production\\n url: https://example.com",
"outputs": "Example: outputs:\\n build-id: ${{ steps.build.outputs.id }}",
"needs": "Example: needs: build or needs: [build, test]",
"uses": "Example: uses: ./.github/workflows/reusable.yml",
"with": "Example: with:\\n node-version: '20'",
"secrets": "Example: secrets:\\n token: ${{ secrets.GITHUB_TOKEN }}",
Copy link

Copilot AI Nov 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The newlines in the example strings are double-escaped (\\\\n), which will render as literal \\n in error messages instead of actual line breaks. They should use single backslash (\\n) or use raw string literals with actual newlines.

Copilot uses AI. Check for mistakes.
return fmt.Sprintf("Example: %s: true", fieldPath)
}
if strings.Contains(errorMsg, "object") {
return fmt.Sprintf("Example: %s:\\n key: value", fieldPath)
Copy link

Copilot AI Nov 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The newline is double-escaped (\\\\n), which will render as literal \\n in the error message instead of an actual line break. It should use single backslash (\\n).

Suggested change
return fmt.Sprintf("Example: %s:\\n key: value", fieldPath)
return fmt.Sprintf("Example: %s:\n key: value", fieldPath)

Copilot uses AI. Check for mistakes.
@pelikhan pelikhan merged commit 64af0c6 into main Nov 15, 2025
96 checks passed
@pelikhan pelikhan deleted the copilot/add-inline-examples-to-errors branch November 15, 2025 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[task] Add inline examples to schema validation errors

2 participants