[syntax-error-quality] Improve compiler error messages for YAML syntax errors and permission scope validation

[github-actions]

📊 Error Message Quality Analysis

Analysis Date: 2026-03-23
Test Cases: 3
Average Score: 64.7/100
Status: ⚠️ Needs Improvement

---

Executive Summary

Three compiler error message scenarios were evaluated across five dimensions: clarity, actionability, context, examples, and consistency. The results revealed a significant quality gap between custom validation errors (excellent) and raw YAML parser passthrough errors (poor). The engine name typo validation is exemplary — it includes "Did you mean" suggestions, valid options, usage examples, and documentation links. However, YAML syntax errors surface cryptic low-level library messages ("unexpected key name") with no actionable guidance, scoring critically below the 55-point threshold. Permission scope validation is acceptable but lacks spell-check suggestions.

Key Findings:

• Strengths: Engine name validation is excellent (did-you-mean, examples, docs link); permission errors include valid scope list; all errors include file:line:column
• Weaknesses: YAML parser errors are cryptic with no fix suggestions; permission errors have no "did you mean" for typos; no cross-linking between related error types
• Critical Issues: YAML syntax errors score 41/100 — a developer encountering "unexpected key name" for a missing colon cannot easily self-service the fix

---

Test Case Results

Test Case 1: YAML Syntax Error (Missing Colon) — Score: 41/100 ❌

Test Configuration

Workflow: cli-version-checker.md (351 lines, claude engine)
Error Type: Category A — Frontmatter YAML syntax error
Error Introduced: engine claude (removed : from engine: claude on line 11)

Expected Compiler Output

The error passes through pkg/parser/yaml_error.go::FormatYAMLError() which calls yaml.FormatError(err, false, true) from goccy/go-yaml with inclSource=true. The output format:

[line:col] unexpected key name
   11 | engine claude
      ^
```

After frontmatter line number adjustment and console formatting, the full output would be approximately:

```
cli-version-checker.md:11:1: error: unexpected key name
   11 | engine claude
      ^
```

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 10/25 | Poor |
| Actionability | 5/25 | Critical |
| Context | 15/20 | Good |
| Examples | 0/15 | None |
| Consistency | 11/15 | Acceptable |
| **Total** | **41/100** | **Poor ❌** |

#### Strengths
- ✅ File and line number are present
- ✅ Source line is shown via `yaml.FormatError` with `inclSource=true`
- ✅ Column pointer (`^`) indicates error position

#### Weaknesses
- ❌ "unexpected key name" does not tell the developer *what* is unexpected
- ❌ No suggestion to add the missing `:` after `engine`
- ❌ No example of correct YAML syntax
- ❌ No reference to documentation or common YAML mistakes
- ❌ The message comes verbatim from the YAML library — not translated to user-friendly language

#### Improvement Suggestions

1. **Translate common YAML errors to plain language**:
   - Current: `"unexpected key name"`
   - Better: `"missing ':' after key 'engine' — YAML key-value pairs require a colon separator"`

2. **Add corrected syntax example**:
   ```
   Did you mean:
     engine: claude
   ```

3. **Cross-reference known frontmatter keys**: If the unrecognized key matches a known field (e.g., `engine`), mention it explicitly and link to documentation.

</details>

<details>
<summary><b>Test Case 2: Invalid Engine Name (Typo)</b> — Score: 85/100 ✅</summary>

#### Test Configuration

**Workflow**: `pr-triage-agent.md` (450 lines, copilot engine)  
**Error Type**: Category B — Configuration error: invalid engine name  
**Error Introduced**: `engine: copiilot` (double `i` typo for `copilot`)

#### Expected Compiler Output

Validation happens in `pkg/workflow/engine_definition.go::EngineCatalog.Resolve()` with `parser.FindClosestMatches()` for spell-check suggestions:

```
pr-triage-agent.md:11:1: error: invalid engine: copiilot. Valid engines are: claude, codex, copilot, gemini.

Did you mean: copilot?

Example:
engine: copilot

See: https://github.com/github/gh-aw/blob/main/docs/src/content/docs/reference/engines.md
```

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 22/25 | Excellent |
| Actionability | 23/25 | Excellent |
| Context | 13/20 | Good |
| Examples | 13/15 | Good |
| Consistency | 14/15 | Excellent |
| **Total** | **85/100** | **Excellent ✅** |

#### Strengths
- ✅ "Did you mean: copilot?" — spell-check suggestion pinpoints the fix
- ✅ Lists all valid engines: `claude, codex, copilot, gemini`
- ✅ Provides usage example: `engine: copilot`
- ✅ Links to documentation
- ✅ Clear file:line:column format

#### Weaknesses
- ⚠️ No source code context lines shown (only file:line:col)
- ⚠️ Could mention custom engine option for advanced users

#### Notes
This error is a model for how all compiler errors should look. The combination of "did you mean", valid options list, example, and docs link is exemplary DX.

</details>

<details>
<summary><b>Test Case 3: Invalid Permission Scope</b> — Score: 68/100 ⚠️</summary>

#### Test Configuration

**Workflow**: `repository-quality-improver.md` (568 lines, complex config)  
**Error Type**: Category C — Semantic error: invalid permission scope  
**Error Introduced**: Added `unknown-scope: write` to the `permissions` block

#### Expected Compiler Output

Validation happens via JSON schema (`pkg/parser/schemas/main_workflow_schema.json`, `additionalProperties: false`) processed in `pkg/parser/schema_errors.go::rewriteAdditionalPropertiesError()` with a hint appended from `knownFieldValidValues`:

```
repository-quality-improver.md:8:3: error: Unknown property: unknown-scope (Valid permission scopes: actions, all, attestations, checks, contents, deployments, discussions, id-token, issues, metadata, models, organization-projects, packages, pages, pull-requests, repository-projects, security-events, statuses, vulnerability-alerts)

Evaluation Scores

Dimension	Score	Rating
Clarity	19/25	Good
Actionability	15/25	Acceptable
Context	12/20	Acceptable
Examples	9/15	Acceptable
Consistency	13/15	Good
Total	68/100	Acceptable ⚠️

Strengths

• ✅ "Unknown property" message is clear
• ✅ Includes full list of valid permission scopes in the hint
• ✅ File:line:column present

Weaknesses

• ⚠️ 21-item scope list is overwhelming inline — hard to scan
• ⚠️ No "did you mean" spell-check for near-miss typos (e.g., pull-request vs pull-requests)
• ⚠️ No example of correct permissions block usage
• ⚠️ No link to GitHub Actions permissions documentation

Improvement Suggestions

1. Add spell-check suggestions for permission scope names: Use FindClosestMatches() (already used for engine errors) to suggest pull-requests when someone types pull-request.

2. Format valid scopes as a compact list rather than inline parenthetical:

   Unknown permission scope: unknown-scope
   
   Valid scopes: actions, attestations, checks, contents, deployments,
                 discussions, id-token, issues, metadata, models, packages,
                 pages, pull-requests, security-events, statuses
   
   See: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/controlling-permissions-for-github_token
   

3. Add usage example showing correct permissions syntax.

---

Overall Statistics

Metric	Value
Tests Run	3
Average Score	64.7/100
Excellent (85+)	1 (Test 2)
Good (70-84)	0
Acceptable (55-69)	1 (Test 3)
Poor (40-54)	1 (Test 1)
Critical (<40)	0

Quality Assessment: ⚠️ Needs Improvement — Average score 64.7/100 is below the 70-point threshold. Test Case 1 (YAML syntax) scores 41/100, below the 55-point individual threshold. Test Case 3 (permission scope) narrowly misses the 70-point bar at 68/100.

---

Priority Improvement Recommendations

🔴 High Priority (Critical for DX)

1. Translate YAML parser errors to plain language

The core problem: raw goccy/go-yaml messages like "unexpected key name" or "mapping values are not allowed in this context" are opaque to users unfamiliar with YAML internals.

Solution: Create a YAML error translation layer in pkg/parser/yaml_error.go:

var yamlErrorTranslations = []struct {
    pattern     string
    replacement string
}{
    {"unexpected key name", "missing ':' after key — YAML mapping entries require 'key: value' format"},
    {"mapping values are not allowed in this context", "unexpected ':' — check indentation or if this key belongs in a mapping block"},
    {"did not find expected key", "incorrect indentation or missing key in mapping"},
    {"could not find expected ':'", "missing ':' in key-value pair"},
}
```

Apply translations in `FormatYAMLError()` before returning the error string.

**Impact**: Transforms the most common class of beginner errors from cryptic to actionable.

**2. Add corrected-syntax hints for YAML errors**

When a known frontmatter key (e.g., `engine`) is detected in a YAML error context, append a correction hint:

```
cli-version-checker.md:11:1: error: missing ':' after key — YAML mapping entries require 'key: value' format
   11 | engine claude
      ^
   
Hint: Did you forget the colon? Try:
   engine: claude

This requires correlating the YAML error position with the parsed key name.

🟡 Medium Priority (Enhance DX)

3. Add "Did you mean?" spell-check to permission scope errors

The FindClosestMatches() function used for engine name suggestions is already available. Apply it to permission scope validation in pkg/parser/schema_errors.go:

// In rewriteAdditionalPropertiesError, after identifying unknown property name:
if suggestions := parser.FindClosestMatches(unknownProp, validPermissionScopes); len(suggestions) > 0 {
    hint += fmt.Sprintf("\n\nDid you mean: %s?", strings.Join(suggestions, ", "))
}
```

**4. Add documentation links to permission scope errors**

Append a `See:` line pointing to the GitHub Actions permissions reference documentation, consistent with the engine error format.

**5. Format long valid-values lists as multi-line blocks**

The current inline parenthetical for 21 permission scopes is hard to scan. Format as:

```
Unknown permission scope: unknown-scope

Valid scopes:
  actions, attestations, checks, contents, deployments, discussions,
  id-token, issues, metadata, models, organization-projects, packages,
  pages, pull-requests, repository-projects, security-events, statuses,
  vulnerability-alerts

See: <docs-url>

🟢 Low Priority (Nice to Have)

6. Add context lines to schema validation errors

Schema validation errors currently show file:line:col but not source context lines. The console FormatError function supports Context with line display and caret pointers — populate this field in pkg/parser/schema_errors.go where the error position is known.

7. Add permissions usage example to scope error

# Example: correct permissions block
permissions:
  contents: read
  issues: write
  pull-requests: write
```

---

### Implementation Guide

#### 1. YAML Error Translation (`pkg/parser/yaml_error.go`)

Add a `translateYAMLError()` helper that maps common goccy/go-yaml error strings to user-friendly messages with fix hints. Call it from `FormatYAMLError()` before returning.

#### 2. Spell-Check for Schema Errors (`pkg/parser/schema_errors.go`)

Extend `rewriteAdditionalPropertiesError()` to use `FindClosestMatches()` when the rejected property name has a close match in the valid-properties list for that JSON schema path.

#### 3. Consistency Model

Use the engine error message format as the gold standard template for all validation errors:

```
<file>:<line>:<col>: error: <clear description of what's wrong>

Did you mean: <suggestion>?

Example:
<correct usage>

See: <docs URL>

---

Success Metrics

1. YAML syntax error score: Target ≥ 70/100 (currently 41/100)
2. Permission scope error score: Target ≥ 75/100 (currently 68/100)
3. Overall average: Target ≥ 80/100 (currently 64.7/100)

---

References:

• §23452446743

Generated by Daily Syntax Error Quality Check · ◷

• expires on Mar 26, 2026, 6:15 PM UTC