Automate Rules File Validation

[JeremyDev87]

Automate Rules File Validation

Summary

Build an automated validation system for rule files in the .ai-rules/ directory to ensure consistency and accuracy of rule files.

Background

Current State

Current rule file validation status:

.ai-rules/
├── rules/           # Markdown files - no validation
│   ├── core.md
│   ├── project.md
│   └── augmented-coding.md
├── agents/          # JSON files - no validation
│   └── *.json (14 files)
└── adapters/        # Markdown files - no validation
    └── *.md (6 files)

Current validation tools:

• scripts/validate-rules.sh file exists but not enabled in CI
• No schema validation for JSON files
• No Markdown linting

Why This Matters

1. Rule file integrity is core to Codingbuddy

Codingbuddy's value:

"Single source of truth for AI coding rules"

If rule files have errors:

• AI assistants follow incorrect rules
• Inconsistencies between different tools
• Users receive confusing results
• Project credibility degraded

Example:

// Invalid JSON (missing comma)
{
  "name": "Frontend Developer"
  "version": "1.3.0"  // ← SyntaxError
}

If such errors get merged:

• MCP server fails to load agent
• Cryptic error message for users
• "It worked fine yesterday?" confusion

2. Manual validation doesn't scale

Current validation process:

Developer A: "Need to check JSON syntax" → Visual inspection
Developer B: "Is the schema correct?" → Compare with other files
Reviewer: "Is this the right structure?" → Relies on memory

Problems:

• Validation burden increases as files grow
• Humans make mistakes (especially in large JSON files)
• Different reviewers apply different standards
• "It was like this before so it's probably fine" inertia

With automation:

Developer: Make change → Commit → Push
CI: Automatically validates all files
    - JSON syntax ✓
    - Schema compliance ✓
    - Markdown linting ✓
    - Required fields exist ✓
→ Immediate feedback if issues found

3. Without schema validation, structural consistency can't be maintained

Why structural differences emerge between agent files:

1. No schema defined initially
2. Each developer writes with their own interpretation
3. Gradual drift over time

With schema validation:

// agent.schema.json
{
  "required": ["name", "version", "description", "role", "activation"],
  "properties": {
    "name": { "type": "string" },
    "version": { "type": "string", "pattern": "^\\d+\\.\\d+\\.\\d+$" },
    ...
  }
}

→ Files that don't match schema fail in CI

4. Markdown quality affects user experience

Rule files are read by both AI assistants and humans.

Markdown quality issues:

• Broken links ([link](broken.md))
• Inconsistent heading levels
• Invalid code block syntax
• Typos and grammar errors

With quality checks:

• markdownlint: Style consistency
• Link validation: Detect broken links
• Spellcheck: Detect typos (optional)

5. Validation automation improves contributor experience

Without automation:

Contributor: Submit PR
Reviewer: "Missing JSON comma", "Schema doesn't match"
Contributor: Fix and resubmit
Reviewer: "Please fix Markdown heading levels"
Contributor: Fix again...
→ Round-trip time increases, both sides fatigued

With automation:

Contributor: Submit PR
CI: "Missing JSON comma (line 42)", "Required field 'version' missing"
Contributor: Fix immediately, push again
CI: Pass ✓
Reviewer: Focus only on logic and content
→ Efficient review process

Precedent in This Project

scripts/validate-rules.sh already exists:

# Currently inactive

This indicates the need for validation was recognized but not fully implemented.

Scope

In Scope

1. JSON schema validation

   • Define JSON schema for agent files
   • Schema-based automatic validation
   • Detailed error messages

2. Markdown linting

   • Configure markdownlint
   • Project-specific rules
   • Broken link checking

3. CI integration

   • Add rule validation job to CI pipeline
   • Only run validation on .ai-rules/ file changes (optimization)
   • Block PR on validation failure
   • Display error messages as PR comments

4. Local development support

   • yarn validate:rules script working
   • Include rule validation in pre-commit hook (optional)
   • Real-time schema validation in VS Code

Out of Scope

• Semantic validation of rule file content (e.g., "Is this rule reasonable?")
• Natural language quality checking
• Automatic rule file generation

Acceptance Criteria

JSON Schema Validation

• agent.schema.json schema file created
• All agent JSON files validate against schema
• Clear error messages on schema violations
• Schema validation runs in CI

Markdown Linting

• markdownlint configured (.markdownlint.json)
• All Markdown files in .ai-rules/ linted
• Linting rules adjusted for project needs
• Markdown linting runs in CI

CI Integration

• Rule validation job added to CI pipeline
• Validation only runs on .ai-rules/ file changes (optimization)
• PR blocked on validation failure
• Error messages displayed as PR comments

Local Development

• yarn validate:rules script works
• Rule validation included in pre-commit hook (optional)
• Real-time schema validation in VS Code

Risk Assessment

Risk	Impact	Mitigation
Existing files don't match schema	Medium	Adjust schema to match existing files, gradual migration
Excessive Markdown rules increase contribution barriers	Low	Apply only reasonable rules, utilize auto-fix
CI execution time increase	Low	Only validate changed files, use caching

References

• JSON Schema
• ajv-cli
• markdownlint
• markdown-link-check