output-validator

Validate agent outputs automatically with configurable quality criteria.

Installation

npm install -g output-validator

Or use locally:

npm install output-validator

Quick Start

CLI

# Validate output against a contract file
output-validator validate "Your agent output here" contract.json

# Validate a file against a contract
output-validator validate ./output.txt ./contract.json

# Use a built-in template
output-validator template agent-response "Your output here"

# Read from stdin
echo "Your output" | output-validator stdin contract.json

# Output as JSON
output-validator validate output.txt contract.json --format json

# Custom threshold
output-validator validate output.txt contract.json --threshold 85

# Create a new contract scaffold
output-validator create my-contract

# List available templates
output-validator templates

API

import { validate, loadContract } from 'output-validator';

// Load a contract
const contract = loadContract('./contract.json');

// Validate output
const result = validate('Your agent output here', contract);

console.log(result.score);        // 0-100
console.log(result.passed);       // boolean
console.log(result.categories);   // CategoryScore[]
console.log(result.recommendations); // Recommendation[]
console.log(result.summary);      // string

// With options
const result = validate(output, contract, {
  scoring: { threshold: 85 },
  context: { userId: '123' },
  includeRecommendations: true,
});

Validation Contracts

A contract defines the rules for validating outputs:

{
  "name": "my-contract",
  "description": "Validation rules for X",
  "version": "1.0.0",
  "rules": [
    {
      "id": "has-content",
      "name": "Has Content",
      "category": "completeness",
      "description": "Output must have meaningful content",
      "check": { "type": "length", "min": 20, "wordCount": true },
      "required": true,
      "weight": 1.5
    }
  ],
  "scoring": {
    "threshold": 70,
    "failOnRequired": true
  }
}

Rule Types

Type	Description	Example
contains	Check for substrings	{ "type": "contains", "value": "hello" }
regex	Regex pattern matching	{ "type": "regex", "pattern": "\\d+" }
length	Min/max length or word count	{ "type": "length", "min": 10, "max": 1000 }
schema	JSON structure validation	{ "type": "schema", "fields": [...] }
custom	Custom JS expression	{ "type": "custom", "expression": "output.length > 10" }

Rule Categories

• completeness — Does the output have all required content?
• relevance — Is the output on-topic and focused?
• actionability — Does the output provide clear, usable information?
• format — Does the output follow formatting requirements?
• security — Are there security concerns (secrets, sensitive data)?
• custom — User-defined category

Scoring

Each rule contributes to the overall score (0-100) based on its weight. Required rules that fail will cause validation to fail regardless of the score.

Built-in Templates

Template	Description
agent-response	General agent response quality
code-output	Code generation output
json-api-response	JSON API response validation
summary	Summary/document quality

Programmatic Usage

Create a contract programmatically

import { validate } from 'output-validator';

const contract = {
  name: 'my-check',
  description: 'Quick validation',
  version: '1.0.0',
  rules: [
    {
      id: 'no-empty',
      name: 'Not Empty',
      category: 'completeness',
      check: { type: 'length', min: 10 },
      required: true,
    },
    {
      id: 'no-bad-words',
      name: 'Clean Language',
      category: 'security',
      check: { type: 'contains', value: ['badword1', 'badword2'], inverse: true },
    },
  ],
};

const result = validate(someOutput, contract);
if (!result.passed) {
  console.log('Issues found:', result.recommendations);
}

Use as middleware

import { validate, loadContract } from 'output-validator';

const contract = loadContract('./quality.contract.json');

function qualityGate(output: string): boolean {
  const result = validate(output, contract, { scoring: { threshold: 80 } });
  return result.passed;
}

Exit Codes

Code	Meaning
0	Validation passed
1	Validation failed
2	Error (invalid contract, missing file, etc.)

License

MIT