docs: NLP metrics and semantic similarity evaluator examples (ROUGE, BLEU, similarity)

[christso]

Scope

This is a documentation/examples task — no core code changes needed. These evaluators are implementable as code judges using the existing @agentv/eval SDK.

What to Create

Add example code judges in examples/ for standard NLP metrics:

1. ROUGE-N (examples/nlp-metrics/judges/rouge.ts)

import { defineCodeJudge } from "@agentv/eval";
// Use rouge npm package to compute ROUGE-N score
// Config: { n: 1, threshold: 0.75 }

2. BLEU (examples/nlp-metrics/judges/bleu.ts)

// Use sacrebleu or bleu-score package
// Config: { threshold: 0.5 }

3. Semantic Similarity (examples/nlp-metrics/judges/similarity.ts)

// Use target proxy to call embedding model
// Compute cosine similarity between expected and actual
// Config: { threshold: 0.8 }

4. Levenshtein Distance (examples/nlp-metrics/judges/levenshtein.ts)

// Compute edit distance, score = 1 - (distance / max_length)
// Config: { threshold: 5 }

Example EVAL.yaml

name: nlp-metrics-demo
cases:
  - id: summarization-test
    input: "Summarize this article..."
    expected_outcome: "Reference summary text"
execution:
  evaluators:
    - name: rouge_score
      type: code_judge
      script: ["bun", "run", "judges/rouge.ts"]
      n: 1
      threshold: 0.75
    - name: semantic_match
      type: code_judge
      script: ["bun", "run", "judges/similarity.ts"]
      threshold: 0.8

Documentation

• Add to docs site: "NLP Metrics Evaluators" guide
• Reference from existing evaluator docs
• Show how to compose with other evaluator types

Acceptance Criteria

• Example code judges for ROUGE, BLEU, Levenshtein, semantic similarity
• Example EVAL.yaml demonstrating usage
• README in example directory
• Documentation page on agentv.dev

[christso]

Agent-Ready Implementation Plan (Final)

Branch

git checkout -b docs/227-nlp-metrics-examples

Reference files to read first

1. packages/eval/src/schemas.ts — CodeJudgeInputSchema — context fields
2. packages/eval/src/index.ts — defineCodeJudge API
3. examples/features/code-judge-sdk/ — Code judge patterns
4. examples/features/code-judge-with-llm-calls/ — Code judge with LLM integration

Code judge context fields (verified)

• context.candidateAnswer — agent's response (string)
• context.criteria — reference/expected text (string). NOT expectedOutcome
• context.config — Record<string, unknown> | null — extra YAML keys. Avoid type as config key

Step-by-step implementation

Step 1: Create example directory

examples/features/nlp-metrics/
├── evals/
│   └── dataset.yaml
├── judges/
│   ├── rouge.ts
│   ├── bleu.ts
│   ├── levenshtein.ts
│   └── similarity.ts
├── package.json
└── README.md

Step 2: Implement judges

• rouge.ts — ROUGE-N (n-gram recall). Implement manually (~20 lines). Config: { n: 1, threshold: 0.5 }. Compare context.candidateAnswer against context.criteria.
• bleu.ts — BLEU (modified n-gram precision + brevity penalty). Implement manually.
• levenshtein.ts — Edit distance. Score = 1 - (distance / max_length). Implement manually (~15 lines).
• similarity.ts — Semantic similarity via embedding API. Read examples/features/code-judge-with-llm-calls/ for how code judges make LLM calls.

Step 3: Create evals/dataset.yaml and README.md

Verification checklist

cd examples/features/nlp-metrics && bun install
bun run typecheck
bun run lint
bun run build

Notes

• Prefer manual implementations for ROUGE, BLEU, Levenshtein to avoid npm deps
• No core code changes needed