Rubric Eval Loop Plugin

A Claude Code / Claude Skills-style plugin for creating production-ready rubrics, upgrading existing skills with rubric-based evaluation gates, and manually testing prompts/outputs before a skill is trusted in real workflows.

What this plugin provides

rubric-eval-loop-plugin/
├── .claude-plugin/
│   └── manifest.json
├── skills/
│   ├── _shared/
│   │   ├── eval_core.py        # shared assertion + scoring engine
│   │   └── feedback.py         # sign-off / expert-feedback audit trail
│   ├── create-rubric/
│   │   ├── SKILL.md
│   │   ├── templates/
│   │   │   ├── checklist-rubric.yaml
│   │   │   ├── level-anchor-rubric.yaml
│   │   │   ├── composite-rubric.yaml
│   │   │   └── eval-cases.json
│   │   └── scripts/
│   │       └── validate_rubric.py
│   ├── add-eval-loop/
│   │   ├── SKILL.md
│   │   ├── snippets/
│   │   │   └── eval-gate-section.md
│   │   └── scripts/
│   │       ├── patch_skill_eval_loop.py
│   │       ├── eval_runner.py
│   │       └── improve_skill_loop.py
│   ├── test-eval/
│   │   ├── SKILL.md
│   │   ├── examples/
│   │   │   ├── prompt-output-cases.json
│   │   │   └── sample-rubric.yaml
│   │   └── scripts/
│   │       └── test_eval_runner.py
│   └── build-eval-loop/           # orchestrator: rubric → patch → test → iterate
│       ├── SKILL.md
│       └── scripts/
│           └── preflight.py
├── tests/                     # pytest suite for the shared engine + scripts
├── requirements.txt
└── docs/
    ├── IMPLEMENTATION_PLAN.md
    ├── RUBRIC_SCHEMA.md
    ├── STORAGE_LEVELS.md
    └── UPGRADE_WORKFLOW.md

Install

Local Development Installation

To test the plugin locally from the source files:

1. Clone the repository:

   git clone https://github.com/liberalterian/create-eval-loop.git

2. Run with a local plugin directory:

   claude --plugin-dir ./create-eval-loop

3. Verify: use /plugin list or /help to confirm the skills are active.

Rubrics generated by create-rubric can live at user, project, or sub-directory level:

~/.claude/evals/                         # user-level, reusable everywhere
./.claude/evals/                         # project-level
./some/subsystem/.claude/evals/          # sub-directory-specific

Core workflow

1. Use create-rubric to define a rubric or eval cases.
2. Use add-eval-loop to patch an existing skill with an evaluation gate.
3. Use test-eval to manually test prompts/outputs, evaluator behavior, and thresholds.
4. Iterate until the rubric, outputs, and acceptance conditions are trustworthy.

Or run the whole thing in one guided flow: build-eval-loop orchestrates steps 1–4 (rubric → patch → test → iterate to green) with human sign-off checkpoints. It is user-invoked only (/build-eval-loop).

Recommended defaults

• Start with checklist rubrics: 3–6 criteria, 5–10 binary observable sub-items each.
• Use a pass condition with both an overall threshold and per-criterion floor.
• Use deterministic assertions for format, schema, and exact behavior.
• Use LLM-as-judge sparingly for subjective quality, and require JSON output.
• Stop after 3–5 optimization rounds or when improvement stalls.
• Store score history and evaluator disagreements for debugging.

Safety and reliability posture

This plugin deliberately separates rubric creation, target skill patching, and manual testing. That keeps upgrades reversible, prevents hidden behavior changes, and gives you an audit trail before a modified skill becomes part of a production workflow.