Evaluation Workflow

Evaluation Workflow

ChefSkills evaluates culinary behavior with scenarios, expected routes, behavior checks, reports, scorecards, and hard safety gates.

Evidence classes

Class	Meaning	Current status
Local simulation	Before/after outputs created and reviewed locally for regression tracking	9 reports, 38 fixtures
Live capture	Provider-neutral prompts run in an external host, preserved, hashed, reviewed, and scored	First foundation packet pending capture

Current reports are medium-confidence local reviewer simulations. They are not live benchmark results, external validation, certification, or a guarantee of future model behavior.

Current public-alpha evidence

• Reports: 9
• Evaluated fixtures: 38
• Blockers: 0
• Baseline average: 3.7599
• ChefSkills-enabled average: 4.8059
• Delta: 1.0461

Rubric and hard gate

Outputs are scored from 0 to 5 on technical accuracy, culinary reasoning, ingredient understanding, workflow quality, sensory reasoning, safety, constraint handling, and communication.

Food safety is a hard gate. A serious unsafe answer fails even when other dimensions are strong.

Source files:

• evaluation/rubric.yaml
• evaluation/safety-gates.yaml
• evaluation/fixtures.yaml
• evaluation/scorecards/summary.json

Live-output harness

Create a provider-neutral packet from an existing regression suite:

python .\scripts\create-live-evaluation-run.py --suite state-reasoning --run-id 2026-09-04-state-reasoning-live

Validate live-run manifests:

python .\scripts\validate-live-evaluation-runs.py

Each packet records fixture ids, route metadata, prompt hashes, model and context notes, planned output paths, output hashes after capture, reviewer decisions, report links, scorecard links, and rerun instructions. The harness does not call a model API or store credentials.

Current packet: 2026-09-04-foundation-live-smoke.

Evaluation flow

1. Select a fixture from evaluation/fixtures.yaml.
2. Run or simulate a baseline output.
3. Run or simulate the skill-enabled output with the expected route.
4. Score both outputs with evaluation/rubric.yaml.
5. Apply evaluation/safety-gates.yaml before averaging.
6. Decide whether to keep, revise, split, merge, defer, or retire the behavior.
7. Record the result with evaluation/report-template.md.
8. Add a JSON scorecard and register the report in evaluation/reports/index.yaml.
9. Regenerate evaluation/scorecards/summary.json.

Validation

python .\scripts\validate-evaluation.py
python .\scripts\validate-live-evaluation-runs.py
python .\scripts\validate-evaluation-reports.py
python .\scripts\validate-scorecards.py
python .\scripts\summarize-scorecards.py --check .\evaluation\scorecards\summary.json

Or run the full repository suite:

.\scripts\validate-all.ps1

Before beta or broader evidence claims, capture and score the first live foundation smoke packet.