docs(evaluation): Understanding Evaluation concept#711
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| <Card title="Judge models" icon="robot" href="/docs/evaluation/concepts/judge-models"> | ||
| The models that read and score a response | ||
| </Card> | ||
| <Card title="Scores & results" icon="chart-mixed" href="/docs/evaluation/concepts/eval-results"> |
There was a problem hiding this comment.
concepts/eval-results is gone after the remap on the base branch, this 404s now. Point it at /docs/evaluation/concepts/output-types and title the card Output types
| --- | ||
|
|
||
| ## About | ||
| Evaluation is Future AGI's quality measurement layer. You define what "good" means once, and the platform scores every response against that definition, returning a result and a plain-language reason for each one. Because the same definition runs everywhere your work lives, a score means the same thing on a dataset, a simulation, a live trace, or a pull request. |
There was a problem hiding this comment.
Concept pages open directly at the first H2, no hero paragraph (same shape as the Observe concept pages). Fold this into the description and the first section. Also "a plain-language reason for each one": the reason is optional, don't promise it on every response
| Evaluation is Future AGI's quality measurement layer. You define what "good" means once, and the platform scores every response against that definition, returning a result and a plain-language reason for each one. Because the same definition runs everywhere your work lives, a score means the same thing on a dataset, a simulation, a live trace, or a pull request. | ||
|
|
||
| Evaluation in Future AGI is a systematic way to measure whether your AI is producing the right outputs. You define what "right" means once, using an eval template, and the platform scores every response automatically against that definition, returning a result and a reason for each one. | ||
| ## The four objects |
There was a problem hiding this comment.
This is the anchor concept and it has no diagram or example. Add the one Mermaid (template -> config -> run -> score) and one short worked example, one groundedness template on one row is enough
| 3. **Pick a judge model**: Choose a Future AGI model (e.g. `turing_flash`) or bring your own via a custom model integration. The judge reads each row and applies the template criteria. | ||
|
|
||
| 4. **Run**: The platform processes every row in parallel. Each row gets a result (pass/fail or a score) and a reason explaining the judgment. | ||
| - A **template** defines *what* to measure: the criteria, the expected output type, and a pass threshold. Templates are reusable and versioned, and are either built by Future AGI or written by you |
There was a problem hiding this comment.
Link first mentions: templates -> eval-templates here, and optimize -> /docs/optimization in the quality loop section. Judge model is linked, these should be too
2 participants
The four-object data model (template → config → run → score), the quality loop, and the three eval types. Retitled to 'Understanding Evaluation', no
## About, 156. Draft — batched Evaluation revamp.🤖 Generated with Claude Code