Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 67 additions & 24 deletions src/pages/docs/evaluation/index.mdx
Original file line number Diff line number Diff line change
@@ -1,45 +1,88 @@
---
title: "Future AGI Evaluation: Measure Prompt and Agent Quality"
description: "Measure and compare the quality of prompts and agents across datasets, simulations, and experiments using built-in or custom eval templates."
title: "Overview"
description: "Measure how well your AI achieves the user's goal"
---

## About
Evaluation is Future AGI's quality measurement layer: it scores every response your AI produces against a definition of "good" that you control.

Evaluation is Future AGI's quality measurement layer. It gives you a consistent, repeatable way to measure whether your prompts and agents are behaving correctly and whether changes you make improve things or introduce regressions.
## What is evaluation?

There are two building blocks: **eval templates** define what to measure (task completion, tone, hallucination, safety, factual accuracy, or a custom rule you write yourself), and **eval configs** define how to measure (the judge model, input mapping, and run settings). Combine them with your data and you get a score, a pass/fail result, and an optional explanation per row or call, plus aggregated summaries, KPIs, and trend data across runs.
An LLM's output is free text, and whether it is *right* is a judgment. Evaluation turns that judgment into a measurement. You define what "good" means once, as an eval, and the platform applies it to every response automatically, the same way every time.

Evaluations run across every surface in Future AGI: datasets, simulations, experiments, playground, replay sessions, and CI/CD pipelines. You can also run them programmatically via the SDK. Using the same templates and configs across contexts keeps results directly comparable without redefining your quality criteria each time.
Each eval scores one goal on one metric, for example:

Future AGI ships 70+ built-in templates covering quality, safety, factuality, RAG retrieval, format, bias, audio, and image evaluation. You can also create custom templates and bundle any combination into **eval groups** to apply multiple evals in a single run.
- **Task completion**: did the response do what was asked
- **Factual accuracy**: are its claims true to the source
- **Safety**: is it free of toxicity, prompt injection, and data leaks
- **Tone**: does it speak the way your product should

## How Evaluation Connects to Other Features
Every run returns a score (pass/fail, a number, or a category) and, for judged evals, a plain-language reason. Because the definition is fixed, the same bar is applied to every response: scores stay comparable across a dataset, a live trace, or a pull request, and a threshold on that score becomes a decision you can automate instead of a vibe check.

- **Datasets**: Run evals across dataset rows and store scores as new columns. [Learn more](/docs/dataset)
- **Simulation**: Score simulated agent conversations for quality, context retention, and escalation. [Learn more](/docs/simulation)
- **Optimization**: Feed eval results into prompt optimization to improve quality automatically. [Learn more](/docs/optimization)
- **CI/CD**: Gate pull requests on eval scores to catch regressions before they ship. [Learn more](/docs/evaluation/features/cicd)
- **Error Feed**: Eval-powered scoring for every traced agent execution. [Learn more](/docs/error-feed)
## The quality loop

## Getting Started
Evaluation is the middle of a loop, not a standalone tool:

<Mermaid chart={`
%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart LR
O["Observe<br/>what the agent did"] --> E["Evaluate<br/>score it against your bar"]
E --> OPT["Optimize<br/>the prompt or model"]
OPT --> G["Enforce<br/>gate the release"]
G --> O
style E fill:#2f2f2f,stroke:#ffffff,stroke-width:2px
`} />

- [Observe](/docs/observe) records what your agent did, and its traces become eval inputs
- Evaluate scores each response and shows the result back on the trace
- [Optimization](/docs/optimization) consumes the scores to improve the prompt or model
- A threshold turns the score into a merge gate in [CI/CD](/docs/evaluation/features/cicd) before anything ships

## Start here

<CardGroup cols={2}>
<Card title="Evaluate via Platform & SDK" icon="play" href="/docs/evaluation/features/evaluate">
Run the first eval from the UI or SDK in minutes.
<Card title="Running Evaluations" icon="play" href="/docs/evaluation/features/evaluate">
Every surface you can run an eval on, and how to get going
</Card>
<Card title="Built-in evals" icon="tags" href="/docs/evaluation/builtin">
70+ templates: quality, safety, factuality, RAG, and more.
156 ready evaluators across quality, safety, RAG, format, and more
</Card>
<Card title="Create custom evals" icon="wand-magic-sparkles" href="/docs/evaluation/features/custom">
Define your own eval rules and output types.
</CardGroup>

## Understand the model

Four objects carry the whole product:

- A **[template](/docs/evaluation/concepts/eval-templates)** defines what to measure
- A **config** points it at your data
- A **run** executes
- A **[score](/docs/evaluation/concepts/output-types)** comes out

A few short pages give you the whole mental model:

<CardGroup cols={2}>
<Card title="Understanding Evaluation" icon="compass" href="/docs/evaluation/concepts/understanding-evaluation">
The template, config, run, and score model, and the quality loop
</Card>
<Card title="Eval groups" icon="table-rows" href="/docs/evaluation">
Bundle multiple evals and run them together.
<Card title="Eval types" icon="list-check" href="/docs/evaluation/concepts/eval-types">
Agent Evaluator, LLM-as-Judge, and Code Eval, and when to reach for each
</Card>
<Card title="Judge models" icon="robot" href="/docs/evaluation/concepts/judge-models">
Pick the right evaluation model for your task.
What decides the score, and how to choose one
</Card>
<Card title="Output types" icon="chart-mixed" href="/docs/evaluation/concepts/output-types">
The value, optional reason, and aggregates you get back
</Card>
</CardGroup>

## How it connects

Beyond the loop, the same evals run offline: [datasets](/docs/dataset) score rows in bulk, and [simulations](/docs/simulation) score synthetic conversations before anything reaches production.

<CardGroup cols={2}>
<Card title="Score traces in Observe" icon="magnifying-glass" href="/docs/observe/guides/setup-evals">
Attach an eval to a project's traces and score new data as it arrives
</Card>
<Card title="Evaluate CI/CD pipeline" icon="code" href="/docs/evaluation/features/cicd">
Run evals automatically on every pull request.
<Card title="Evaluate in CI/CD" icon="code-branch" href="/docs/evaluation/features/cicd">
Turn an eval threshold into a merge gate that blocks regressions
</Card>
</CardGroup>