Core Concepts

Core Concepts

Docs-site version: Core concepts. Deep dive on evaluation: guides/EVALUATION_GUIDE.md.

The data

Term	Meaning
Span	One OTLP span. Typed by observation.type: AGENT, TOOL, GENERATION (an LLM call), THINKING, RETRIEVER, GUARDRAIL, CHAIN, DELEGATE, SKILL, …
Trace	One agent run — a root span and its children. In chat products, one turn.
Conversation (thread / session)	Traces sharing a conversation id. The unit most evaluators and the gate care about.
Agent	The thing being gated, clustered and trended. Declared by tracely.agent.id; auto-registered on first trace. One codebase serving many customers uses tenant= so each tenant is its own agent.
Env	prod / staging / ci / dev. Production failures become cases; ci traces are what the gate grades.
Internal runs	Tracely records its own work (evaluations, scenario runs, assistant turns) as traces with internal_kind set. They are never evaluated and are hidden from every list and metric except behind the Evals filter chip.

Evaluation

Evaluators are columns on the trace table, not a separate tab. Each one grades at a level and writes a score into the grid, streamed live over SSE.

	Structural	LLM judge
What	Deterministic checks: run_outcome, tool_success, tool_consistency, latency (+ budget), required_tools	A rubric prompt graded by a model (score / boolean / JSON with schema)
Needs a model key	No	Yes — the workspace's own OpenRouter key (Settings). No key → judges skip, nothing crashes.
Levels	Fixed per check	CONVERSATION, AGENT_RUN, SPAN, TOOL, GENERATION, CHAIN

Other knobs: basic mode (context auto-injected) vs advanced (@VARIABLE template prompts, e.g. @HISTORY, @LIST_AGENT); batch vs sequential execution; per-evaluator targeting (target_agent, target_env) and deterministic sampling; advisory evaluators record a verdict without flipping the roll-up.

The one verdict policy: a trace / turn / conversation fails iff it has a FAIL on a non-advisory evaluator. Python and SQL implement it identically (domain/evaluation/verdict.py and async_reader).

Calibration: label judge verdicts against human review on the Calibration page → per-evaluator agreement, false_pass / false_fail rates. Check an evaluator's false_fail before letting it gate a release.

Failure intelligence

Clusters group failing runs by structural and semantic similarity (embeddings in pgvector), so 31 broken runs become one issue with a count, an analysis, a suggested fix and a severity. Clusters are derived — deleting one just prunes noise; a later Analyze re-forms any issue whose failing traces still exist.

Regression

A case is a failing trace frozen into a hermetic fixture bundle: recorded input, recorded tool/LLM outputs, reference trajectory, plus a fail-to-pass contract (it must fail on the code that produced it, or it stays a DRAFT). Promote a single trace or a whole cluster (its most representative member).

Gate

A gate run grades an agent on a commit/PR and posts a blocking commit status + PR comment. Three inputs — see CI Gate:

• Scenarios — conversations Tracely drives against your agent's registered HTTP endpoint. SCRIPTED (turns with optional expect / tools) or ADVERSARIAL (a goal an attacker model pursues — goal achieved = FAIL).
• Replay — re-run your code on each promoted case's recorded input with recorded fixtures. Offline, $0.
• Gate — grade the env=ci traces your pipeline already emitted.

A gate that checked nothing is never green: ungraded conversations count against the pass rate, and an all-ungraded suite reports NO_COVERAGE and blocks.

Insights

Trends (daily traces vs failures, gate pass rate, latency percentiles, token spend) and per-agent meta-analysis (Spearman correlations + z-score outliers, LLM-synthesized). Rolling summary is per-turn conversation memory that backs the judge's @HISTORY. Alerts fire on events (gate_failed, trace_failed, cluster_new) or polled thresholds and run a small flow of steps (condition → Slack / email / webhook / LLM prompt).