New (Preview) Feature Released: Workflow Evals

[HighnessAtharva]

Workflow evals are now in preview. You can run one workflow on several models, have a judge model score each run, and compare the results side by side.

Open Workflows, pick an agent and a workflow, then switch from Graph to Evaluations.

What You Get

Each evaluation shows one row per model:

• Score: a 0 to 100 result for the run.
• Correctness: the judge's grade out of 4.
• Judged efficiency: the judge's grade out of 4 for how directly the workflow reached the result.
• Measured efficiency: a number computed from the run itself.
• Tokens, Calls and Duration: the real usage of each run.

Three charts sit under the table:

• Score: one bar per model.
• Score vs. usage: score plotted against tokens, calls or time.
• Model comparison: correctness, judged efficiency and measured efficiency side by side, with a switch to usage.

Read the Judge's Reasoning

Click a model to open its judgment. The Judgment tab gives the correctness and efficiency grades, a short summary and a list of findings. Each finding has a View step link that jumps to the step in the run. The Transcript tab holds the full run.

In the example above, the judge marked a run down on efficiency because a validation check failed before a read confirmed the report.

Start a New Evaluation

Click New evaluation and set:

1. Models: the models to compare.
2. Judge: the model that scores the runs. Use your strongest model.
3. Parallel runs: how many runs go at the same time. Parallel runs can modify the same files or services, which can affect judging.
4. Run timeout (seconds): the default is 900.
5. Inputs: the workflow inputs as JSON.
6. Additional judge instructions: optional extra guidance for the judge.

Click Run evaluation.

Judge Again Without Rerunning

Click Judge again, pick a different judge model and click Judge executions. The judge reads the saved transcripts, so the workflow does not run again. Use this to check a score with a second judge or to try a stricter one.

Other controls on the page: View inputs, Scoring and Delete.

Tell Us What You Think

This feature is a preview, so scoring and layout can change. Reply here with:

• the workflows you evaluate
• the models you compare
• any score that looked wrong to you