With the new evals comparison feature, it will be great to view aggregated metrics for the different runs for quick comparison, instead of just comparing metrics for each test case. #1642 will add summary metrics to evaluators. Once that's available, these can be added in evals comparison view.
Needs design.