Support for Statistical Aggregation (mean, variance...) across dataset runs

[Leo310]

Describe the feature or potential improvement

I am requesting a feature to enhance the evaluation of non-deterministic systems, particularly those with LLMs. Due to the inherent variability of these models, a single-run evaluation score is insufficient for accurately assessing system reliability.

To provide a statistically robust assessment of system performance and stability, I propose adding support for calculating and tracking aggregated statistical measures over multiple evaluation runs against the same test set.

Core Requirements

1. Statistical Metrics: Calculate and display the following metrics for evaluation scores across a series of runs:

   • Mean
   • Standard Deviation
   • Coefficient of Variation

2. Aggregation Grouping: Allow users to explicitly group multiple evaluation runs so that the system knows which scores to aggregate.

3. Visualization: Present the score distributions clearly in the UI using histograms or box plots alongside the numerical statistics. This will help users visualize the variance and shape of the score distribution.

Benefits

This capability is crucial for:

• Quantifying the uncertainty in a system's performance.

• Establishing a reliable baseline for performance metrics.

• Tracking system stability and variance across different versions (e.g., before and after a model fine-tune) and making the reported evaluation scores statistically meaningful.

Additional information

No response

[marliessophie]

Hey, thanks for sharing feedback.
We have a related discussion post tracking request for allowing iterations on experimetns: https://github.com/orgs/langfuse/discussions/5855. Once we have this, that should unlock your request around calculating mean, SD, etc. Hope this helps.