feature request: two-sample t-test

[DVeron-RC]

Would it be possible to add a two-sample t-test to tell if 2 samples are statistically different ?
For example, consider the case where you have two long running branches: master and next which regularly run the same bench like https://bencher.dev/perf/git?lower_value=false&upper_value=false&lower_boundary=false&upper_boundary=true&x_axis=date_time&branches=595859eb-071c-48e9-97cf-195e0a3d6ed1%2C37f822b4-6514-4709-988b-b5fd9e713872&testbeds=02dcb8ad-6873-494c-aabc-9a6237601308&benchmarks=5e5c6ae1-ec8e-4c25-b27d-dcf773d33a51&measures=63dafffb-98c4-4c27-ba43-7112cae627fc&tab=branches&plots_search=0d7f6186-f80a-4fbe-9022-75b6caf5164e&key=true&reports_per_page=4&branches_per_page=8&testbeds_per_page=8&benchmarks_per_page=8&plots_per_page=8&reports_page=1&branches_page=1&testbeds_page=1&benchmarks_page=1&plots_page=1&end_time=1745971200000&start_time=1740787200000&utm_medium=share&utm_source=bencher&utm_content=img&utm_campaign=perf%2Bimg&utm_term=git&clear=true&heads=%2C5e4f5a49-5dfa-4aa0-bd0f-d40de2fc7004

I think here a question a dev can ask themself is whether their feature branch is introducing a performance regression compared to the main one or not.

If I understood them correctly, the current tests performs prediction intervals tests, answering the question whether or not it is reasonable to accept the new point as part of a distribution the previous points are part of.

If you are using start_point, it only extends the population of your previous point and you get your comparison next population versus the master population only for the first point.

Instead I would like all the points produce by the next branch compared with all the point produced by the master one.

Start_point is a nice criteria to tell which points should be flagged as part of the master population and which points should be flagged as part of the next population.

[epompeii]

@DVeron-RC where in your workflow do you imagine a two-sample t-test fitting in?
That is, when would you be comparing both of these populations?

I looked into two-sample t-tests when originally adding the statistical thresholds to Bencher. However, I determined that two-sample t-tests aren't a fit for the current Bencher workflow. Bencher processes results one at a time, comparing against the historical data based on the Start Point.
I'm open to the idea. I just still don't see where two-sample t-tests would fit in.

[DVeron-RC]

I have nightly CI that run the same bench on 2 different branches.
The next branch that aggregate the features for the next release and the main branch where the next branch is merged.

I would like this piece of data when I want to merge my next branch in my master one.

I don't think this test is computation heavy so I think running it as a threshold on every report published on the next branch can be relevant, comparing every points before the start_point against every point after it I think.

[epompeii]

Thank you for the context!

Would doing the following work for you, on each push to next:

1. Comparing the most recent push to next against the most recent history of next
2. Comparing the most recent push to next against the most recent history of master

Until #187 is implemented, this would require you to save your results to a file and then provide them to two separate bencher run invocations. This approach does reduce the statistical fidelity some compared to a two-sample t-test if you have say 100 results on next. You'll only be comparing the latest not all 100. However, this keeps with the current paradigm of processing one result at a time. With Bare Metal runners, the fidelity should still be sufficient.

[DVeron-RC]

Thank you for your proposition.

As you pointed out only using one point will not significantly improve statistical fidelity.
I think comparing two long running branches is a common use case,
and to do so you will probably need to compare 2 samples over comparing a point against a sample.
So I feel your proposition would help, in the scenario where the next branch slowly regressed;
but it is a missed opportunity.

[epompeii]

So I feel your proposition would help, in the scenario where the next branch slowly regressed;
but it is a missed opportunity.

For slow regressions, I'm targeting adding change point detection: #365

Would that work for your use case?

[DVeron-RC]

Your propositions are great improvement for bencher imo, and if change point detection was present I would be using it.
I think it is useful for feature branches or within next branch, where as a dev I want to know when a regression was introduced.

However, when a release is cut I do not care for change points, what only matters is to tell whether the next metrics are better/worse/same than the master metrics.
I think that in the scenario of long running branches, 2 samples test will provide better statistical fidelity than change point detection.