Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scripts: add Welch's T test #222

Merged
merged 4 commits into from
Oct 7, 2019
Merged

scripts: add Welch's T test #222

merged 4 commits into from
Oct 7, 2019

Conversation

uetchy
Copy link
Contributor

@uetchy uetchy commented Oct 7, 2019

Add a Python script that consumes the JSON output of hyperfine and calculates the probability of welch's t test.

Can edit PR freely.

Fixes #217

@sharkdp
Copy link
Owner

sharkdp commented Oct 7, 2019

Very cool, I love it!

I made a couple of small changes. The output looks like this:

▶ python welch_ttest.py almost-the-same.json 
Command 1: sleep 0.1000000
Command 2: sleep 0.1000001

t = 0.2, p = 0.842

The two benchmarks are almost the same (p >= 0.05).
▶ python welch_ttest.py slightly-different.json 
Command 1: sleep 0.100
Command 2: sleep 0.101

t = -4.33, p = 6.75e-05

There is a difference between the two benchmarks (p < 0.05).

@sharkdp sharkdp merged commit 94d458c into sharkdp:master Oct 7, 2019
@uetchy uetchy deleted the welch-ttest branch October 8, 2019 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Calculates whether the distributions of two benchmarks are different
2 participants