Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Account median testing time #1153

Closed
wants to merge 5 commits into from

Conversation

sanmai
Copy link
Member

@sanmai sanmai commented Mar 11, 2020

Goal is to show users why MT is slow on their project.

We mostly have every class covered by a single test. Therefore for us MT is fairly quick. Median time is on range 0.01 s. which is very good, if my calculations aren't wrong. Basically this means that for the half of all mutations there's only 0.01 s. of tests to check if it escapes, not including test dependencies.

Yet there could be outliers, hence single metric won't work. Min, mean, sd, median, and max for the values needed to be shown. 68–95–99.7 percentiles are of interest too.

This PR:

Report of interest from ApacheBench:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4   81  71.5     49     222
Processing:     1   18  10.5     17      53
Waiting:        0   12   9.5      9      52
Total:          7   99  80.4     67     270

Percentage of the requests served within a certain time (ms)
  50%     67
  66%     82
  75%    170
  80%    179
  90%    244
  95%    250
  98%    258
  99%    258
 100%    270 (longest request)

@sanmai
Copy link
Member Author

sanmai commented Oct 17, 2020

I don't think this is any longer relevant post #1171.

@sanmai sanmai closed this Oct 17, 2020
@sanmai sanmai deleted the pr/2020-03/time-to-test branch October 17, 2020 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant