New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
More understandable ips comparison #49
More understandable ips comparison #49
Conversation
Good call on it not being clear. I worry about changing the number displayed there only because it might confuse existing users, which seem to have understood my intention despite my bad wording. Perhaps we leave the number the same and just remove he word "slower"? |
👍 I always do If you're worried about backwards compat, we could preserve an option call it |
@schneems ok! I'm happy to display more info as well, to try and make it clearer. So we could show the current value as well as a subtracted one. I want the feature to be clear, so whatever we have to do to make that possible, we should do it. |
How about something like this?
|
I was considering doing the comparisons against the slowest run, and showing how many percent faster the other runs were:
It seems to me that when doing performance work, you want to know how much faster you can make a piece of code. Not necessarily how much slower the old code was. Correct me if I’m wrong. |
I like that, I'm usually looking for speed improvements. Not sure about others though. |
da37d1f
to
458f6c5
Compare
Thanks. Maybe related or maybe not. Sometimes, 2 different methods are about the same speed. standard deviation may be able to tell us this. What algorithm can we use to have them say "basically the same"? |
Use the stddev. If the difference is within the the stddev, they are the same speed. On Tue, Aug 11, 2015 at 1:30 PM, Keenan Brock notifications@github.com
|
I'd prefer to keep it showing the fastest and then how much slower the others are. Change that would be definitely confusing to existing users. |
I had looked into including the std dev in the calculations. So I could see if some were the same. But had trouble making the information clear when showing a range. Almost feel like it could be: |
I would like to move the discussion back to the main topic again: The current I think that it would be most confusing for existing users to only slightly change the existing output. Keeping the Less confusing would be to completely change the output. I think changing it to either of the proposed Since it has already been mentioned that such changes might cause a major version bump: I’m wondering if there may be a need for custom output formatters? Should it be possible to choose between the “% faster than baseline” or “% slower than baseline” formatter by using a command line flag? |
We could use |
When two pieces of code run the same number of iterations in the same amount of time, that shouldn't be reported as one being "1.00x slower" than the other. By comparing all slower results against the fastest result, and not the other way around, we get some more realistic reporting.
458f6c5
to
57ba971
Compare
Take a look at 7961c61 and give me your feedback please! |
Why was this closed? |
No one responded to my feedback request 8 months ago. |
Or, rather, you did but the change seemed fine. |
When two pieces of code run the same number of iterations in the same amount of time, that shouldn't be reported as one being "1.00x slower" than the other. By comparing all slower results against the fastest result, and not the other way around, we get some more realistic reporting.