New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark: ignore significance when using --runs 1 #8299

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
4 participants
@AndreasMadsen
Member

AndreasMadsen commented Aug 27, 2016

Checklist
  • commit message follows commit guidelines
Affected core subsystem(s)

benchmark

Description of change

Because the standard deviation can't be calculated when there is only
one observation the R scripts raises an error. However it may still be
useful to run them for non-statistical purposes.

This changes the behaviour such when there is only one observation, the
values that depends on the standard deviation becomes Not Applicable (NA).

Fixes: #8288

/cc @mscdex


examples:

                                                               improvement significant   p.value
 http/check_is_http_token.js n=50000000 key="Content-Encoding"      2.42 %          NA        NA
 http/check_is_http_token.js n=50000000 key="Content-encoding"      1.79 %             0.8908776
 chunks length     rate confidence.interval
      0      4 12834.16            5850.699
      1      4 11944.62                  NA
@addaleax

View changes

benchmark/compare.R Outdated
significant = '*';
p.value = NA;
significant = 'NA';
# Check if there is enogth data to calulate the calculate the p-value

This comment has been minimized.

@addaleax

addaleax Aug 27, 2016

Member

typo: enogth

@addaleax

View changes

benchmark/compare.R Outdated
# Check if there is enogth data to calulate the calculate the p-value
if (length(old.rate) > 1 && length(new.rate) > 1) {
# Perform a statistics test to see of there actually is a difference in
# performace.

This comment has been minimized.

@addaleax

addaleax Aug 27, 2016

Member

typo: performace

@addaleax

This comment has been minimized.

Member

addaleax commented Aug 27, 2016

Can’t really review R code, so rubber-stamp LGTM from me.

@mscdex

This comment has been minimized.

Contributor

mscdex commented Aug 27, 2016

I can't really review the code either, but one minor nit is the last line of the commit message is a tad too long (>72).

benchmark: ignore significance when using --runs 1
Because the standard deviation can't be calculated when there is only
one observation the R scripts raises an error. However it may still be
useful to run them for non-statistical purposes.

This changes the behaviour such when there is only one observation, the
values that depends on the standard deviation becomes Not Applicable
(NA).
@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Aug 27, 2016

Yeah, I'm not amazing at R either :)

@addaleax

This comment has been minimized.

Member

addaleax commented Aug 27, 2016

Hmm… we could give pinging the rest of @nodejs/math aka @fhinkel a try? 😄

@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Sep 15, 2016

ping

@addaleax

This comment has been minimized.

Member

addaleax commented Sep 16, 2016

If the count remains at 1 rubber-stamp approval (mine), I’d say it’s okay to land this.

I know you’re busy these days, so please don’t feel pressured, but if you’re ever up for porting some of the R code to JS — and if you think that’s feasible – that might be kind of nice for attracting more reviewers. :)

@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Sep 16, 2016

@addaleax Great, I will land it then.

I know you’re busy these days, so please don’t feel pressured, but if you’re ever up for porting some of the R code to JS — and if you think that’s feasible – that might be kind of nice for attracting more reviewers. :)

I have already written ttest, so it is feasible. The problem is that it's a big dependency and it's actually super hard to write a correct t-test implementation. When I wrote ttest I checked all other js implementations and found at least on error in each, so I have every reason to expect that ttest is buggy too.

@AndreasMadsen

This comment has been minimized.

@addaleax

This comment has been minimized.

Member

addaleax commented Sep 16, 2016

The problem is that it's a big dependency

Yeah, that’s a general problem we have in core. Maybe we should just get a package.json for our top level directory or something…

@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Sep 16, 2016

Landed in d3834a1

@gibfahn gibfahn referenced this pull request Jun 15, 2017

Closed

Auditing for 6.11.1 #230

2 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment