Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent timing results #257

Closed
2 tasks done
tyronerees opened this issue Sep 26, 2019 · 1 comment · Fixed by #285
Closed
2 tasks done

Inconsistent timing results #257

tyronerees opened this issue Sep 26, 2019 · 1 comment · Fixed by #285

Comments

@tyronerees
Copy link
Member

tyronerees commented Sep 26, 2019

Running the same test, on the same dedicated VM, two days apart gives the following runtime tables:
180919
190919

At the moment, there is too much noise in the times to make any judgement from these tables. In order to improve this, we should:

  • run the benchmarks a number of times (five by default, but make this a control parameter), and discard the highest and lowest1
  • consider using the timeit module, instead of time.clock, as it's claimed to be more robust at benchmarking, especially across platforms. This will do the averaging in the previous point naturally see docs.

1 some minimizers, e.g., the Differential Evolution (de ) minimizer in bumps, will be stochastic, and will naturally give different runtimes. This should be noted somewhere in the table.

@tyronerees tyronerees added Priority - high block This must wait for an open issue to be fixed labels Sep 26, 2019
@tyronerees
Copy link
Member Author

Needs #263 to be addressed before fixing

@tyronerees tyronerees removed the block This must wait for an open issue to be fixed label Oct 30, 2019
@wathen wathen moved this from To do to Review in progress in ALC Project Sprint #2 Nov 5, 2019
@tyronerees tyronerees moved this from To do to In progress in ALC Maintenance Nov 2019 Nov 11, 2019
@wathen wathen moved this from In progress to Reviewer approved in ALC Maintenance Nov 2019 Nov 12, 2019
ALC Project Sprint #2 automation moved this from Review in progress to Done Nov 12, 2019
ALC Maintenance Nov 2019 automation moved this from Reviewer approved to Done Nov 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

1 participant