automate benchmarking

[orbeckst]

Performance benchmarking should be run continuously.

Current plan

• use a CRON job on c3potato.beckstein.physics.asu.edu
• publish benchmark data in this repo in the master branch
• publish benchmark html/js in this repo in the gh-branch (which shows up as https://www.mdanalysis.org/benchmarks/)

Background

• current manual procedure
• astropy/astropy-benchmarks have developed a cron script cron.sh that we can adapt

[orbeckst]

Note that astropy does something clever (or confusing): the master branch are the actual benchmark tests (our mdanalysis/benchmarks), the results branch contains results (our benchmarks master), and any commits to their result branch trigger Travis, which automatically deploys the asv web site page to the gh-pages branch.

[orbeckst]

It would be useful if the automated benchmarks did not re-run benchmarks that were already run. Ideally,

• recognize commits that have not been benchmarked and only run those
• recognize benchmarks that have not been run and only run those

[orbeckst]

Possibly use --skip-existing (from asv run --help):

  --skip-existing-successful
                        Skip running benchmarks that have previous successful
                        results
  --skip-existing-failed
                        Skip running benchmarks that have previous failed
                        results
  --skip-existing-commits
                        Skip running benchmarks for commits that have existing
                        results
  --skip-existing, -k   Skip running benchmarks that have previous successful
                        or failed results

[orbeckst]

Or use (see https://asv.readthedocs.io/en/latest/using.html )

asv run NEW

You can benchmark all commits since the last one that was benchmarked on this machine. This is useful for running in nightly cron jobs:

[orbeckst]

crontab is running daily 3:30am AZ = 10:30 GMT so results should become available a few minutes to a few hours later (depending on how many new commits/benchmarks were added).

If something is not working properly, please open an issue.