Performance Regression Testing

[neelabhg]

When working on this project, I'm hesitant to make changes because there doesn't seem to be a way to ensure that any changes 1. don't break things, and 2. don't degrade performance. There are some basic tests currently present, but I think we should add more functional/integration tests to the project.

One place where I think a discussion is necessary is how to ensure the VM stays performant. It'd be great if we could track VM performance as changes are made and this project evolves. Something like the Chrome Performance Dashboard. These performance tests and metric tracking should happen automatically, perhaps during continuous integration testing.

Here are some relevant issues I found:

• [Feature Request] Continuous Performance Testing in travis-ci travis-ci/travis-ci#352
• [Feature Request] Performance Testing travis-ci/travis-ci#3313
• Performance test tqdm/tqdm#51

Whenever I search for "web performance testing" or "js performance testing", I encounter tools and tips for profiling and optimizing "front-end" things, such as layout rendering, painting, CSS animations, scrolling, etc. These things are important, but I'm not sure which metrics we should focus on. In any case, it wouldn't hurt to monitor/test some front-end metrics. For this, the following tools look promising:

• browser-perf
• perfjankie
• phantomas

Having a "performance dashboard" will help contributors make changes with confidence, and provide interesting history to track jor1k's performance over time. We could also later add some benchmarks for comparison with other emulators, with the idea being that the benchmarks are run upon every commit in this project.

The first step needed is to define some metrics that we should track. Some of these metrics may also be helpful for #24. So, any insights/starting points?