Performance Testing for EvalML

[jeremyliweishih]

As we flesh out functionality for EvalML, it will be imperative to monitor performance across test datasets to ensure regressions do not occur as we continue to develop.

Metrics to consider:

• Time to run
• Performance metrics (R2/Acc/etc.)
• Variance of pipelines

Datasets to consider:
https://www.openml.org/home

[dsherry]

Historical note for any later readers: we changed the name "integration test" to "performance test" recently.

[dsherry]

Design document for the 2020 Q1/Q2 performance testing project is here

[dsherry]

@jeremyliweishih is going to grab this for the May milestone (AKA next month). We'll use this week to scope out what we want to go into that milestone, and what future work we'll have after that point. Exciting!

[dsherry]

This is left over from the perf test prototype work. Closing now that we have V2 (the looking glass project) under way.