Add add_to_leaderboard() to AutoSearchBase

[jeremyliweishih]

Fixes #628.

[codecov]

Codecov Report

Merging #874 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #874   +/-   ##
=======================================
  Coverage   99.75%   99.75%           
=======================================
  Files         195      195           
  Lines        8532     8585   +53     
=======================================
+ Hits         8511     8564   +53     
  Misses         21       21           

Impacted Files	Coverage Δ
evalml/automl/auto_search_base.py	98.00% <100.00%> (+0.08%)	⬆️
evalml/tests/automl_tests/test_autobase.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 511f6e9...a29e532. Read the comment docs.

[jeremyliweishih]

After speaking to @dsherry - we will wait on #719 and see how that affects the evaluate API before merging this in and updating this API.

[dsherry]

@jeremyliweishih I think its fine to merge this before #719--but we should plan to get another PR up once that's merged.

[dsherry]

@jeremyliweishih looks good!

So now users could do:

automl = AutoClassificationSearch(...)
automl.search(X_train, y_train)
custom_pipeline = CustomPipeline({'custom': 'parameters', ...})
autom.add_to_rankings(pipeline, X_train, y_train)

@jeremyliweishih I have a few more thoughts we should address before merging:

• I left a comment about renaming the main method to add_to_rankings or similar, since we don't use the word "leaderboard" anywhere. That name was my idea originally haha oops
• We should add a check that the data provided here matches the training data passed to search, right? It feels inappropriate to allow other data to be used. Otherwise the CV scores aren't fully comparable. We could check the input data shape--in particular the number and name of the columns should be identical. We could also compute/store/compare a hash of the X and y data, but that may be too much to add for now.
• Add a test to ensure passing a trained vs untrained pipeline into this method doesn't produce different results
• What happens when the user tries to add a pipeline which is identical to one which was already added? Will this create a new entry in the results structure and in full_rankings? Ideally it would not. Suggested change: check if the pipeline class/params have already been evaluated and return early if so.
• As in search, what happens if X or y are np/list type? Should get converted to pandas df

[dsherry]

Looks good! I left a comment on the impl return value and a proposal on how to simplify it. Let's close that conversation, but otherwise this is ready to go.

[dsherry]

👍 thanks