Hyperparameter optimization

[acrellin]

Add estimator hyperparameter optimization capability; misc. performance improvements and bug fixes. Fixes #103

[acrellin]

Still need to improve the web UI, but this works for now.

[bnaul]

When is this a list of dicts versus a dict? Trying to wrap my head around the different use cases...

[acrellin]

It never will be in the webapp usage, but package-API users who know a bit about hyperparameter optimization may want to utilize this - check out http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.GridSearchCV.html

[acrellin]

Sphinx is suddenly failing in the Drone build... All tests passing locally though.

[bnaul]

👹

Did you rebase everything after the latest merge of doc stuff? The error I see in Drone shouldn't be happening with the new import trickery.

[acrellin]

I agree - but only the last few commits are failing and I definitely didn't
change anything Sphinx or Drone-related... Take a look at
http://nipy.bic.berkeley.edu:8080/github.com/mltsp/mltsp/hyperparameter_optimization/5a5e0a30c3c181c27168930add729f417c44525c
Then after that it started failing.

On Tue, Dec 22, 2015 at 2:30 PM, Brett Naul notifications@github.com
wrote:

[image: 👹]

Did you rebase everything after the latest merge of doc stuff? The error I
see in Drone shouldn't be happening with the new import trickery.

—
Reply to this email directly or view it on GitHub
#128 (comment).

[bnaul]

Does anything happen if you git pull --rebase [your_upstream] master locally? Something with this branch being out-of-date seems by far the most likely culprit.

[acrellin]

Ok, just rebased again and pushed - looks happy now.

[acrellin]

What do you all think of the idea of the optimize-hyperparameters-and-fit-to-training-data function returning the the best-fitting estimator only, as opposed to the GridSearchCV object, which contains the estimator as an attribute and implements the predict and predict_proba methods? I can see arguments for both sides, like easier access to the classes_, etc. attributes vs. retaining potentially useful info like grid_scores.

[bnaul]

Yikes...guess there's no way for this to be simple but maybe it would help a bit to define ast.literal_eval(model_params[k]) before the if block. Also, should there be an all around the last list comprehension?

[bnaul]

@acrellin I can also see arguments for both sides but I guess I'd vote for returning the GridSearchCV object, it's easy enough to extract the model but going the other direction would be impossible.

[bnaul]

This looks good to me 👍; wanna rebase so we can see that sweet green 💵?

[bnaul]

This failed the first time for some reason (said RethinkDB wasn't running) but re-building fixed it.

[acrellin]

I've seen that before, too... A bit troubling

[bnaul]

if isinstance(model_params, dict):
    model_params = [model_params]
for param_dict in model_params:
    ...

[acrellin]

All tests are passing on my machine (both with Python 3.5 and a clean 2.7 virtual env), and none of the relevant data files have changes between http://nipy.bic.berkeley.edu:8080/github.com/mltsp/mltsp/hyperparameter_optimization/6761809af2e20bafe7c3ec7b9b415047382f7134 and http://nipy.bic.berkeley.edu:8080/github.com/mltsp/mltsp/hyperparameter_optimization/fa3c460e84b35c9f2c28f80f80110efc2595a3af. Anyone have any ideas re: the sudden xray errors?

[acrellin]

@stefanv @bnaul Please take a look and let me know what you think when you have a chance.

[acrellin]

http://nipy.bic.berkeley.edu:8080/github.com/acrellin/mltsp/hyperparameter_optimization/240726d4cfab818e58eaa19008b2bc77ce73bcd5

[bnaul]

👍 @stefanv ?

[stefanv]

Here, how about:

def robust_literal_eval(value):
    try:
        return ast.literal_eval(value)
    except:
        return value

and then

d = {k: robust_literal_eval(v) for k, v in some_dict.items()} 

[stefanv]

👍 looks really good--thanks for your patience with us, Ari :)