[MRG+1] Changes default for return_train_score to False

[thechargedneutron]

Reference Issue

Fixes #9621

What does this implement/fix? Explain your changes.

Changes the default value of return_train_score to warn. Raises a warning if train score takes more time.

Any other comments?

The warning message needs improvement.

[thechargedneutron]

@jnothman Kindly suggest a suitable warning message.
Also please review. I guess this would need a lot of changes, mostly in documentations.

[jnothman]

Parentheses in a strange place here...

And can't compare absolute time to 5 in a single fit. It will often only be a substantial amount of time over a large number of fits, as in the raising issue.

[thechargedneutron]

How about adding a new variable alongside return_train_score in BaseSearchCV, containing the time of the first call to GridSearchCV. Also we need to pass that time variable to the _fit_and_score function.

[thechargedneutron]

@jnothman Added a parameter in the function which takes care of the actual start time of GridSearchCV. Please see if this is valid or not.

[jnothman]

It's not a bad approach, but I suspect it won't receive the consensus to get merge, just because it makes the _fit_and_score interface more messy...

@amueller, what do you think of this approach?

[jnothman]

This still may mean that training score calculation time took only .5 of a second, which I don't think is quite enough. Maybe better would be to (naively) estimate overall training score time: (time.time() - grid_search_start_time) * score_train_time / (fit_time + score_time + score_train_time) > 5..?

[thechargedneutron]

@jnothman I also agree upon the fact that it makes the _fit_and_score method more messy. But I could not find and alternative to keep track of total time that GridSearchCV would take. You or @amueller may suggest a way which serves the purpose without changing _fit_and_score interface.

[jnothman]

The only other way to do it without changing the _fit_and_score interface is to only raise the warning after fitting all estimators. And if we're going to change _fit_and_score's interface, this is a good solution.

…

On 3 September 2017 at 21:21, Kumar Ashutosh ***@***.***> wrote: @jnothman <https://github.com/jnothman> I also agree upon the fact that it makes the _fit_and_score method more messy. But I could not find and alternative to keep track of total time that GridSearchCV would take. You or @amueller <https://github.com/amueller> may suggest a way which serves the purpose without changing _fit_and_score interface. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9677 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz65tS2hXIxc9O6p11nNoBA8BmsN8Kks5seoukgaJpZM4PLCkj> .

[thechargedneutron]

@amueller @lesteve Any suggestions on whether I should go on to change _fit_and_score method interface or would it be better to raise the warning only after the call to the function _fit_and_score ends?

[thechargedneutron]

@jnothman Should I continue with changing the function interface or raise warning only after completing _fit_and_score call ?

[jnothman]

I hope we'll get someone else's opinion soon... I suspect we'll land up with warning at the end, but I'm not sure.

…

[lesteve]

What about the seemingly simpler solution of changing the default to return_train_score=False, potentially by adding a FutureWarning that the default is going to be return_train_score=False in 0.22?

[thechargedneutron]

@lesteve Yes, this is also a simple solution, will be done once others agree upon this.

[jnothman]

I suppose that's an acceptable solution.

…

On 7 September 2017 at 21:47, Kumar Ashutosh ***@***.***> wrote: @lesteve <https://github.com/lesteve> Yes, this is also a simple solution, will be done once others agree upon this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9677 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz63eCrLSWgNrp09pyhuLx8FrI2W7Nks5sf9fKgaJpZM4PLCkj> .

[thechargedneutron]

@jnothman Kindly review and suggest a suitable FutureWarning message.

[thechargedneutron]

Suggestions for a "nicer" warning message. Otherwise, I think this will work. :)

[thechargedneutron]

Even after doing the required changes (change in test still not made), the following lines of code does not produce a deprecation warning, is it how DeprecationDict is supposed to behave or I am missing out something.

from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV
iris = datasets.load_iris()
parameters = {'kernel':('linear','rbf'), 'C':[1,10]}
svc = svm.SVC()
clf = GridSearchCV(svc, parameters)
clf.fit(iris.data, iris.target)
sorted(clf.cv_results_.keys())

[lesteve]

Warning should be produced when accessing the cv_results_ key for training scores:

from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV
iris = datasets.load_iris()
parameters = {'kernel':('linear','rbf'), 'C':[1,10]}
svc = svm.SVC()
clf = GridSearchCV(svc, parameters)
clf.fit(iris.data, iris.target)
clf.cv_results_['split0_train_score']  # this is the line that should produce a warning

Note that you should not get any warning if you use return_train_score=True.

from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV
iris = datasets.load_iris()
parameters = {'kernel':('linear','rbf'), 'C':[1,10]}
svc = svm.SVC()
clf = GridSearchCV(svc, parameters, return_train_score=True)
clf.fit(iris.data, iris.target)
clf.cv_results_['split0_train_score']  # no warning because return_train_score is set to True

Basically what we want users who do not set return_train_score and access cv results train score to get a warning and tell them they should set return_train_score to True because training scores will not be present by default in 0.20.

[thechargedneutron]

I am not sure how DeprecationDict is working. Need help on how to implement.

[lesteve]

You need to help us help you!

Can you be specific about your problem is? what have you tried? If there is a failure that you don't understand, can you copy and paste it here?

[jnothman]

See my PR at thechargedneutron#2

[lesteve]

I restarted the failing build in Travis hoping the timeout was just a glitch.

[jnothman]

Do you think this approach is decent, @lesteve? Too messy? It implies that a user no longer gets a warning informing them why the fit is so slow which was the original point...!

[lesteve]

I have not looked at the diff of this PR (will do shortly). To be perfectly honest, I think your DeprecationDict suggestions is really neat and this is the best we can do:

• we tried to have a warning only if the scoring for train was slow and we failed because it was a bit messy to implement. Personally I think return_train_score=False is a reasonable default (compute more things only if asked by the user).
• I think the DeprecationDict approach is the best way to transition from return_train_score=True to return_train_score=False. People will get the warning if they do not set return_train_score and access and then they can decide what they really want.
• In 0.21 return_train_score=False will be the default so the code will be fast by default.

[lesteve]

I think this message should be changed slightly to say something along the lines of "Looks like you are using training scores, you want to set return_train_score=True because return_train_score default will change in 0.21"

[lesteve]

@jnothman I pushed two main changes, it would be nice to have your opinion on these:

• I simplified the warning message, since it only happens when accessing a training score
• DeprecationDict is only used when return_train_score='warn'. This is me being overly cautious/pessimistic mainly and preferring to use a plain dict when return_train_score is set explicitly. I can revert this if you feel this is too much.

I am going to reset the LGTM count and add a +1 from me.

[amueller]

shouldn't we assert that there is no warning for the other vals and that there is no warning for the other keys for 'warn'?

Otherwise LGTM.

[jnothman]

Thanks all