Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Allow to choose the out-of-bag scoring metric #3455

Open
arjoly opened this Issue Jul 20, 2014 · 7 comments

Comments

Projects
None yet
4 participants
Owner

arjoly commented Jul 20, 2014

Estimators in the forest module (random forest and extra trees) and in the bagging module allows to compute the out-of-bag estimates of the performance of the forest.

A nice things to add would to allow the choice of the scoring function using the scorer interface. The oob_score parameter would be equal would be the string corresponding to the appropriate scorer.

Thus, you would have

oob_score : bool or string, (default=False)
        Whether to use out-of-bag samples to estimate
        the generalization error. The oob scoring function could be chosen
        by passing  a string (see model evaluation documentation) or
        a scorer callable object / function with signature
        ``scorer(estimator, X, y)``.

Owner

glouppe commented Jul 20, 2014

In fact, one of the things that I don't like with the current implementation of oob scores is that they don't at least rely on the underlying score method of the ensemble. By default, a better implementation should do that (instead of reimplementing the zero-one loss and the squared error loss in the forest module), or use a given scorer, if any is provided, as you suggests.

Owner

glouppe commented Jul 20, 2014

Also, I think such a refactoring should go in pair with #3436, which again adds boilerplate code for reimplementing the zero-one loss and the squared error loss.

Owner

arjoly commented Jul 20, 2014

+1 for the refactoring

Contributor

staple commented Sep 28, 2014

Hi - Is this ticket available? I'm interested in working on it.

Owner

arjoly commented Sep 29, 2014

It's available.

@staple staple added a commit to staple/scikit-learn that referenced this issue Sep 30, 2014

@staple staple [WIP] Choose out of bag scoring metric. Fixes #3455 265ffc3
Contributor

staple commented Sep 30, 2014

Hi - I created a wip, please let me know if this approach with a DummyPredictor seems reasonable, see description in the pull request: #3723

@staple staple added a commit to staple/scikit-learn that referenced this issue Oct 7, 2014

@staple staple [WIP] Choose out of bag scoring metric. Fixes #3455 b502e59

bmwilly commented Jul 13, 2017

Is there any plan on merging this? I can't find any discussion more recent than this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment