[MRG+2] Fix excessive memory usage in random forest prediction

[mikebenfield]

Fix #8244.

That issue was about random forests using excessive memory in predicting. The problem was that the forest held onto each prediction from each estimator, only to combine them at the end. It was suggested in the thread to solve this by parallelizing over the instances instead of the estimators. Instead, I still parallelize over the trees, but keep an array out into which I immediately sum each output. Python's GIL makes this thread safe.

My testing indicates runtime performance is the same, but memory usage is substantially decreased. See this gist. Running python prob_classification.py n gives a memory increment (around line 587 of forest.py) in MiB of

n	master	fix2-forest-memory
100	13.3	1.6
200	25.5	2.0
300	38.1	2.2
400	50.1	2.3
500	62.6	2.4
600	75.7	2.2

I wasn't sure where you'd like the functions _run_estimator_ and _run_estimator2. Let me know if I should move them.

[glemaitre]

@mikebenfield Could you add the predict runtime of master and your PR for different sample size [1000, 10000, 100000, 1000000].

[glemaitre]

@mikebenfield Could you add the predict runtime of master and your PR for different sample size [1000, 10000, 100000, 1000000].

[mikebenfield]

@glemaitre Using the same script I posted before, with n_estimators=200, I get this (in seconds, on my little 2-core laptop):

n_samples	master	fix2-forest-memory
100	0.108	0.107
1000	0.107	0.107
10000	0.113	0.103
100000	0.990	0.932
1000000	16.175	14.363

[mikebenfield]

@glemaitre Using the same script I posted before, with n_estimators=200, I get this (in seconds, on my little 2-core laptop):

n_samples	master	fix2-forest-memory
100	0.108	0.107
1000	0.107	0.107
10000	0.113	0.103
100000	0.990	0.932
1000000	16.175	14.363

[glemaitre]

@ogrisel wdyt of this solution?

[glemaitre]

@ogrisel wdyt of this solution?

[jnothman]

This looks like a very good thing. Thanks!

[jnothman]

Given that this now works without parallel_helper due to the threading backend, I suspect we can remove all uses of parallel_helper in random forests. Another PR is welcome.

[jnothman]

this isn't the right condition, I think. Isn't it examining a 2d array whose length is the number of samples. Rather you can test for the presence of attribute shape to distinguish array from list.

[jnothman]

The fact that tests are passing makes me think I've missed something or tests are too weak...

[mikebenfield]

In this revised function, out is always a list. It contains only one element in the single output case. The function could have either of these two interfaces:

1. accept either a list of arrays or a single array, or
2. only accept a list of arrays.

I figured (2) was simpler, but I can change to (1) if desired.

[glemaitre]

I don't have a strong opinion on any of the solution.
However, using if isinstance() is straight forward regarding the aim of the if statement.
With your solution, I think that you would need a comment preceding the if statement for the sake of clarity.

[jnothman]

oh of course. you're testing out, not the prediction. LGTM!

…

On 3 Apr 2017 8:46 am, "mikebenfield" ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In sklearn/ensemble/forest.py <#8672 (comment)> : > - -def _run_estimator(f, X, out): - out += f(X, check_input=False) - - -def _run_estimator2(f, X, out): - all_proba = f(X, check_input=False) - for i in range(len(out)): - out[i] += all_proba[i] +# This is a utility function for joblib's Parallel. It can't go locally in +# ForestClassifier or ForestRegressor, because joblib complains that it cannot +# pickle it when placed there. + +def accumulate_prediction(predict, X, out): + prediction = predict(X, check_input=False) + if len(out) == 1: In this revised function, out is always a list. It contains only one element in the single output case. The function could 1. accept either a list of arrays or a single array, or 2. only accept a list of arrays. I figured (2) was simpler, but I can change to (1) if desired. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8672 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6wohebeB_ySN_ixEZlH-9UPmaPIYks5rsCUzgaJpZM4MuxbP> .

[jnothman]

oh of course. you're testing out, not the prediction. LGTM!

…

On 3 Apr 2017 8:46 am, "mikebenfield" ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In sklearn/ensemble/forest.py <#8672 (comment)> : > - -def _run_estimator(f, X, out): - out += f(X, check_input=False) - - -def _run_estimator2(f, X, out): - all_proba = f(X, check_input=False) - for i in range(len(out)): - out[i] += all_proba[i] +# This is a utility function for joblib's Parallel. It can't go locally in +# ForestClassifier or ForestRegressor, because joblib complains that it cannot +# pickle it when placed there. + +def accumulate_prediction(predict, X, out): + prediction = predict(X, check_input=False) + if len(out) == 1: In this revised function, out is always a list. It contains only one element in the single output case. The function could 1. accept either a list of arrays or a single array, or 2. only accept a list of arrays. I figured (2) was simpler, but I can change to (1) if desired. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8672 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6wohebeB_ySN_ixEZlH-9UPmaPIYks5rsCUzgaJpZM4MuxbP> .

[jnothman]

you may wish to add an entry in what's new

…

On 3 Apr 2017 8:54 am, "Joel Nothman" ***@***.***> wrote: oh of course. you're testing out, not the prediction. LGTM! On 3 Apr 2017 8:46 am, "mikebenfield" ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In sklearn/ensemble/forest.py > <#8672 (comment)> > : > > > - > -def _run_estimator(f, X, out): > - out += f(X, check_input=False) > - > - > -def _run_estimator2(f, X, out): > - all_proba = f(X, check_input=False) > - for i in range(len(out)): > - out[i] += all_proba[i] > +# This is a utility function for joblib's Parallel. It can't go locally in > +# ForestClassifier or ForestRegressor, because joblib complains that it cannot > +# pickle it when placed there. > + > +def accumulate_prediction(predict, X, out): > + prediction = predict(X, check_input=False) > + if len(out) == 1: > > In this revised function, out is always a list. It contains only one > element in the single output case. The function could > > 1. accept either a list of arrays or a single array, or > 2. only accept a list of arrays. > > I figured (2) was simpler, but I can change to (1) if desired. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#8672 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAEz6wohebeB_ySN_ixEZlH-9UPmaPIYks5rsCUzgaJpZM4MuxbP> > . >

[jnothman]

you may wish to add an entry in what's new

…

On 3 Apr 2017 8:54 am, "Joel Nothman" ***@***.***> wrote: oh of course. you're testing out, not the prediction. LGTM! On 3 Apr 2017 8:46 am, "mikebenfield" ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In sklearn/ensemble/forest.py > <#8672 (comment)> > : > > > - > -def _run_estimator(f, X, out): > - out += f(X, check_input=False) > - > - > -def _run_estimator2(f, X, out): > - all_proba = f(X, check_input=False) > - for i in range(len(out)): > - out[i] += all_proba[i] > +# This is a utility function for joblib's Parallel. It can't go locally in > +# ForestClassifier or ForestRegressor, because joblib complains that it cannot > +# pickle it when placed there. > + > +def accumulate_prediction(predict, X, out): > + prediction = predict(X, check_input=False) > + if len(out) == 1: > > In this revised function, out is always a list. It contains only one > element in the single output case. The function could > > 1. accept either a list of arrays or a single array, or > 2. only accept a list of arrays. > > I figured (2) was simpler, but I can change to (1) if desired. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#8672 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAEz6wohebeB_ySN_ixEZlH-9UPmaPIYks5rsCUzgaJpZM4MuxbP> > . >

[jmschrei]

I'm a bit confused about the [y_hat] aspect. It seems to indicate that regardless of if y_hat is multi-output or not, that accumulate_prediction should treat it as if it's single output? Can you clarify what's happening here?

[jmschrei]

Thanks for this! It looks great, except for one small confusion I had. Otherwise it LGTM.

[jmschrei]

Thanks for this! It looks great, except for one small confusion I had. Otherwise it LGTM.

[jnothman]

That's just to make the regression predicting look more like predict_proba

…

On 3 Apr 2017 4:22 pm, "Jacob Schreiber" ***@***.***> wrote: Thanks for this! It looks great, except for one small confusion I had. Otherwise it LGTM. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8672 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6641s4SjiSMtvVhqrOtB_T8WNtA6ks5rsJAmgaJpZM4MuxbP> .

[jnothman]

That's just to make the regression predicting look more like predict_proba

…

On 3 Apr 2017 4:22 pm, "Jacob Schreiber" ***@***.***> wrote: Thanks for this! It looks great, except for one small confusion I had. Otherwise it LGTM. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8672 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6641s4SjiSMtvVhqrOtB_T8WNtA6ks5rsJAmgaJpZM4MuxbP> .

[jmschrei]

Ah, got it, didn't see that it was in RFRegresor.

[jmschrei]

Ah, got it, didn't see that it was in RFRegresor.

[jmschrei]

Thanks for the contribution!

[jmschrei]

Thanks for the contribution!

[constantinpape]

Cool, this will make the random-forest useful for predicting larger data again.
Is there an upcoming release for sklearn, so this could be used in a portable manner (i,e, not having to build from master)?

[constantinpape]

Cool, this will make the random-forest useful for predicting larger data again.
Is there an upcoming release for sklearn, so this could be used in a portable manner (i,e, not having to build from master)?

[jnothman]

We're aiming for a release before the end of May, right, @amueller?

…

On 4 April 2017 at 09:08, Constantin Pape ***@***.***> wrote: Cool, this will make the random-forest useful for predicting larger data again. Is there an upcoming release for sklearn, so this could be used in a portable manner (i,e, not having to build from master)? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8672 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz627n1rFmjoPtgy3Ie0_l-j8jRCY4ks5rsXvqgaJpZM4MuxbP> .

[jnothman]

We're aiming for a release before the end of May, right, @amueller?

…

On 4 April 2017 at 09:08, Constantin Pape ***@***.***> wrote: Cool, this will make the random-forest useful for predicting larger data again. Is there an upcoming release for sklearn, so this could be used in a portable manner (i,e, not having to build from master)? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8672 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz627n1rFmjoPtgy3Ie0_l-j8jRCY4ks5rsXvqgaJpZM4MuxbP> .

[GaelVaroquaux]

That's very nice work, @mikebenfield ! Thanks!

[GaelVaroquaux]

That's very nice work, @mikebenfield ! Thanks!