PERF Discard old layers immediately when making neural net predictions

[alexhenrie]

This function does not need to hold any layer in memory except the current one, and discarding the previous layers immediately after use makes neural net predictions about 3% faster.

Test program:

from sklearn.datasets import load_digits
from sklearn.neural_network import MLPClassifier
from neurtu import delayed, Benchmark

digits = load_digits(return_X_y=True)
X = digits[0][:,:50]
y = digits[0][:,50]

clf = MLPClassifier(solver='lbfgs', alpha=1e-5,
                    hidden_layer_sizes=(50, 20), random_state=1, max_iter=1000)
clf.fit(X, y)

train = delayed(clf).predict([[x for x in range(50)] for y in range(50)])
print(Benchmark(wall_time=True, cpu_time=True, repeat=100)(train))

Before:

      wall_time  cpu_time                                                                                                     
mean   0.000455  0.000454
max    0.000466  0.000467
std    0.000002  0.000002

After:

mean   0.000439  0.000439
max    0.000454  0.000458
std    0.000002  0.000003

[rth]

Yes, it makes sense. My concern is that _predict is essentially _forward_pass but without storing activation. Could you maybe rename it to _forward_pass_fast and put it just after the _forward_pass function?

Let's see what other reviewers think about this improvement in general.

[alexhenrie]

That sounds reasonable to me. I have updated the pull request to rename _predict to _forward_pass_fast.

[NicolasHug]

Thanks @alexhenrie , some minor suggestsions but LGTM

[NicolasHug]

Nit but for consistency with the other version:

Suggested change

	if i != self.n_layers_ - 2:
	if (i + 1) != (self.n_layers_ - 1):

[alexhenrie]

The original version uses i + 1 several times, but this simplified version only uses i, so it's more clear here to not have the extra addition in the if statement.

[NicolasHug]

Please add a short description indicating how this differs from _forward_pass e.g.

This is the same as _forward_pass but does not record the activations of all layers and only returns the last layer's activation.

[alexhenrie]

I have added the text you suggested. Thanks!

[rth]

@alexhenrie There are some linting issues..

[alexhenrie]

The linting issues are now fixed.

[rth]

Thanks @alexhenrie !

[alexhenrie]

Thank you!