PERF Use np.maximum instead of np.clip in relu function

[alexhenrie]

Neural net training is about 2% faster without checking in relu whether any value is greater than the maximum possible value for the data type (which is always false, so there's no need to check).

Test program:

from sklearn.datasets import load_digits
from sklearn.neural_network import MLPClassifier
from neurtu import delayed, Benchmark

digits = load_digits(return_X_y=True)
X = digits[0][:,:10]
y = digits[0][:,11]

clf = MLPClassifier(solver='lbfgs', alpha=1e-5,
                    hidden_layer_sizes=(5, 2), random_state=1, max_iter=1000)

train = delayed(clf).fit(X, y)
print(Benchmark(wall_time=True, cpu_time=True, repeat=10)(train))

Before:

      wall_time  cpu_time                                                                                                     
mean   2.116878  2.108431
max    2.126080  2.115298
std    0.004060  0.005196

After:

      wall_time  cpu_time                                                                                                     
mean   2.080088  2.069779
max    2.088962  2.074299
std    0.004015  0.002257

[thomasjpfan]

How does this scale with greater hidden_layer_sizes?

[alexhenrie]

np.maximum is faster in all cases, although you're right that the difference is not noticeable after increasing hidden_layer_sizes from (5, 2) to (10, 4). This change also makes the code simpler and easier to understand, so I think it should be accepted as "code maintenance" even if the performance improvement is deemed insignificant.

[jnothman]

I'm surprised this wasn't np.clip(X, 0, out=X)

[jnothman]

Thanks @alexhenrie

[alexhenrie]

Thank you!