[MRG] 32-bit support for MLP #17759

postmalloc · 2020-06-27T18:19:57Z

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Includes float32 support for MLPClassifier and MLPRegressor.

harishB97 · 2020-06-29T07:20:58Z

Hi @d3b0unce,
May I know whether the call for check_array in line 328 is necessary. When we call validate_input following that in line 333, it calls the check_array and does the type casting once again. Can we pass the dtype tuple directly to _validate_data in line 977.

postmalloc · 2020-06-29T12:07:08Z

@harishB97, thank you for pointing it out. That call was indeed redundant. Passing the tuple directly to _validate_data now.

rth

Thanks @d3b0unce ! I think there is 1 occurrences missing,

scikit-learn/sklearn/neural_network/_multilayer_perceptron.py

Line 688 in 430c208

activations.append(np.empty((X.shape[0],

Also could you please add a check to test that coef, intercept (and predictions only in the case of MLPRegressor) are of dtype float32 when X is float32?

rth · 2020-06-29T14:40:02Z

sklearn/neural_network/_multilayer_perceptron.py

-                                   multi_output=True)
+                                   multi_output=True,
+                                   dtype=(np.float64, np.float32))
+        self._dtype = X.dtype


So right now _validate_input is indeed only called on fit, but generally I think it would be safer to make it not store anything on self in case someone uses it for predict in the future. Using X.dtype later should be enough?

That makes sense. I had assigned it to self as I needed dtype in _init_coef where X is not in scope. Now I am passing dtype as a parameter to _init_coef.

postmalloc · 2020-06-29T15:53:43Z

Thank you @rth, for the suggestions! I added tests to check dtypes of parameters.

thomasjpfan

Thank you for the PR @d3b0unce !

thomasjpfan · 2020-06-30T03:59:10Z

sklearn/neural_network/_multilayer_perceptron.py

@@ -985,7 +993,7 @@ def _validate_input(self, X, y, incremental):
                                 " `self.classes_` has %s. 'y' has %s." %
                                 (self.classes_, classes))

-        y = self._label_binarizer.transform(y)
+        y = self._label_binarizer.transform(y).astype(np.bool)


What was the reason behind this change?

I added this because the call to _label_binarizer.transform returns int64, which was upcasting other network parameters to float64 during _backprop.

Let's include this explanation as a comment in the code. (This downcast to bool is to prevent upcasting when working with float32 data)

thomasjpfan · 2020-06-30T04:06:08Z

sklearn/neural_network/tests/test_mlp.py

+    assert_allclose(pred_64, pred_32, rtol=1e-04)
+
+
+def test_mlp_param_dtypes():


This can be parametrize:

@pytest.mark.parametrize('dtype', [np.float32, np.float64]) def test_mlp_param_dtypes(dtype): # Checks input dtype is used for attributes and prediction X, y = X_digits[:300].astype(dtype), y_digits[:300] mlp = MLPRegressor(alpha=1e-5, hidden_layer_sizes=(5, 3), random_state=1, max_iter=50) mlp.fit(X, y) pred = mlp_64.predict(X) assert pred.dtype == dtype ...

thomasjpfan · 2020-06-30T04:06:34Z

sklearn/neural_network/tests/test_mlp.py

+                           hidden_layer_sizes=(5, 3),
+                           random_state=1, max_iter=100)
+    mlp_64.fit(X_digits[:300], y_digits[:300])
+    pred_64 = mlp_64.predict(X_digits[300:])


We should check for predict_proba as well?

Yes, added a test

postmalloc · 2020-06-30T04:56:53Z

Thank you @thomasjpfan for the suggestions! I've committed the changes.

rth

One last comment otherwise LGTM, thanks:

rth · 2020-07-01T11:21:33Z

sklearn/neural_network/tests/test_mlp.py

+    # Checks if input dtype is used for network parameters
+    # and predictions
+    X, y = X_digits.astype(dtype), y_digits
+    mlp = MLPRegressor(alpha=1e-5,


Could you also parametrize this test function with Estimator in [MLPClassifier, MLPRegressor] . Fitting on y should work in both cases since it is int. And then the last assert only do it in the case of MLPRegressor.

rth · 2020-07-01T13:35:29Z

Also please add an entry to the change log at doc/whats_new/v0.24.rst. Like the other entries there, please reference this pull request with :pr: and credit yourself with :user:.

rth

Thanks @d3b0unce !

thomasjpfan

One small comment, otherwise LGTM!

Thank you @d3b0unce !

thomasjpfan · 2020-07-03T18:20:56Z

sklearn/neural_network/_multilayer_perceptron.py

@@ -985,7 +993,7 @@ def _validate_input(self, X, y, incremental):
                                 " `self.classes_` has %s. 'y' has %s." %
                                 (self.classes_, classes))

-        y = self._label_binarizer.transform(y)
+        y = self._label_binarizer.transform(y).astype(np.bool)


Let's include this explanation as a comment in the code. (This downcast to bool is to prevent upcasting when working with float32 data)

postmalloc · 2020-07-04T04:37:37Z

Thanks @thomasjpfan! Added the comment.

…7759)

float32 support for MLP

efe404e

github-actions bot added the module:neural_network label Jun 27, 2020

postmalloc changed the title ~~[MRG] 32-bit support for MLP~~ [WIP] 32-bit support for MLP Jun 27, 2020

Make dtype variable private

721deb1

postmalloc changed the title ~~[WIP] 32-bit support for MLP~~ [MRG] 32-bit support for MLP Jun 27, 2020

postmalloc added 3 commits June 29, 2020 14:57

Remove redundant call to check_array

1d03737

Use smaller network in tests

5a53f43

Remove default param; reduce iters

30a5a95

rth reviewed Jun 29, 2020

View reviewed changes

Remove dtype member variable; Add tests for MLP param dtypes

2afcb04

Cast activations to dtype in _pred

b8fc68b

thomasjpfan reviewed Jun 30, 2020

View reviewed changes

Test predict_proba; Parameterize tests

871dbfd

rth approved these changes Jul 1, 2020

View reviewed changes

postmalloc added 2 commits July 1, 2020 17:13

Parameterize estimators

6357210

Merge branch 'master' into float32_nn

9e187bb

postmalloc added 2 commits July 1, 2020 19:24

Update whatsnew; remove trailing spaces

eedac56

Tag classes in whatsnew

546d659

rth approved these changes Jul 1, 2020

View reviewed changes

thomasjpfan approved these changes Jul 3, 2020

View reviewed changes

Add comment to explain downcast

5ff84f8

rth merged commit 9fc0006 into scikit-learn:master Jul 4, 2020
7 checks passed

glemaitre pushed a commit to glemaitre/scikit-learn that referenced this pull request Jul 17, 2020

ENH 32-bit support for MLPClassifier and MLPRegressor (scikit-learn#1…

d330d0a

…7759)

jayzed82 pushed a commit to jayzed82/scikit-learn that referenced this pull request Oct 22, 2020

ENH 32-bit support for MLPClassifier and MLPRegressor (scikit-learn#1…

e3c6b38

…7759)

[MRG] 32-bit support for MLP #17759

[MRG] 32-bit support for MLP #17759

postmalloc commented Jun 27, 2020

harishB97 commented Jun 29, 2020

postmalloc commented Jun 29, 2020

rth left a comment

rth Jun 29, 2020 •

edited

postmalloc Jun 29, 2020

postmalloc commented Jun 29, 2020

thomasjpfan left a comment

thomasjpfan Jun 30, 2020

postmalloc Jun 30, 2020

thomasjpfan Jul 3, 2020

thomasjpfan Jun 30, 2020

postmalloc Jun 30, 2020

thomasjpfan Jun 30, 2020

postmalloc Jun 30, 2020

postmalloc commented Jun 30, 2020

rth left a comment

rth Jul 1, 2020

postmalloc Jul 1, 2020

rth commented Jul 1, 2020

rth left a comment •

edited

thomasjpfan left a comment

thomasjpfan Jul 3, 2020

postmalloc commented Jul 4, 2020

		assert_allclose(pred_64, pred_32, rtol=1e-04)


		def test_mlp_param_dtypes():

[MRG] 32-bit support for MLP #17759

[MRG] 32-bit support for MLP #17759

Conversation

postmalloc commented Jun 27, 2020

Reference Issues/PRs

What does this implement/fix? Explain your changes.

harishB97 commented Jun 29, 2020

postmalloc commented Jun 29, 2020

rth left a comment

Choose a reason for hiding this comment

rth Jun 29, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

postmalloc commented Jun 29, 2020

thomasjpfan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

postmalloc commented Jun 30, 2020

rth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rth commented Jul 1, 2020

rth left a comment • edited

Choose a reason for hiding this comment

thomasjpfan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

postmalloc commented Jul 4, 2020

rth Jun 29, 2020 •

edited

rth left a comment •

edited