`MLPRegressor` quits fitting too soon due to `self._no_improvement_count`

[nnadeau]

Description

• MLPRegressor quits fitting too soon due to self._no_improvement_count.
  • self._no_improvement_count has a magic number limit of 2.
• _update_no_improvement_count() uses self.best_loss_ to check if no improvement has occured.
  • However, if a local minima occurred or the loss curve fluctuates, the fitting can quit early.
• Yes, batch_size tuning can improve loss curve fluctuations, but that is outside the scope of this issue.

Steps/Code to Reproduce

    estimator = MLPRegressor(
        hidden_layer_sizes=(100,) * 4,
        activation='relu',
        max_iter=int(1e4),
        verbose=True,
        random_state=RANDOM_STATE,
        tol=0
    )
    logging.info('Estimator: {}'.format(estimator.get_params()))

    y_pred = sklearn.model_selection.cross_val_predict(
        estimator=estimator,
        X=x,
        y=y_true,
        cv=CV,
    )

• Example loss curve (blue) shown in plot below.
• A local minima occured at iteration 182 (dashed red), updating self.best_loss_.
• The fitting quits at iteration 184 due to self._no_improvement_count > 2.
• Regardless of the quality of the model design, there was the possibility of further improvement.

Expected Results

• MLPRegressor does not quit fitting unexpectedly early.
• _update_no_improvement_count() uses the previous loss, not the best loss
  • The docs describe self.tol as: [...] two consecutive iterations [...]
• The ability to set/tune the limit of self._no_improvement_count
• The ability to ignore self._no_improvement_count

Actual Results

• Please see above plot.

Versions

Windows-10-10.0.15063-SP0
Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 18:41:36) [MSC v.1900 64 bit (AMD64)]
NumPy 1.13.1
SciPy 0.19.1
Scikit-Learn 0.18.2

[amueller]

I have observed this, too. Can we maybe have a reference on what a good default would be from the literature, or is it too batch-size dependent? What does Keras do?

[amueller]

I have observed this, too. Can we maybe have a reference on what a good default would be from the literature, or is it too batch-size dependent? What does Keras do?