New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`MLPRegressor` quits fitting too soon due to `self._no_improvement_count` #9456

Closed
nnadeau opened this Issue Jul 27, 2017 · 1 comment

Comments

Projects
None yet
2 participants
@nnadeau
Contributor

nnadeau commented Jul 27, 2017

Description

  • MLPRegressor quits fitting too soon due to self._no_improvement_count.
    • self._no_improvement_count has a magic number limit of 2.
  • _update_no_improvement_count() uses self.best_loss_ to check if no improvement has occured.
    • However, if a local minima occurred or the loss curve fluctuates, the fitting can quit early.
  • Yes, batch_size tuning can improve loss curve fluctuations, but that is outside the scope of this issue.

Steps/Code to Reproduce

    estimator = MLPRegressor(
        hidden_layer_sizes=(100,) * 4,
        activation='relu',
        max_iter=int(1e4),
        verbose=True,
        random_state=RANDOM_STATE,
        tol=0
    )
    logging.info('Estimator: {}'.format(estimator.get_params()))

    y_pred = sklearn.model_selection.cross_val_predict(
        estimator=estimator,
        X=x,
        y=y_true,
        cv=CV,
    )
  • Example loss curve (blue) shown in plot below.
  • A local minima occured at iteration 182 (dashed red), updating self.best_loss_.
  • The fitting quits at iteration 184 due to self._no_improvement_count > 2.
  • Regardless of the quality of the model design, there was the possibility of further improvement.

mlp-regressor-fitting

Expected Results

  • MLPRegressor does not quit fitting unexpectedly early.
  • _update_no_improvement_count() uses the previous loss, not the best loss
    • The docs describe self.tol as: [...] two consecutive iterations [...]
  • The ability to set/tune the limit of self._no_improvement_count
  • The ability to ignore self._no_improvement_count

Actual Results

  • Please see above plot.

Versions

Windows-10-10.0.15063-SP0
Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 18:41:36) [MSC v.1900 64 bit (AMD64)]
NumPy 1.13.1
SciPy 0.19.1
Scikit-Learn 0.18.2
@amueller

This comment has been minimized.

Show comment
Hide comment
@amueller

amueller Jul 27, 2017

Member

I have observed this, too. Can we maybe have a reference on what a good default would be from the literature, or is it too batch-size dependent? What does Keras do?

Member

amueller commented Jul 27, 2017

I have observed this, too. Can we maybe have a reference on what a good default would be from the literature, or is it too batch-size dependent? What does Keras do?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment