Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EarlyStoppingCallback considers first epoch as bad #854

Closed
alyaxey opened this issue Jun 24, 2020 · 1 comment · Fixed by #855
Closed

EarlyStoppingCallback considers first epoch as bad #854

alyaxey opened this issue Jun 24, 2020 · 1 comment · Fixed by #855
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@alyaxey
Copy link
Contributor

alyaxey commented Jun 24, 2020

🐛 Bug Report

EarlyStoppingCallback considers first epoch as bad. This can lead for example to always stopping after first epoch if patience=1.

How To Reproduce

You can train a model with early stopping and patience=1 and see that it always stops after first epoch. Or you can use the unit test below that I added to pull request.

Code sample

from unittest.mock import MagicMock, PropertyMock

from catalyst.core import EarlyStoppingCallback


def test_patience1():
    """@TODO: Docs. Contribution is welcome."""
    early_stop = EarlyStoppingCallback(1)
    runner = MagicMock()
    type(runner).stage_name = PropertyMock(return_value="training")
    type(runner).valid_metrics = PropertyMock(return_value={"loss": 0.001})
    stop_mock = PropertyMock(return_value=False)
    type(runner).need_early_stop = stop_mock

    early_stop.on_epoch_end(runner)

    assert stop_mock.mock_calls == []

Expected behavior

Training doesn't stop after first epoch. And the unit test passes.

Environment

Catalyst version: 20.06
PyTorch version: 1.5.1
Is debug build: No
CUDA used to build PyTorch: None
TensorFlow version: N/A
TensorBoard version: 2.2.2

OS: Mac OSX 10.15.5
GCC version: Could not collect
CMake version: version 3.8.0

Python version: 3.7
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] catalyst-codestyle==20.4
[pip3] catalyst-sphinx-theme==1.1.1
[pip3] efficientnet-pytorch==0.6.3
[pip3] numpy==1.18.5
[pip3] segmentation-models-pytorch==0.1.0
[pip3] tensorboard==2.2.2
[pip3] tensorboard-plugin-wit==1.6.0.post3
[pip3] tensorboardX==2.0
[pip3] torch==1.5.1
[pip3] torchvision==0.6.1
[conda] catalyst-codestyle        20.4                      <pip>
[conda] catalyst-sphinx-theme     1.1.1                     <pip>
[conda] efficientnet-pytorch      0.6.3                     <pip>
[conda] numpy                     1.18.5                    <pip>
[conda] segmentation-models-pytorch 0.1.0                     <pip>
[conda] tensorboard               2.2.2                     <pip>
[conda] tensorboard-plugin-wit    1.6.0.post3               <pip>
[conda] tensorboardX              2.0                       <pip>
[conda] torch                     1.5.1                     <pip>
[conda] torchvision               0.6.1                     <pip>
@alyaxey alyaxey added bug Something isn't working help wanted Extra attention is needed labels Jun 24, 2020
@github-actions
Copy link

Hi! Thank you for your contribution! Great first issue!

Scitator pushed a commit that referenced this issue Jun 25, 2020
* fixed EarlyStoppingCallback, first epoch was always considered bad

* added test for early stop

* updated CHANGELOG
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant