Placement of `trainer.predict` call in one test causes other test to fail in `ubuntu-20.04` environment

[daniellepintz]

🐛 Bug

In test_tqdm_progress_bar_disabled_when_not_rank_zero when the trainer.predict call is moved to the end of the test, it deterministically causes test_model_checkpoint_score_and_ckpt[True-False-True-val_log] to fail with this error in the CI:

>           assert math.isclose(score, expected_score, rel_tol=1e-4)
E           assert False
E            +  where False = <built-in function isclose>(tensor(-9.9727e-05), -9.970580140361562e-05, rel_tol=0.0001)
E            +    where <built-in function isclose> = math.isclose

See error here: https://github.com/PyTorchLightning/pytorch-lightning/runs/4908767698?check_suite_focus=true)

To Reproduce

The current version on master does not break anything (note the placement of predict call):
https://github.com/PyTorchLightning/pytorch-lightning/blob/d132a9c3b73bad3e893f809a53401722087bec5e/tests/callbacks/test_tqdm_progress_bar.py#L629-L653

However if we move the predict call to the end of the test it causes the checkpoint test to fail:

@mock.patch("pytorch_lightning.trainer.trainer.Trainer.is_global_zero", new_callable=PropertyMock, return_value=False)
def test_tqdm_progress_bar_disabled_when_not_rank_zero(is_global_zero):
    """Test that the progress bar is disabled when not in global rank zero."""
    progress_bar = TQDMProgressBar()
    model = BoringModel()
    trainer = Trainer(
        callbacks=[progress_bar],
        fast_dev_run=True,
    )

    progress_bar.enable()
    trainer.fit(model)
    assert progress_bar.is_disabled

    progress_bar.enable()
    trainer.validate(model)
    assert progress_bar.is_disabled

    progress_bar.enable()
    trainer.test(model)
    assert progress_bar.is_disabled

    progress_bar.enable()
    trainer.predict(model)
    assert progress_bar.is_disabled

Expected behavior

Both of the two test configurations above should not cause the checkpoint test to fail.

Environment

This test is specifically failing in ubuntu-20.04 environment.

Additional Context

This was discovered in #11377

cc @Borda @akihironitta

[daniellepintz]

Unable to repro in #11720 so closing...