Some tests only fail when running all test but not when run individually

[rdnfn]

Currently in the dev branch sometimes the tests fail

• Only the --sb3 flag is added to pytest and all tests are run

  traitlets.traitlets.TraitError: The 'center' trait of an AppLayout instance expected a Widget or None, not the FigureCanvasAgg at '0xffff63828be0'.
  [...]
  =========================================== short test summary info ============================================
  FAILED tests/test_widget.py::test_basic_widget_functionality - traitlets.traitlets.TraitError: The 'center' t...
  ================================= 1 failed, 53 passed, 4654 warnings in 58.70s =================================

• Otherwise - if I run only run pytest tests/test_widget.py it passes, or if I run pytest it passes.

Minimal example:

vscode ➜ /bauwerk (dev/general ✗) $ pytest --sb3 tests/test_exp_script.py tests/test_widget.py 
============================================= test session starts ==============================================
platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0
rootdir: /bauwerk, configfile: setup.cfg
plugins: anyio-3.5.0, hydra-core-1.2.0
collected 2 items                                                                                              

tests/test_exp_script.py .                                                                               [ 50%]
tests/test_widget.py F                                                                                   [100%]

=================================================== FAILURES ===================================================
_______________________________________ test_basic_widget_functionality ________________________________________

    def test_basic_widget_functionality():
>       widget = bauwerk.widget.core.Game(step_time=0.001)

tests/test_widget.py:24: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
bauwerk/widget/core.py:117: in __init__
    self.game_lower_part = widgets.AppLayout(
/opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget_templates.py:157: in __init__
    super().__init__(**kwargs)
/opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget_box.py:64: in __init__
    super().__init__(**kwargs)
/opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget.py:443: in __init__
    super().__init__(**kwargs)
/opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget_templates.py:80: in __init__
    super().__init__(**kwargs)
/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:1079: in __init__
    setattr(self, key, value)
/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:606: in __set__
    self.set(obj, value)
/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:580: in set
    new_value = self._validate(obj, value)
/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:612: in _validate
    value = self.validate(obj, value)
/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:1851: in validate
    self.error(obj, value)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <traitlets.traitlets.Instance object at 0xffff90325730>, obj = AppLayout()
value = <matplotlib.backends.backend_agg.FigureCanvasAgg object at 0xffff70f0a790>, error = None, info = None

    def error(self, obj, value, error=None, info=None):
        """Raise a TraitError
    
        Parameters
        ----------
        obj : HasTraits or None
            The instance which owns the trait. If not
            object is given, then an object agnostic
            error will be raised.
        value : any
            The value that caused the error.
        error : Exception (default: None)
            An error that was raised by a child trait.
            The arguments of this exception should be
            of the form ``(value, info, *traits)``.
            Where the ``value`` and ``info`` are the
            problem value, and string describing the
            expected value. The ``traits`` are a series
            of :class:`TraitType` instances that are
            "children" of this one (the first being
            the deepest).
        info : str (default: None)
            A description of the expected value. By
            default this is infered from this trait's
            ``info`` method.
        """
        if error is not None:
            # handle nested error
            error.args += (self,)
            if self.name is not None:
                # this is the root trait that must format the final message
                chain = " of ".join(describe("a", t) for t in error.args[2:])
                if obj is not None:
                    error.args = ("The '%s' trait of %s instance contains %s which "
                        "expected %s, not %s." % (self.name, describe("an", obj),
                        chain, error.args[1], describe("the", error.args[0])),)
                else:
                    error.args = ("The '%s' trait contains %s which "
                        "expected %s, not %s." % (self.name, chain,
                        error.args[1], describe("the", error.args[0])),)
            raise error
        else:
            # this trait caused an error
            if self.name is None:
                # this is not the root trait
                raise TraitError(value, info or self.info(), self)
            else:
                # this is the root trait
                if obj is not None:
                    e = "The '%s' trait of %s instance expected %s, not %s." % (
                        self.name, class_of(obj), self.info(), describe("the", value))
                else:
                    e = "The '%s' trait expected %s, not %s." % (
                        self.name, self.info(), describe("the", value))
>               raise TraitError(e)
E               traitlets.traitlets.TraitError: The 'center' trait of an AppLayout instance expected a Widget or None, not the FigureCanvasAgg at '0xffff70f0a790'.

/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:692: TraitError
=============================================== warnings summary ===============================================
../opt/conda/lib/python3.9/site-packages/scipy/__init__.py:146
  /opt/conda/lib/python3.9/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.4
    warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"

../opt/conda/lib/python3.9/site-packages/_pytest/config/__init__.py:1252
  /opt/conda/lib/python3.9/site-packages/_pytest/config/__init__.py:1252: PytestConfigWarning: Unknown config option: collect_ignore
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

tests/test_exp_script.py::test_default_run
  /opt/conda/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if not hasattr(tensorboard, "__version__") or LooseVersion(

tests/test_exp_script.py::test_default_run
  /opt/conda/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    ) < LooseVersion("1.15"):

tests/test_exp_script.py::test_default_run
  /opt/conda/lib/python3.9/site-packages/botocore/httpsession.py:34: DeprecationWarning: 'urllib3.contrib.pyopenssl' module is deprecated and will be removed in a future release of urllib3 2.x. Read more in this issue: https://github.com/urllib3/urllib3/issues/2680
    from urllib3.contrib.pyopenssl import orig_util_SSLContext as SSLContext

tests/test_exp_script.py: 2268 warnings
  /opt/conda/lib/python3.9/site-packages/cvxpy/interface/numpy_interface/ndarray_interface.py:49: DeprecationWarning: `np.complex` is a deprecated alias for the builtin `complex`. To silence this warning, use `complex` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.complex128` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    if result.dtype in [numpy.complex, numpy.float64]:

tests/test_exp_script.py::test_default_run
tests/test_exp_script.py::test_default_run
  /opt/conda/lib/python3.9/site-packages/matplotlib/cbook/__init__.py:1369: DeprecationWarning: setting an array element with a sequence. This was supported in some cases where the elements are arrays with a single element. For example `np.array([1, np.array([2])], dtype=int)`. In the future this will raise the same ValueError as `np.array([1, [2]], dtype=int)`.
    return np.asarray(x, float)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================================== short test summary info ============================================
FAILED tests/test_widget.py::test_basic_widget_functionality - traitlets.traitlets.TraitError: The 'center' t...
================================= 1 failed, 1 passed, 2275 warnings in 22.27s ==================================

---

And when leaving out experiment script:

vscode ➜ /bauwerk (dev/general ✗) $ pytest --sb3 tests/test_widget.py 
============================================= test session starts ==============================================
platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0
rootdir: /bauwerk, configfile: setup.cfg
plugins: anyio-3.5.0, hydra-core-1.2.0
collected 1 item                                                                                               

tests/test_widget.py .                                                                                   [100%]

=============================================== warnings summary ===============================================
../opt/conda/lib/python3.9/site-packages/scipy/__init__.py:146
  /opt/conda/lib/python3.9/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.4
    warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"

../opt/conda/lib/python3.9/site-packages/_pytest/config/__init__.py:1252
  /opt/conda/lib/python3.9/site-packages/_pytest/config/__init__.py:1252: PytestConfigWarning: Unknown config option: collect_ignore
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

tests/test_widget.py::test_basic_widget_functionality
  /opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget.py:443: DeprecationWarning: Passing unrecognized arguments to super(Toolbar).__init__().
  __init__() missing 1 required positional argument: 'canvas'
  This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.
    super().__init__(**kwargs)

tests/test_widget.py::test_basic_widget_functionality
  /opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget.py:621: DeprecationWarning: Deprecated in traitlets 4.1, use the instance .metadata dictionary directly, like x.metadata[key] or x.metadata.get(key, default)
    if trait.get_metadata('sync'):

tests/test_widget.py::test_basic_widget_functionality
  /opt/conda/lib/python3.9/site-packages/matplotlib/cbook/__init__.py:1369: DeprecationWarning: setting an array element with a sequence. This was supported in some cases where the elements are arrays with a single element. For example `np.array([1, np.array([2])], dtype=int)`. In the future this will raise the same ValueError as `np.array([1, [2]], dtype=int)`.
    return np.asarray(x, float)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================== 1 passed, 5 warnings in 1.06s =========================================

[rdnfn]

Update: this is likely due to the call of bauwerk.utils.sb3.enable_wandb_plot_logging() function that changes the backend.

[rdnfn]

Further testing has shown that even if the same function is used to change the backend, it does not seem to work properly, i.e. running plt.switch_backend("agg") in one module, and then running plt.switch_backend("nbAgg") will lead to errors.

I have now moved to completely separate the testing of the experiment script, so that these two backends do never need to be used in the same test run. They now also run as separate github action runs. To run only the experiment script tests use pytest -m exp. Closing as resolved because of this fix.