Some tests failing on Debian with pytest-xvfb 3.0.0

[juliangilbey]

Hello! I'm trying to upgrade the Debian version of this package to 3.0.0, but some of the tests are now failing. I don't know why this might be, as they run fine on GitHub Actions. Here are the failing tests (the xephyr tests fail in the same way if xserver-xephyr is installed; they are not included here). I wonder if you have any idea what might be wrong and how we might address it? It seems that the --no-xvfb marker might be being ignored? As a comparison, the tests work fine on 2.0.0; if you want to look at my attempt at packaging 3.0.0 for Debian, the git repo is at https://salsa.debian.org/python-team/packages/pytest-xvfb

$ python3.11 -m pytest tests/test_xvfb.py::test_empty_display tests/test_xvfb.py::test_no_xvfb_marker tests/test_xvfb.py::test_xvfb_with_xauth
============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /build/pytest-xvfb-eZZpZI/pkg-tests
plugins: xvfb-3.0.0
collected 16 items                                                             

tests/test_xvfb.py FFss.F.FssssFFss                                      [100%]

=================================== FAILURES ===================================
___________________________ test_empty_display[None] ___________________________

pytester = <Pytester PosixPath('/tmp/pytest-of-user/pytest-13/test_empty_display0')>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fdff0f27d90>
backend_args = []

    def test_empty_display(
        pytester: pytest.Pytester, monkeypatch: pytest.MonkeyPatch, backend_args: list[str]
    ) -> None:
        if backend_args == ["--xvfb-backend", "xephyr"]:
            pytest.skip("Xephyr needs a host display")
    
        monkeypatch.setenv("DISPLAY", "")
        pytester.makepyfile(
            """
            import os
    
            def test_display():
                assert 'DISPLAY' in os.environ
        """
        )
        result = pytester.runpytest(*backend_args)
>       assert os.environ["DISPLAY"] == ""
E       AssertionError: assert ':152' == ''
E         + :152

/build/pytest-xvfb-eZZpZI/pkg-tests/tests/test_xvfb.py:84: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/pytest-of-user/pytest-13/test_empty_display0
plugins: xvfb-3.0.0
collected 1 item

test_empty_display.py .                                                  [100%]

============================== 1 passed in 0.01s ===============================
___________________________ test_empty_display[xvfb] ___________________________

pytester = <Pytester PosixPath('/tmp/pytest-of-user/pytest-13/test_empty_display1')>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fdff095d690>
backend_args = ['--xvfb-backend', 'xvfb']

    def test_empty_display(
        pytester: pytest.Pytester, monkeypatch: pytest.MonkeyPatch, backend_args: list[str]
    ) -> None:
        if backend_args == ["--xvfb-backend", "xephyr"]:
            pytest.skip("Xephyr needs a host display")
    
        monkeypatch.setenv("DISPLAY", "")
        pytester.makepyfile(
            """
            import os
    
            def test_display():
                assert 'DISPLAY' in os.environ
        """
        )
        result = pytester.runpytest(*backend_args)
>       assert os.environ["DISPLAY"] == ""
E       AssertionError: assert ':153' == ''
E         + :153

/build/pytest-xvfb-eZZpZI/pkg-tests/tests/test_xvfb.py:84: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/pytest-of-user/pytest-13/test_empty_display1
plugins: xvfb-3.0.0
collected 1 item

test_empty_display.py .                                                  [100%]

============================== 1 passed in 0.00s ===============================
___________________ test_no_xvfb_marker[None-args1-2 passed] ___________________

pytester = <Pytester PosixPath('/tmp/pytest-of-user/pytest-13/test_no_xvfb_marker1')>
args = ['--no-xvfb'], outcome = '2 passed', backend_args = []

    @pytest.mark.parametrize(
        "args, outcome",
        [
            ([], "1 passed, 1 skipped"),
            (["--no-xvfb"], "2 passed"),
        ],
    )
    def test_no_xvfb_marker(
        pytester: pytest.Pytester, args: list[str], outcome: str, backend_args: list[str]
    ) -> None:
        pytester.makepyfile(
            """
            import pytest
    
            @pytest.mark.no_xvfb
            def test_marked():
                pass
    
            def test_unmarked():
                pass
        """
        )
        res = pytester.runpytest(*args, *backend_args)
>       res.stdout.fnmatch_lines(f"*= {outcome}*")
E       Failed: nomatch: '*= 2 passed*'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0'
E           and: 'rootdir: /tmp/pytest-of-user/pytest-13/test_no_xvfb_marker1'
E           and: 'plugins: xvfb-3.0.0'
E           and: 'collected 2 items'
E           and: ''
E           and: 'test_no_xvfb_marker.py s.                                                [100%]'
E           and: ''
E           and: '========================= 1 passed, 1 skipped in 0.00s ========================='
E       remains unmatched: '*= 2 passed*'

/build/pytest-xvfb-eZZpZI/pkg-tests/tests/test_xvfb.py:243: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/pytest-of-user/pytest-13/test_no_xvfb_marker1
plugins: xvfb-3.0.0
collected 2 items

test_no_xvfb_marker.py s.                                                [100%]

========================= 1 passed, 1 skipped in 0.00s =========================
___________________ test_no_xvfb_marker[xvfb-args1-2 passed] ___________________

pytester = <Pytester PosixPath('/tmp/pytest-of-user/pytest-13/test_no_xvfb_marker3')>
args = ['--no-xvfb'], outcome = '2 passed'
backend_args = ['--xvfb-backend', 'xvfb']

    @pytest.mark.parametrize(
        "args, outcome",
        [
            ([], "1 passed, 1 skipped"),
            (["--no-xvfb"], "2 passed"),
        ],
    )
    def test_no_xvfb_marker(
        pytester: pytest.Pytester, args: list[str], outcome: str, backend_args: list[str]
    ) -> None:
        pytester.makepyfile(
            """
            import pytest
    
            @pytest.mark.no_xvfb
            def test_marked():
                pass
    
            def test_unmarked():
                pass
        """
        )
        res = pytester.runpytest(*args, *backend_args)
>       res.stdout.fnmatch_lines(f"*= {outcome}*")
E       Failed: nomatch: '*= 2 passed*'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0'
E           and: 'rootdir: /tmp/pytest-of-user/pytest-13/test_no_xvfb_marker3'
E           and: 'plugins: xvfb-3.0.0'
E           and: 'collected 2 items'
E           and: ''
E           and: 'test_no_xvfb_marker.py s.                                                [100%]'
E           and: ''
E           and: '========================= 1 passed, 1 skipped in 0.00s ========================='
E       remains unmatched: '*= 2 passed*'

/build/pytest-xvfb-eZZpZI/pkg-tests/tests/test_xvfb.py:243: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/pytest-of-user/pytest-13/test_no_xvfb_marker3
plugins: xvfb-3.0.0
collected 2 items

test_no_xvfb_marker.py s.                                                [100%]

========================= 1 passed, 1 skipped in 0.00s =========================
__________________________ test_xvfb_with_xauth[None] __________________________

pytester = <Pytester PosixPath('/tmp/pytest-of-user/pytest-13/test_xvfb_with_xauth0')>
backend_args = []

    @pytest.mark.skipif(not xauth_available, reason="no xauth")
    def test_xvfb_with_xauth(pytester: pytest.Pytester, backend_args: list[str]) -> None:
        original_auth = os.environ.get("XAUTHORITY")
        pytester.makeini(
            """
            [pytest]
            xvfb_xauth = True
        """
        )
        pytester.makepyfile(
            """
            import os
    
            def test_xauth():
                print('\\nXAUTHORITY: ' + os.environ['XAUTHORITY'])
                assert os.path.isfile(os.environ['XAUTHORITY'])
                assert os.access(os.environ['XAUTHORITY'], os.R_OK)
        """
        )
        result = pytester.runpytest("-s", *backend_args)
        # Get and parse the XAUTHORITY: line
        authline = next(l for l in result.outlines if l.startswith("XAUTHORITY:"))
        authfile = authline.split(" ", 1)[1]
    
        assert result.ret == 0
        # Make sure the authfile is deleted
>       assert not os.path.exists(authfile)
E       AssertionError: assert not True
E        +  where True = <function exists at 0x7fdff1e434c0>('/tmp/PyVirtualDisplay.hsrb32v7.Xauthority')
E        +    where <function exists at 0x7fdff1e434c0> = <module 'posixpath' (frozen)>.exists
E        +      where <module 'posixpath' (frozen)> = os.path

/build/pytest-xvfb-eZZpZI/pkg-tests/tests/test_xvfb.py:347: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/pytest-of-user/pytest-13/test_xvfb_with_xauth0
configfile: tox.ini
plugins: xvfb-3.0.0
collected 1 item

test_xvfb_with_xauth.py 
XAUTHORITY: /tmp/PyVirtualDisplay.hsrb32v7.Xauthority
.

============================== 1 passed in 0.01s ===============================
__________________________ test_xvfb_with_xauth[xvfb] __________________________

pytester = <Pytester PosixPath('/tmp/pytest-of-user/pytest-13/test_xvfb_with_xauth1')>
backend_args = ['--xvfb-backend', 'xvfb']

    @pytest.mark.skipif(not xauth_available, reason="no xauth")
    def test_xvfb_with_xauth(pytester: pytest.Pytester, backend_args: list[str]) -> None:
        original_auth = os.environ.get("XAUTHORITY")
        pytester.makeini(
            """
            [pytest]
            xvfb_xauth = True
        """
        )
        pytester.makepyfile(
            """
            import os
    
            def test_xauth():
                print('\\nXAUTHORITY: ' + os.environ['XAUTHORITY'])
                assert os.path.isfile(os.environ['XAUTHORITY'])
                assert os.access(os.environ['XAUTHORITY'], os.R_OK)
        """
        )
        result = pytester.runpytest("-s", *backend_args)
        # Get and parse the XAUTHORITY: line
        authline = next(l for l in result.outlines if l.startswith("XAUTHORITY:"))
        authfile = authline.split(" ", 1)[1]
    
        assert result.ret == 0
        # Make sure the authfile is deleted
>       assert not os.path.exists(authfile)
E       AssertionError: assert not True
E        +  where True = <function exists at 0x7fdff1e434c0>('/tmp/PyVirtualDisplay.pp7i4tj7.Xauthority')
E        +    where <function exists at 0x7fdff1e434c0> = <module 'posixpath' (frozen)>.exists
E        +      where <module 'posixpath' (frozen)> = os.path

/build/pytest-xvfb-eZZpZI/pkg-tests/tests/test_xvfb.py:347: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/pytest-of-user/pytest-13/test_xvfb_with_xauth1
configfile: tox.ini
plugins: xvfb-3.0.0
collected 1 item

test_xvfb_with_xauth.py 
XAUTHORITY: /tmp/PyVirtualDisplay.pp7i4tj7.Xauthority
.

============================== 1 passed in 0.01s ===============================
=============================== warnings summary ===============================
tests/test_xvfb.py::test_empty_display[None]
  /usr/lib/python3.11/subprocess.py:1127: ResourceWarning: subprocess 5808 is still running
    _warn("subprocess %s is still running" % self.pid,
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_xvfb.py::test_empty_display[xvfb]
  /usr/lib/python3.11/subprocess.py:1127: ResourceWarning: subprocess 5812 is still running
    _warn("subprocess %s is still running" % self.pid,
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_xvfb.py::test_no_xvfb_marker[None-args0-1 passed, 1 skipped]
  /usr/lib/python3.11/subprocess.py:1127: ResourceWarning: subprocess 5816 is still running
    _warn("subprocess %s is still running" % self.pid,
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_xvfb.py::test_no_xvfb_marker[xvfb-args0-1 passed, 1 skipped]
  /usr/lib/python3.11/subprocess.py:1127: ResourceWarning: subprocess 5820 is still running
    _warn("subprocess %s is still running" % self.pid,
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_xvfb.py::test_xvfb_with_xauth[None]
  /usr/lib/python3.11/subprocess.py:1127: ResourceWarning: subprocess 5824 is still running
    _warn("subprocess %s is still running" % self.pid,
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_xvfb.py::test_xvfb_with_xauth[xvfb]
  /usr/lib/python3.11/subprocess.py:1127: ResourceWarning: subprocess 5829 is still running
    _warn("subprocess %s is still running" % self.pid,
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_xvfb.py::test_empty_display[None] - AssertionError: assert ':152' == ''
FAILED tests/test_xvfb.py::test_empty_display[xvfb] - AssertionError: assert ':153' == ''
FAILED tests/test_xvfb.py::test_no_xvfb_marker[None-args1-2 passed] - Failed: nomatch: '*= 2 passed*'
FAILED tests/test_xvfb.py::test_no_xvfb_marker[xvfb-args1-2 passed] - Failed: nomatch: '*= 2 passed*'
FAILED tests/test_xvfb.py::test_xvfb_with_xauth[None] - AssertionError: assert not True
FAILED tests/test_xvfb.py::test_xvfb_with_xauth[xvfb] - AssertionError: assert not True
============== 6 failed, 2 passed, 8 skipped, 6 warnings in 0.86s ==============
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=11>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=13>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=22>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=25>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=15>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=17>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=21>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=23>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=24>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=27>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=26>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=29>

[juliangilbey]

Oh, I've found the problem. The Debian version used the PyPI version of the package, which does not contain pytest.ini, so the new option included there (--runpytest=subprocess) is not passed on to pytest. I guess this could be fixed by adding pytest.ini to the MANIFEST.in file.