Run tests using `unittest` and `test.support` with pytest

[eggplants]

a detailed description of the bug or problem you are having

pytest runs fine for most unittest tests, but does not seem to work well for those that use Python's internal modules, test and test.support.

I ran into such a test in berkeleydb's test. (See below.)

$ pytest
==================================================== test session starts =====================================================
platform darwin -- Python 3.9.10, pytest-7.0.1, pluggy-1.0.0
rootdir: /Users/eggplants/prog/berkeleydb-18.1.5
plugins: subtests-0.7.0, cov-3.0.0
collected 0 items / 26 errors

=========================================================== ERRORS ===========================================================
_____________________________________________ ERROR collecting tests/test_all.py _____________________________________________
ImportError while importing test module '/Users/eggplants/prog/berkeleydb-18.1.5/tests/test_all.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.pyenv/versions/3.9.10/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_all.py:51: in <module>
    from test.support.socket_helper import find_unused_port
E   ModuleNotFoundError: No module named 'test.support'; 'test' is not a package
...

I would like to be able to run pytest on such tests if possible. It enables to execute pytest-monkeytype for collecting actual types from tests.

output of pip list from the virtual environment you are using

$ pip list
Package                       Version
----------------------------- ---------
alabaster                     0.7.12
attrs                         21.4.0
Babel                         2.9.1
berkeleydb                    18.1.5
black                         21.9b0
certifi                       2021.10.8
charset-normalizer            2.0.12
click                         8.0.4
coverage                      6.3.2
doctest-ignore-unicode        0.1.2
docutils                      0.17.1
flake8                        4.0.1
flake8-black                  0.2.4
html5lib                      1.1
idna                          3.3
imagesize                     1.3.0
importlib-metadata            4.11.2
iniconfig                     1.1.1
isodate                       0.6.1
isort                         5.10.1
Jinja2                        3.0.3
markdown-it-py                2.0.1
MarkupSafe                    2.1.0
mccabe                        0.6.1
mdit-py-plugins               0.3.0
mdurl                         0.1.0
mypy                          0.940
mypy-extensions               0.4.3
myst-parser                   0.17.0
networkx                      2.7.1
nose                          1.3.7
packaging                     21.3
pathspec                      0.9.0
pbr                           5.8.1
pip                           22.0.4
platformdirs                  2.5.1
pluggy                        1.0.0
py                            1.11.0
pycodestyle                   2.8.0
pyflakes                      2.4.0
Pygments                      2.11.2
pyparsing                     3.0.7
pytest                        7.0.1
pytest-cov                    3.0.0
pytest-subtests               0.7.0
pytz                          2021.3
PyYAML                        6.0
regex                         2022.3.2
requests                      2.27.1
setuptools                    60.9.3
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        4.4.0
sphinxcontrib-apidoc          0.3.0
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-kroki           1.3.0
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
toml                          0.10.2
tomli                         1.2.3
types-setuptools              57.4.10
typing_extensions             4.1.1
urllib3                       1.26.8
webencodings                  0.5.1
zipp                          3.7.0

pytest and operating system versions

$ pytest --version
pytest 7.0.1
$ uname -vorm
21.3.0 Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64 x86_64 Darwin

minimal example if possible

https://pypi.org/project/berkeleydb/#files

from berkeleydb-18.1.5/tests/test_all.py:

"""
Copyright (c) 2008-2022, Jesus Cea Avion <jcea@jcea.es>
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

    1. Redistributions of source code must retain the above copyright
    notice, this list of conditions and the following disclaimer.

    2. Redistributions in binary form must reproduce the above
    copyright notice, this list of conditions and the following
    disclaimer in the documentation and/or other materials provided
    with the distribution.

    3. Neither the name of Jesus Cea Avion nor the names of its
    contributors may be used to endorse or promote products derived
    from this software without specific prior written permission.

    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
    CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
    INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
    MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
    DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS
    BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
    EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
            TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
            DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
    ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
    TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
    THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
    SUCH DAMAGE.
    """

"""Run all test cases.
"""

import sys
import os, os.path
import unittest
import threading
import platform
import pathlib

import berkeleydb
from berkeleydb import db, dbutils, dbshelve, \
        hashopen, btopen, rnopen, dbobj

if sys.version_info >= (3, 9):
    from test.support.socket_helper import find_unused_port
else:
    from test.support import find_unused_port

if sys.version_info >= (3, 10):
    from test.support.os_helper import rmtree, unlink
else:
    from test.support import rmtree, unlink
...

[The-Compiler]

I don't see how this would be pytest-specific, that import just doesn't seem to exist at all:

$ python3.9 -c "from test.support.socket_helper import find_unused_port"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'test.support'; 'test' is not a package

What happens if you try to run the same, with the same Python you're invoking pytest with?

[eggplants]

https://docs.python.org/3/library/test.html

Each module in the test package whose name starts with test_ is a testing suite for a specific module or feature. All new tests should be written using the unittest or doctest module.

https://docs.python.org/3/library/test.html#module-test.support

The test.support module provides support for Python’s regression test suite.

test and its modules are Python's internal packages and included when running unittest and doctest.

Also they does not appear to be possible to import from outside unittest and doctest.

[The-Compiler]

It doesn't seem to work for me via unittest either:

$ python3 -m unittest test_x.py 
E
======================================================================
ERROR: test_x (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_x
Traceback (most recent call last):
  File "/usr/lib/python3.10/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/home/florian/test_x.py", line 1, in <module>
    from test.support.socket_helper import find_unused_port
ModuleNotFoundError: No module named 'test'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

Does it for you? After installing python-tests (which seems to be how to get the test package on Archlinux, split from python), it then works fine for unittest and pytest (and also pure Python).

I still don't see how anything about this would be pytest-specific.

[eggplants]

In my environment:

$ uname -vorm
21.3.0 Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64 x86_64 Darwin
$ python -V
Python 3.9.10
$ wget https://files.pythonhosted.org/packages/c2/3f/a96d5caff3bbf0980fb47518ddb015a1df291f5ef31ee8c50ead21cb513a/berkeleydb-18.1.5.tar.gz
...
$ unar berkeleydb-18.1.5.tar.gz
...
$ cd berkeleydb-18.1.5/tests
$ pytest test_all.py
==================================================== test session starts =====================================================
platform darwin -- Python 3.9.10, pytest-7.0.1, pluggy-1.0.0
rootdir: /Users/eggplants/prog/berkeleydb-18.1.5
plugins: subtests-0.7.0, cov-3.0.0
collected 0 items / 1 error

=========================================================== ERRORS ===========================================================
_____________________________________________ ERROR collecting tests/test_all.py _____________________________________________
ImportError while importing test module '/Users/eggplants/prog/berkeleydb-18.1.5/tests/test_all.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../.pyenv/versions/3.9.10/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_all.py:51: in <module>
    from test.support.socket_helper import find_unused_port
E   ModuleNotFoundError: No module named 'test.support'; 'test' is not a package
================================================== short test summary info ===================================================
ERROR test_all.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================== 1 error in 0.13s ======================================================
$ python -m unittest test_all.py

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Berkeley DB 18.1.40: (May 29, 2020)
berkeleydb.db.version():   (18, 1, 40)
berkeleydb.db.full_version(): ('Berkeley DB Release 18.1, library version 18.1.40: (May 29, 2020)', 18, 1, 0, 0, 40)
berkeleydb.db.__version__: 18.1.5
py module:            /Users/eggplants/.pyenv/versions/3.9.10/lib/python3.9/site-packages/berkeleydb/__init__.py
extension module:     /Users/eggplants/.pyenv/versions/3.9.10/lib/python3.9/site-packages/berkeleydb/__init__.py
Test working dir:     /var/folders/w6/33w9tbtn5n92g_3l9n356h7r0000gn/T/z-Berkeley_DB
python version:       3.9.10 (main, Mar 13 2022, 21:54:23) [Clang 13.0.0 (clang-1300.0.29.30)] 64bit
My pid:               13843
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
.
----------------------------------------------------------------------
Ran 1 test in 0.008s

OK

[eggplants]

Somehow below my example runs without any errors in both pytest and unittest. (Or is my example useless?)

$ cat test_x.py
import unittest
from test.support.socket_helper import find_unused_port
class TestStringMethods(unittest.TestCase):
    def test_a(self): assert 1 == 1
$ pytest test_x.py
==================================================== test session starts =====================================================
platform darwin -- Python 3.9.10, pytest-7.0.1, pluggy-1.0.0
rootdir: /Users/eggplants/prog/test
plugins: subtests-0.7.0, cov-3.0.0
collected 1 item

test_x.py .                                                                                                            [100%]

===================================================== 1 passed in 0.03s ======================================================
$ python -m unittest test_x.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

[The-Compiler]

This seems to be due to the test.py in berkeleydb-18.1.5/ and pytest detecting that as rootdir and adding it to the import path. Thus, import test imports that file, which indeed is not a package...

I assume if you remove that test.py, the tests will work... Using --import-mode=importlib seems to help too.

[eggplants]

I assume if you remove that test.py, the tests will work... Using --import-mode=importlib seems to help too.

It works! Thank you for your support!

[eggplants]

$ pwd
/Users/eggplants/prog/berkeleydb-18.1.5/tests
$ rm ../test.py
$ pytest
==================================================== test session starts =====================================================
platform darwin -- Python 3.9.10, pytest-7.0.1, pluggy-1.0.0
rootdir: /Users/eggplants/prog/berkeleydb-18.1.5
plugins: subtests-0.7.0, cov-3.0.0
collected 1021 items

test_all.py ..
test_associate.py .FFFF.....................FEFEFEFE............FFFF.............                                      [  5%]

...

[The-Compiler]

Yay! You might want to make berkeleydb maintainers aware of the test docs, though:

Note: test.support is not a public module. It is documented here to help Python developers write tests. The API of this module is subject to change without backwards compatibility concerns between releases.

The find_unused_port function they are using is pretty much this:

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    sock.bind(('localhost', 0))
    return sock.getsockname()[1]

and I'm pretty sure the rmtree and unlink helpers are similarly simple using pathlib or shutil...

Doesn't look they have a bugtracker however, only a mailinglist.

[eggplants]

I am not going to report this issue because it is too much trouble.

... And, although it is not relevant, it is also not acceptable that many tests are failing.

187 failed, 832 passed, 2 skipped, 8 errors in 376.94s (0:06:16)