1.6.0: pytest is failing in pooch/tests/test_core.py::test_check_availability_on_ftp unit

[kloczek]

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pooch-1.6.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pooch-1.6.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pooch-1.6.0, configfile: pyproject.toml
plugins: anyio-3.6.2, localftpserver-1.1.3
collected 126 items

pooch/tests/test_core.py .........................F......                                                                                                            [ 25%]
pooch/tests/test_downloaders.py sss........ss....ssss.                                                                                                               [ 42%]
pooch/tests/test_hashes.py ......sssss...sssss.......                                                                                                                [ 63%]
pooch/tests/test_integration.py .                                                                                                                                    [ 64%]
pooch/tests/test_processors.py ................................                                                                                                      [ 89%]
pooch/tests/test_utils.py ............                                                                                                                               [ 99%]
pooch/tests/test_version.py .                                                                                                                                        [100%]

================================================================================= FAILURES =================================================================================
______________________________________________________________________ test_check_availability_on_ftp ______________________________________________________________________

    @pytest.mark.network
    def test_check_availability_on_ftp():
        "Should correctly check availability of existing and non existing files"
        # Check available remote file on FTP server
        pup = Pooch(
            path=DATA_DIR,
            base_url="ftp://data-out.unavco.org/pub/products/velocity/rel_201712/",
            registry={
                "pbo.final_igs08.20171202.vel": "md5:0b75d4049dedd0e179615f4b5e956156",
                "doesnot_exist.zip": "jdjdjdjdflld",
            },
        )
>       assert pup.is_available("pbo.final_igs08.20171202.vel")

pooch/tests/test_core.py:488:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pooch/core.py:699: in is_available
    ftp.connect(host=parsed_url["netloc"])
/usr/lib64/python3.8/ftplib.py:154: in connect
    self.sock = socket.create_connection((self.host, self.port), self.timeout,
/usr/lib64/python3.8/socket.py:808: in create_connection
    raise err
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

address = ('data-out.unavco.org', 21), timeout = <object object at 0x7ff5548303b0>, source_address = None

    def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
                          source_address=None):
        """Connect to *address* and return the socket object.

        Convenience function.  Connect to *address* (a 2-tuple ``(host,
        port)``) and return the socket object.  Passing the optional
        *timeout* parameter will set the timeout on the socket instance
        before attempting to connect.  If no *timeout* is supplied, the
        global default timeout setting returned by :func:`getdefaulttimeout`
        is used.  If *source_address* is set it must be a tuple of (host, port)
        for the socket to bind as a source address before making the connection.
        A host of '' or port 0 tells the OS to use the default.
        """

        host, port = address
        err = None
        for res in getaddrinfo(host, port, 0, SOCK_STREAM):
            af, socktype, proto, canonname, sa = res
            sock = None
            try:
                sock = socket(af, socktype, proto)
                if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
                    sock.settimeout(timeout)
                if source_address:
                    sock.bind(source_address)
>               sock.connect(sa)
E               ConnectionRefusedError: [Errno 111] Connection refused

/usr/lib64/python3.8/socket.py:796: ConnectionRefusedError
========================================================================= short test summary info ==========================================================================
SKIPPED [2] pooch/tests/test_downloaders.py:51: requires tqdm
SKIPPED [1] pooch/tests/test_downloaders.py:65: requires paramiko
SKIPPED [1] pooch/tests/test_downloaders.py:137: requires paramiko to run SFTP
SKIPPED [1] pooch/tests/test_downloaders.py:149: requires paramiko to run SFTP
SKIPPED [2] pooch/tests/test_downloaders.py:179: requires tqdm
SKIPPED [1] pooch/tests/test_downloaders.py:208: requires tqdm
SKIPPED [1] pooch/tests/test_downloaders.py:233: requires paramiko
SKIPPED [5] pooch/tests/test_hashes.py:126: requires xxhash
SKIPPED [5] pooch/tests/test_hashes.py:144: requires xxhash
FAILED pooch/tests/test_core.py::test_check_availability_on_ftp - ConnectionRefusedError: [Errno 111] Connection refused
================================================================ 1 failed, 106 passed, 19 skipped in 43.10s ================================================================
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-e59b8484-0a67-4761-b803-e36562acd70d
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-tkloczko/garbage-e59b8484-0a67-4761-b803-e36562acd70d'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba/test_rmtree_errorhandler_rerai0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_rerai0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba/test_rmtree_errorhandler_reado0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_reado0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba/test_safe_delete_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_delete_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba/test_safe_set_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_set_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba/test_safe_get_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_get_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-tkloczko/garbage-de25aea0-37a6-45b7-b898-161b93a777ba'
  warnings.warn(

Here is list of installed modules in build env

Package                       Version
----------------------------- -----------------
alabaster                     0.7.12
anyio                         3.6.2
appdirs                       1.4.4
async-generator               1.10
attrs                         22.1.0
Babel                         2.11.0
beautifulsoup4                4.11.1
Brlapi                        0.8.3
build                         0.9.0
certifi                       2022.9.24
cffi                          1.15.1
charset-normalizer            3.0.1
click                         8.1.3
cryptography                  38.0.4
cssselect                     1.1.0
distro                        1.8.0
dnspython                     2.2.1
docutils                      0.19
editables                     0.3
exceptiongroup                1.0.0
extras                        1.0.0
fixtures                      4.0.0
gpg                           1.17.1-unknown
h11                           0.13.0
hatchling                     1.11.1
hpack                         4.0.0
hyperframe                    6.0.1
idna                          3.4
imagesize                     1.4.1
importlib-metadata            5.1.0
iniconfig                     1.1.1
Jinja2                        3.1.2
libcomps                      0.1.19
louis                         3.24.0
lxml                          4.9.1
MarkupSafe                    2.1.1
mccabe                        0.7.0
mypy-extensions               0.4.3
numpy                         1.23.1
outcome                       1.1.0
packaging                     21.3
pathspec                      0.10.2
pbr                           5.9.0
pep517                        0.13.0
pip                           22.3.1
platformdirs                  2.5.2
pluggy                        1.0.0
ply                           3.11
pycodestyle                   2.9.1
pycparser                     2.21
pydata-sphinx-theme           0.12.0
pyflakes                      2.5.0
pyftpdlib                     1.5.6
Pygments                      2.13.0
PyGObject                     3.42.2
pyOpenSSL                     22.0.0
pyparsing                     3.0.9
pytest                        7.2.0
pytest-localftpserver         1.1.3
python-dateutil               2.8.2
pytz                          2022.4
PyYAML                        6.0
requests                      2.28.1
rpm                           4.17.0
scour                         0.38.2
setuptools                    65.6.3
setuptools-scm                7.0.5
six                           1.16.0
sniffio                       1.2.0
snowballstemmer               2.2.0
sortedcontainers              2.4.0
soupsieve                     2.3.2.post1
Sphinx                        5.3.0
sphinx-book-theme             0.3.3
sphinx-panels                 0.6.0
sphinxcontrib-applehelp       1.0.2.dev20221204
sphinxcontrib-devhelp         1.0.2.dev20221204
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1.dev20221204
sphinxcontrib-qthelp          1.0.3.dev20221204
sphinxcontrib-serializinghtml 1.1.5
testtools                     2.5.0
tomli                         2.0.1
trio                          0.21.0+dev
typing_extensions             4.4.0
urllib3                       1.26.12
wheel                         0.38.4
zipp                          3.11.0

[santisoler]

Hi @kloczek. Thanks for opening this issue and sorry for the delayed reply.

The test function you are trying to run will fail because the UNAVCO ftp server was decommissioned. We already merged #322 to solve it.

I imagine that you are packaging the latest release of Pooch and this is blocking you from doing it. If you are in a rush, I would recommend ignoring this error: it's not due to a bug in our code as pointed out in that PR. If you want to have full tests pass, I think we are about to release a Pooch version soon, considering that we fixed a few bugs recently.

[kloczek]

Units which requires public network acces should be marked by @pytest.mark.network mark to allow skipp such units on systems without access to the public network by running pytest with -m "not network" like in many other modules.
Oher thing is that probably in tis case better would be use https://github.com/oz123/pytest-localftpserver/

[santisoler]

Thanks for the tip. Pooch currently uses the network pytest mark on some test functions, but I suspect some test functions that require connectivity aren't decorated yet. Would you like to open an issue describing the problem and providing possible solutions to it?

BTW, Pooch v.1.7.0 have been just released, which includes the fixes introduced in #322. I think now you shouldn't have any problem running the test locally (while connected to internet).

[leouieda]

I've opened #367 to double check that all network-accessing tests are properly marked. That seems to be the real issue here so closing this one.