Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.0.0: pytest is failing in two units #132

Closed
kloczek opened this issue Jun 19, 2023 · 6 comments · Fixed by #134
Closed

1.0.0: pytest is failing in two units #132

kloczek opened this issue Jun 19, 2023 · 6 comments · Fixed by #134

Comments

@kloczek
Copy link

kloczek commented Jun 19, 2023

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

  • python3 -sBm build -w --no-isolation
  • because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
  • install .whl file in </install/prefix> using 'installer` module
  • run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
  • build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-requests-futures-1.0.0-16.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-requests-futures-1.0.0-16.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
============================= test session starts ==============================
platform linux -- Python 3.8.17, pytest-7.3.2, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/requests-futures-1.0.0
collected 12 items

test_requests_futures.py ....F...sF.s                                    [100%]

=================================== FAILURES ===================================
________________________ RequestsTestCase.test_redirect ________________________

self = <test_requests_futures.RequestsTestCase testMethod=test_redirect>

    def test_redirect(self):
        """ Tests for the ability to cleanly handle redirects. """
        sess = FuturesSession()
        future = sess.get(httpbin('redirect-to?url=get'))
        self.assertIsInstance(future, Future)
        resp = future.result()
        self.assertIsInstance(resp, Response)
>       self.assertEqual(200, resp.status_code)
E       AssertionError: 200 != 504

test_requests_futures.py:115: AssertionError
__________ RequestsProcessPoolTestCase.test_futures_existing_session ___________

self = <test_requests_futures.RequestsProcessPoolTestCase testMethod=test_futures_existing_session>

    def test_futures_existing_session(self):
        self.session.headers['Foo'] = 'bar'
>       self._assert_futures_session(session=self.session)

test_requests_futures.py:190:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test_requests_futures.py:203: in _assert_futures_session
    self.assertEqual(200, resp.status_code)
E   AssertionError: 200 != 504
=========================== short test summary info ============================
SKIPPED [1] test_requests_futures.py:183: fully supported on python >= 3.5
SKIPPED [1] test_requests_futures.py:282: Exception raised when unsupported
FAILED test_requests_futures.py::RequestsTestCase::test_redirect - AssertionE...
FAILED test_requests_futures.py::RequestsProcessPoolTestCase::test_futures_existing_session
============== 2 failed, 8 passed, 2 skipped in 151.45s (0:02:31) ==============

Python 3.8.17. Here is list of installed modules in build env

Package                       Version
----------------------------- -------
alabaster                     0.7.13
asttokens                     2.2.1
Babel                         2.12.1
backcall                      0.2.0
build                         0.10.0
charset-normalizer            3.1.0
decorator                     5.1.1
distro                        1.8.0
docutils                      0.19
exceptiongroup                1.1.1
executing                     1.2.0
gpg                           1.20.0
idna                          3.4
imagesize                     1.4.1
importlib-metadata            6.6.0
iniconfig                     2.0.0
installer                     0.7.0
ipython                       8.12.0
jedi                          0.18.2
Jinja2                        3.1.2
libcomps                      0.1.19
MarkupSafe                    2.1.2
matplotlib-inline             0.1.6
packaging                     23.1
parso                         0.8.3
pexpect                       4.8.0
pickleshare                   0.7.5
pluggy                        1.0.0
prompt-toolkit                3.0.38
ptyprocess                    0.7.0
pure-eval                     0.2.2
Pygments                      2.15.1
pyproject_hooks               1.0.0
pytest                        7.3.1
python-dateutil               2.8.2
pytz                          2023.2
requests                      2.31.0
setuptools                    67.7.2
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        6.2.1
sphinxcontrib-applehelp       1.0.4
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
stack-data                    0.6.2
tomli                         2.0.1
traitlets                     5.9.0
typing_extensions             4.5.0
urllib3                       1.26.15
wcwidth                       0.2.6
wheel                         0.40.0
zipp                          3.15.0
@kloczek
Copy link
Author

kloczek commented Jun 19, 2023

Interesting. After --deselect those two units another unit popped out.
Looks like it is some dependency between units

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-requests-futures-1.0.0-16.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-requests-futures-1.0.0-16.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network' --deselect test_requests_futures.py::RequestsProcessPoolTestCase::test_futures_existing_session --deselect test_requests_futures.py::RequestsTestCase::test_redirect
============================= test session starts ==============================
platform linux -- Python 3.8.17, pytest-7.3.2, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/requests-futures-1.0.0
collected 12 items / 2 deselected / 10 selected

test_requests_futures.py .......sFs                                      [100%]

=================================== FAILURES ===================================
_______________ RequestsProcessPoolTestCase.test_futures_session _______________
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib64/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/tkloczko/rpmbuild/BUILD/requests-futures-1.0.0/requests_futures/sessions.py", line 34, in wrap
    return background_callback(self, resp) or resp
  File "/home/tkloczko/rpmbuild/BUILD/requests-futures-1.0.0/test_requests_futures.py", line 159, in global_cb_return_result
    assert r, Response
AssertionError: <class 'requests.models.Response'>
assert <Response [504]>
"""

The above exception was the direct cause of the following exception:

self = <test_requests_futures.RequestsProcessPoolTestCase testMethod=test_futures_session>

    @skipIf(session_required, 'not supported in python < 3.5')
    def test_futures_session(self):
>       self._assert_futures_session()

test_requests_futures.py:181:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test_requests_futures.py:223: in _assert_futures_session
    resp = future.result()
/usr/lib64/python3.8/concurrent/futures/_base.py:444: in result
    return self.__get_result()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = None

    def __get_result(self):
        if self._exception:
            try:
>               raise self._exception
E               AssertionError: <class 'requests.models.Response'>
E               assert <Response [504]>

/usr/lib64/python3.8/concurrent/futures/_base.py:389: AssertionError
=========================== short test summary info ============================
SKIPPED [1] test_requests_futures.py:183: fully supported on python >= 3.5
SKIPPED [1] test_requests_futures.py:282: Exception raised when unsupported
FAILED test_requests_futures.py::RequestsProcessPoolTestCase::test_futures_session
======= 1 failed, 7 passed, 2 skipped, 2 deselected in 94.77s (0:01:34) ========

@ross
Copy link
Owner

ross commented Jun 19, 2023

The tests currently do hit the network as part of their running and aren't fully mocked so an env w/o network access will definitely cause problems, likely all the tests will fail.

HTTPBIN = environ.get('HTTPBIN_URL', 'https://nghttp2.org/httpbin/')

IIRC httpbin may be an installable thing, but not sure how reasonable it'd be to spin it up as part of unit tests. The other option would be to fully mock all the calls, but back when I originally wrote this I decided that wouldn't be a thorough enough test to be meaningful since 100% of the things being checked would be mocked out and everything would completely depend on what I expected the mocks to return and not what an actual HTTP server does. Mainly the worry would be that I could make changes to the request that would cause the server to behave differently, but my mocks wouldn't so I wouldn't find problems.

Not sure what the best path forward here is. These tests have worked well enough for 10+ years. Not particularly motivated to mock them just to get an RPM build happy.

Alternatively we could potentially do a test for network connectivity and skip pretty much all the tests if we can't talk to the network.

@kloczek
Copy link
Author

kloczek commented Jun 19, 2023

Is it possible to mark units like tar using @pytest.mark.network? 🤔
https://docs.pytest.org/en/7.1.x/example/markers.html
Many other modules uses already that mark in pytest test suite in case of units which needs access to anything more than localhost.
I've started test using pytest-find-dependencies however it still did not finish ..

@ross
Copy link
Owner

ross commented Jun 19, 2023

#134 should get the mark in place. If you want to give it a try and 👍 on that PR I'll ship it.

@kloczek
Copy link
Author

kloczek commented Jun 19, 2023

One sec 😋

@kloczek
Copy link
Author

kloczek commented Jun 19, 2023

Just tested that and looks like it works 👍 😄

Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.E3TH4T
+ umask 022
+ cd /home/tkloczko/rpmbuild/BUILD
+ cd /home/tkloczko/rpmbuild/BUILD
+ rm -rf requests-futures-1.0.0
+ /usr/bin/gzip -dc /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-1.0.0.tar.gz
+ /usr/bin/tar -xof -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd requests-futures-1.0.0
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Fix-rst-title-underline.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-No-mutable-kwarg-in-ElapsedFutureSession-example.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-DOcumenting-how-to-iterate-over-a-list-of-requests-r.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Document-how-to-access-the-request-url-and-add-addit.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-docs-fix-simple-typo-interpretor-interpreter.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Configure-the-generation-of-a-universal-wheel.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Fix-the-failing-test-by-switching-to-https-nghttp2.o.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Create-FUNDING.yml.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-combine-requirements.txt-files.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-unified-requirements.txt-in-MANIFEST.in.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Fix-setup.py-long_description_content_type.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Add-release-script-and-twine-build-deps.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Document-support-for-py3.6-3.7-and-3.8.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Apply-suggestions-from-code-review.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Updates-README.rst-with-svg-badge.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-scripts-to-rule-them-all-full-setup.py-requirements-.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Missed-cibuild-setup-py.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Switch-to-Actions-for-CI.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Add-stale-issue-pr-action.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-black-formatting.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-ignore-re-formatting-rev-with-.git-blame-ignore-revs.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-run-black-formatting-during-ci-and-pre-commit.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Update-requires-to-get-rid-of-py.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Bump-certifi-from-2022.9.24-to-2022.12.7.patch
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Update-requirements-.txt.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Bump-markdown-it-py-from-2.1.0-to-2.2.0.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Bump-requests-from-2.28.2-to-2.31.0.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-Update-requirements-.txt-1.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-py3.7-is-gone-add-3.11.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-requests-futures-pytest-mark-network-pyproject.toml-isort.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ RPM_EC=0

[..]

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-requests-futures-1.0.0-16.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-requests-futures-1.0.0-16.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.17, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/requests-futures-1.0.0
configfile: pyproject.toml
plugins: find-dependencies-0.5.2
collected 12 items / 10 deselected / 2 selected

tests/test_requests_futures.py ..                                                                                                                                                     [100%]

============================================================================= 2 passed, 10 deselected in 0.09s ==============================================================================
+ RPM_EC=0
++ jobs -p
+ exit 0
Processing files: python-requests-futures-1.0.0-16.fc35.noarch
Executing(%doc): /bin/sh -e /var/tmp/rpm-tmp.JrwKRR
+ umask 022
+ cd /home/tkloczko/rpmbuild/BUILD
+ cd requests-futures-1.0.0
+ DOCDIR=/home/tkloczko/rpmbuild/BUILDROOT/python-requests-futures-1.0.0-16.fc35.x86_64/usr/share/doc/python-requests-futures
+ export LC_ALL=C
+ LC_ALL=C
+ export DOCDIR
+ /usr/bin/mkdir -p /home/tkloczko/rpmbuild/BUILDROOT/python-requests-futures-1.0.0-16.fc35.x86_64/usr/share/doc/python-requests-futures
+ cp -pr README.rst /home/tkloczko/rpmbuild/BUILDROOT/python-requests-futures-1.0.0-16.fc35.x86_64/usr/share/doc/python-requests-futures
+ RPM_EC=0
++ jobs -p
+ exit 0
Provides: python-requests-futures = 1.0.0-16.fc35 python3.8dist(requests-futures) = 1 python3dist(requests-futures) = 1
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires: python(abi) = 3.8 python3.8dist(requests) >= 1.2
Obsoletes: python3-requests-futures
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/tkloczko/rpmbuild/BUILDROOT/python-requests-futures-1.0.0-16.fc35.x86_64
Wrote: /home/tkloczko/rpmbuild/SRPMS/python-requests-futures-1.0.0-16.fc35.src.rpm
Wrote: /home/tkloczko/rpmbuild/RPMS/python-requests-futures-1.0.0-16.fc35.noarch.rpm

@ross ross closed this as completed in #134 Jun 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants