Multiple test failures when trying to run tests locally

[notatallshaw]

Description

I have been trying to submit PRs to Pip recently but I have having problems running the test suite locally as I get ~14 failures on a clean branch of Pip running the following instructions with Python 3.11: https://pip.pypa.io/en/stable/development/getting-started/#running-tests

My current workaround has to make sure I pass the linting steps and let Pip's CI workflow run the tests and see what test failures that shows. However this is a little frustrating as my local computer can complete all tests about 20x faster.

Expected behavior

No test failures when following doc instrunctions on how to run tests

pip version

pip 23.3.dev0

Python version

3.11

OS

Linux

How to Reproduce

1. git clone https://github.com/pypa/pip (or your fork)
2. cd pip
3. python3.11 -m venv .venv
4. source .venv/bin/activate
5. python -m pip install -e .
6. python -m pip install nox
7. nox -s test-3.11 -- -n auto

Output

FAILED tests/functional/test_install.py::test_pep518_with_user_pip - AssertionError: Script returned code: 1
FAILED tests/functional/test_install.py::test_install_existing_memory_distribution - AssertionError: assert 'Requirement already satisfied: example in <memory>' in 'Collecting example\n  Downloading example-0.1.0.tar.gz (860 bytes)\n  Preparing metadata (setup.py): started\n  Prepa...essfully built ...
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_install_user_conflict_in_globalsite - AssertionError: Script returned code: 1
FAILED tests/functional/test_new_resolver_user.py::test_new_resolver_install_user_reinstall_global_site - AssertionError: Script returned code: 1
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_upgrade_user_conflict_in_globalsite - AssertionError: Script returned code: 1
FAILED tests/functional/test_new_resolver_user.py::test_new_resolver_install_user_conflict_in_global_site - AssertionError: Script returned code: 1
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_install_user_conflict_in_globalsite_and_usersite - AssertionError: Script returned code: 1
FAILED tests/functional/test_new_resolver_user.py::test_new_resolver_install_user_conflict_in_global_and_user_sites - AssertionError: Script returned code: 1
FAILED tests/functional/test_pep668.py::test_fails[install] - AssertionError: Script passed unexpectedly:
FAILED tests/functional/test_pep668.py::test_fails[install-user] - assert 'I am externally managed' in "ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.\n"
FAILED tests/functional/test_pep668.py::test_fails[install-dry-run] - AssertionError: Script passed unexpectedly:
FAILED tests/functional/test_pep668.py::test_fails[uninstall] - AssertionError: Script passed unexpectedly:
FAILED tests/functional/test_uninstall_user.py::Tests_UninstallUserSite::test_uninstall_from_usersite_with_dist_in_global_site - AssertionError: Script returned code: 1

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

What platform are you on, and how did you install Python?

[notatallshaw]

I'm on Ubuntu via WSL2 (on latest Windows 10) and I installed Python using the apt repo ppa:deadsnakes/ppa.

But given you're asking I tried a clean install:

1. Installed Ubuntu 20.04.6 LTS via the Microsoft Store
2. sudo apt update
3. sudo apt upgrade
4. sudo add-apt-repository ppa:deadsnakes/ppa
5. sudo apt install python3.11 python3.11-venv
6. mkdir pip_tests
7. cd pip_tests
8. git clone https://github.com/pypa/pip
9. cd pip
10. python3.11 -m venv .venv
11. source .venv/bin/activate
12. python -m ensurepip (requirement already satisfied)
13. python -m pip install nox .
14. nox -s test-3.11 -- -n auto

Output:

FAILED tests/functional/test_install.py::test_pep518_with_user_pip - AssertionError: Script returned code: 1
FAILED tests/functional/test_install.py::test_install_existing_memory_distribution - AssertionError: assert 'Requirement already satisfied: example in <memory>' in 'Collecting example\n  Downloading example-0.1.0.tar.gz (860 by...
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_install_user_conflict_in_globalsite - AssertionError: Script returned code: 1
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_upgrade_user_conflict_in_globalsite - AssertionError: Script returned code: 1
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_install_user_conflict_in_globalsite_and_usersite - AssertionError: Script returned code: 1
FAILED tests/functional/test_new_resolver_user.py::test_new_resolver_install_user_reinstall_global_site - AssertionError: Script returned code: 1
FAILED tests/functional/test_new_resolver_user.py::test_new_resolver_install_user_conflict_in_global_site - AssertionError: Script returned code: 1
FAILED tests/functional/test_new_resolver_user.py::test_new_resolver_install_user_conflict_in_global_and_user_sites - AssertionError: Script returned code: 1
FAILED tests/functional/test_pep668.py::test_fails[install] - AssertionError: Script passed unexpectedly:
FAILED tests/functional/test_pep668.py::test_fails[install-user] - assert 'I am externally managed' in "ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.\n"
FAILED tests/functional/test_pep668.py::test_fails[install-dry-run] - AssertionError: Script passed unexpectedly:
FAILED tests/functional/test_pep668.py::test_fails[uninstall] - AssertionError: Script passed unexpectedly:
FAILED tests/functional/test_uninstall_user.py::Tests_UninstallUserSite::test_uninstall_from_usersite_with_dist_in_global_site - AssertionError: Script returned code: 1

Is there a reccomend different way to install / set up? At some point in the past I was able to get the test suite to pass on Ubuntu via WSL2 (although I've never got it to pass in Windows directly).

[pradyunsg]

Okie, I think I know what's happening here -- it's the Debian patches which are breaking assumptions in the test suite about what stuff will look like.

Can you try an installation via pyenv and confirm whether you see failures with that?

[notatallshaw]

I seem to be getting the same errors, but this could well be user error, I do not under pyenv very well.

Here is what I did:

1. Install pyenv, configure .bashrc, and start fresh shell
2. pyenv install 3.11 (took several attempts to produce no errors and warnings)
3. pyenv global 3.11
4. Check python -V which now shows 3.11.6
5. cd pip_tests/pip
6. rm -rf .venv/
7. python -m venv .venv
8. source .venv/bin/activate
9. python -m pip install nox
10. python -m pip install -e .
11. nox -s test-3.11 -- -n auto

Output:

FAILED tests/functional/test_pep668.py::test_fails[uninstall] - AssertionError: Script passed unexpectedly:
FAILED tests/functional/test_install.py::test_pep518_with_user_pip - AssertionError: Script returned code: 1
FAILED tests/functional/test_install.py::test_install_existing_memory_distribution - AssertionError: assert 'Requirement already satisfied: example in <memory>' in 'Collecting example\n  Downloading example-0.1.0.tar....
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_install_user_conflict_in_globalsite - AssertionError: Script returned code: 1
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_upgrade_user_conflict_in_globalsite - AssertionError: Script returned code: 1
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_install_user_conflict_in_globalsite_and_usersite - AssertionError: Script returned code: 1
FAILED tests/functional/test_uninstall_user.py::Tests_UninstallUserSite::test_uninstall_from_usersite_with_dist_in_global_site - AssertionError: Script returned code: 1
FAILED tests/functional/test_pep668.py::test_fails[install] - AssertionError: Script passed unexpectedly:
FAILED tests/functional/test_pep668.py::test_fails[install-user] - assert 'I am externally managed' in "ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualen...
FAILED tests/functional/test_pep668.py::test_fails[install-dry-run] - AssertionError: Script passed unexpectedly:
FAILED tests/functional/test_new_resolver_user.py::test_new_resolver_install_user_reinstall_global_site - AssertionError: Script returned code: 1
FAILED tests/functional/test_new_resolver_user.py::test_new_resolver_install_user_conflict_in_global_site - AssertionError: Script returned code: 1
FAILED tests/functional/test_new_resolver_user.py::test_new_resolver_install_user_conflict_in_global_and_user_sites - AssertionError: Script returned code: 1

Am I supposed to use pyenv to create some kind of environment for testing or do I still use venv with pyenv version activated? The documentation appears very sparse and my success rate with pyenv is very low.

[notatallshaw]

For my latest run I started to suspect that nox was the issue here not pyenv, however the documentation for nox also appears extremely sparse...

Reading this it appears nox should have the default behavior of not reusing virtual envs, but untrusting of this documention I tried passing in the --no-reuse-existing-virtualenvs flag anyway, e.g.: nox -s test-3.11 --no-reuse-existing-virtualenvs -- -n auto

And low and behold, I only got 1 test failure:

FAILED tests/unit/test_req.py::TestRequirementSet::test_download_info_archive_cache_with_invalid_origin - AssertionError: assert 'Ignoring invalid cache entry origin file' in 'Created temporary directory: /tmp/pip-ephem-wheel-cache-6t26lcsh'

Progress!

[pradyunsg]

Reading this it appears nox should have the default behavior of not reusing virtual envs

We override it in our noxfile.

[notatallshaw]

Well I ran the tests another time with no additional changes and that final test failure no longer shows up 😕

Anyway, I'll make a docs PR about how distro versions of Python might cause functional test failures.