pip-tools: Fix test failures for Python 3.8-3.10

[ento]

Description of changes

Fixes test failures for python38Packages.pip-tools, python39Packages.pip-tools, and python310Packages.pip-tools.

For Python 3.8/3.9: changed the current fix of patching up scripts/compile.py to ignore StopIteration to ignoring the test_bad_setup_file test case.

Diagnosis

The StopIteration exception was coming from this line in the build package, which looks for packages named pip in purelib:
https://github.com/pypa/build/blob/0.8.0/src/build/env.py#L272

By inserting a line that raises an exception with the result of running find {purelib}, I saw that purelib was empty when I tried to build python39Packages.pip-tools

E Exception: b'/build/build-env-7folokzz/lib/python3.9/site-packages\n'

...while it had a bunch of stuff with python310Packages.pip-tools

E Exception: b'/build/build-env-ofcylinh/lib/python3.10/site-packages\n/build/build-env-ofcylinh/lib/python3.10/site-packages/distutils-precedenc
e.pth\n/build/build-env-ofcylinh/lib/python3.10/site-packages/distutils_hack\n/build/build-env-ofcylinh/lib/python3.10/site-packages/distutils_hack/
init.py\n/build/build-env-ofcylinh/lib/python3.10/site-packages/_distutils_hack/override.py
...

pip is supposed to be installed by the venv module if you invoke venv.EnvBuilder with with_pip=True, which build does on this line:
https://github.com/pypa/build/blob/0.8.0/src/build/env.py#L265

So why isn't pip installed in the virtualenv?

Printing the output of the call to ensurepip from within venv showed that, in the context of pytest runs during nix-build, it thought requirements were already satisfied:

E Exception: b'Looking in links: /build/tmpkvtlev1e\nRequirement already satisfied: setuptools in /nix/store/lp6lgvkiqqcc8n62fdqz0vpjahqz4q4f-python3.9-setuptools-63.2.0/lib/python3.9/site-packages (63.2.0.post0)\nRequirement already satisfied: pip in /nix/store/1r5lg8wcm89cxbncyd6mf6dhjg1n2mdk-python3.9-pip-22.1.2/lib/python3.9/site-packages (22.1.2)\n'

This behavior differs in Python 3.10 (or more precisely, 3.10.6) because it includes a fix for ensurepip's bug where the 'isolated' flag of the Python process wasn't propagated.

Outside of nix-build, even for Python 3.8/3.9, running pip-compile in a directory that emulates the test case test_bad_setup_file (echo "BAD SYNTAX" > setup.py) doesn't result in a StopIteration ... I guess because, while PYTHONPATH gets set up to point to site-packages directories of propagatedBuildInputs, once the package is built, PYTHONPATH is left clean..? The output of the call to ensurepip when running ./result/bin/pip-compile shows that setuptools and pip do get installed, as expected:

Exception: b'Looking in links: /tmp/tmphe5b70rr\nProcessing /tmp/tmphe5b70rr/setuptools-58.1.0-py3-none-any.whl\nProcessing /tmp/tmphe5b70rr/pip-22.0.4
-py3-none-any.whl\nInstalling collected packages: setuptools, pip\nSuccessfully installed pip-22.0.4 setuptools-58.1.0\n'

In any case, given that this StopIteration exception only happens during testing and it's something that will be fixed upstream once the ensurepip bug fix lands in Python 3.8/3.9, I thought it better to avoid patching pip-compile's source code and ignore the test case instead.

For Python 3.10: Patched the test_bad_setup_file test case to provide a --find-links directory that contains a .whl file of the wheel package.

Diagnosis

Even with the 'empty virtualenv' issue above fixed, the test was still failing for Python 3.10.

Looking at the details of the exception revealed that the build package was trying to install wheel but wasn't able to do so because we don't allow network access during builds.

/nix/store/kpyjv1549lvl90ldh412kfr5zp870lfw-python3.10-build-0.8.0/lib/python3.10/site-packages/build/util.py:53: in project_wheel_metadata
    env.install(builder.build_system_requires)
/nix/store/kpyjv1549lvl90ldh412kfr5zp870lfw-python3.10-build-0.8.0/lib/python3.10/site-packages/build/env.py:216: in install
    _subprocess(cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = ['/build/build-env-kau0euyl/bin/python', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', ...]

    def _subprocess(cmd: List[str]) -> None:
        """Invoke subprocess and output stdout and stderr if it fails."""
        try:
            subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        except subprocess.CalledProcessError as e:
            print(e.output.decode(), end='', file=sys.stderr)
>           raise Exception(e.output.decode())
E           Exception: Requirement already satisfied: setuptools>=40.8.0 in /build/build-env-kau0euyl/lib/python3.10/site-packages (from -r /build/build-reqs-3eozu0uw.txt (line 1)) (63.2.0)
E           WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7ffff59bcb20>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/wheel/
E           WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7ffff59bc0a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/wheel/
E           WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7ffff59bd240>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/wheel/
E           WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7ffff59bd3f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/wheel/
E           WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7ffff59bd5a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/wheel/
E           ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
E           ERROR: No matching distribution found for wheel
E           WARNING: There was an error checking the latest version of pip.

I couldn't pin-point exactly when it started happening to be the case, but the wheel package is no longer installed by default in virtualenvs created by venv.

Confirmed that, with the fix in place, what seems to be the expected exception is raised

Patched build.ProjectBuilder._handle_backend to raise the caught exception and saw that it's SyntaxError, which sounds like what you'd expect if you try to evaluate a setup.py that only contains the string "BAD SYNTAX".

/nix/store/i5n1k47cs2ilfxd9idm34gv4wvvj8xkc-python3.10-build-0.8.0/lib/python3.10/site-packages/build/util.py:54: in project_wheel_metadata
    env.install(builder.get_requires_for_build('wheel'))
/nix/store/i5n1k47cs2ilfxd9idm34gv4wvvj8xkc-python3.10-build-0.8.0/lib/python3.10/site-packages/build/__init__.py:368: in get_requires_for_build
    with self._handle_backend(hook_name):
/nix/store/c1vb2z3c64i0sd92iz7fv0lb720qcvhb-python3-3.10.6/lib/python3.10/contextlib.py:153: in __exit__
    self.gen.throw(typ, value, traceback)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <build.ProjectBuilder object at 0x7ffff4b49030>
hook = 'get_requires_for_build_wheel'

    @contextlib.contextmanager
    def _handle_backend(self, hook: str) -> Iterator[None]:
        with _working_directory(self.srcdir):
            try:
                yield
            except pep517.wrappers.BackendUnavailable as exception:
                raise BuildBackendException(
                    exception,
                    f"Backend '{self._backend}' is not available.",
                    sys.exc_info(),
                )
            except subprocess.CalledProcessError as exception:
>               raise Exception(exception.output.decode())
E               Exception: Traceback (most recent call last):
E                 File "/nix/store/ag52yb06nkdxdxijwyg5sbjvhcgmycxs-python3.10-pep517-0.12.0/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
E                   main()
E                 File "/nix/store/ag52yb06nkdxdxijwyg5sbjvhcgmycxs-python3.10-pep517-0.12.0/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 345, in main
E                   json_out['return_val'] = hook(**hook_input['kwargs'])
E                 File "/nix/store/ag52yb06nkdxdxijwyg5sbjvhcgmycxs-python3.10-pep517-0.12.0/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
E                   return hook(config_settings)
E                 File "/nix/store/1pxd3xyarxfr6bxrkz2px70j59pa4z04-python3.10-setuptools-63.2.0/lib/python3.10/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
E                   return self._get_build_requires(
E                 File "/nix/store/1pxd3xyarxfr6bxrkz2px70j59pa4z04-python3.10-setuptools-63.2.0/lib/python3.10/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
E                   self.run_setup()
E                 File "/nix/store/1pxd3xyarxfr6bxrkz2px70j59pa4z04-python3.10-setuptools-63.2.0/lib/python3.10/site-packages/setuptools/build_meta.py", line 281, in run_setup
E                   super(_BuildMetaLegacyBackend,
E                 File "/nix/store/1pxd3xyarxfr6bxrkz2px70j59pa4z04-python3.10-setuptools-63.2.0/lib/python3.10/site-packages/setuptools/build_meta.py", line 174, in run_setup
E                   exec(code, locals())
E                 File "<string>", line 1
E                   BAD SYNTAX
E                       ^^^^^^
E               SyntaxError: invalid syntax

I didn't attempt to fix the build for Python 3.7, as the build error seemed to stem from a circular dependency that I wasn't sure how to fix:

Executing pipInstallPhase
/build/setuptools_scm-7.0.5/dist /build/setuptools_scm-7.0.5
Processing ./setuptools_scm-7.0.5-py3-none-any.whl
ERROR: Could not find a version that satisfies the requirement importlib-metadata; python_version < "3.8" (from setuptools-scm) (from versions: none)
ERROR: No matching distribution found for importlib-metadata; python_version < "3.8"

^Seems to require adding importlib-metadata to setuptools-scm's build inputs, but importlib-metadata already depends on setuptools-scm.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[Artturin]

python39Packages.pip-tools
and
python310Packages.pip-tools
currently build

94a2d5e
fa886e4