When $TMPDIR is within $PWD, pip wheel dies on infinite recursion #7872

hroncok · 2020-03-19T16:26:44Z

Environment

pip version: master, 20.0.2, 19.3.1... possibly more
Python version: 3.9.0a4, 3.8.2, 3.7.7... possibly all
OS: Linux (Fedora)

Description

During debugging the problem described in #7555 (comment) we ahve realized that simply setting $TMPDIR to $PWD/.tmp would make our problems go away. However, when doing that, pip wheel tries to copy the entire $PWD to $TMPDIR/pip-req-build-... and it dies out on recursion:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/usr/lib/python3.8/site-packages/pip/_internal/commands/wheel.py", line 161, in run
    resolver.resolve(requirement_set)
  File "/usr/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 178, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/usr/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 336, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 285, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 486, in prepare_linked_requirement
    local_path = unpack_url(
  File "/usr/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 284, in unpack_url
    return unpack_file_url(link, location, download_dir, hashes=hashes)
  File "/usr/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 230, in unpack_file_url
    _copy_source_tree(link_path, location)
  File "/usr/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 213, in _copy_source_tree
    shutil.copytree(source, target, **kwargs)
  File "/usr/lib64/python3.8/shutil.py", line 554, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/usr/lib64/python3.8/shutil.py", line 510, in _copytree
    raise Error(errors)
shutil.Error: [('/builddir/build/BUILD/mistune-0.8.3/.../.tmp/pip-req-build-0pdab7qx/.tmp/pip-req-build-0pdab7qx/__pycache__/mistune.cpython-38.pyc', '/builddir/build/BUILD/mistune-0.8.3/.../.tmp/pip-req-build-0pdab7qx/.tmp/pip-req-build-0pdab7qx/.tmp/pip-req-build-0pdab7qx/__pycache__/mistune.cpython-38.pyc', "[Errno 36] File name too long: '/builddir/build/BUILD/mistune-0.8.3/.../.tmp/pip-req-build-0pdab7qx/.tmp/pip-req-build-0pdab7qx/.tmp/pip-req-build-0pdab7qx/__pycache__/mistune.cpython-38.pyc'"), ...]

Expected behavior

The target directory itself should be ignored when copying to it.

How to Reproduce

$ export TMPDIR=${PWD}/.tmp
$ mkdir ${TMPDIR}
$ pip wheel .  # with an extension module

Output

+ /usr/bin/python3 -m pip wheel --wheel-dir ./pyproject-macros-wheeldir --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --no-clean --progress-bar off --verbose .
Created temporary directory: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-ephem-wheel-cache-40ziqhu0
Created temporary directory: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-tracker-kg6rp8lh
Initialized build tracking at /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-tracker-kg6rp8lh
Created build tracker: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-tracker-kg6rp8lh
Entered build tracker: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-tracker-kg6rp8lh
Created temporary directory: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-wheel-d6bw8m93
Processing /builddir/build/BUILD/mistune-0.8.3
  Created temporary directory: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-build-0pdab7qx
Removed build tracker: '/builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-tracker-kg6rp8lh'
ERROR: Exception:
(see above)

The text was updated successfully, but these errors were encountered:

hroncok · 2020-03-19T16:26:59Z

I have a fix ready and will submit a Pull Request shorty.

hroncok · 2020-03-19T17:02:53Z

#7873

During a build of extension module within `pip wheel` the source directory is recursively copied in a temporary directory. See pypa#7555 When the temporary directory is inside the source directory (for example by setting `TMPDIR=$PWD/tmp`) this caused an infinite recursion that ended in: [Errno 36] File name too long We prevent that buy never copying the target to the target in _copy_source_tree. Fixes pypa#7872

118: Update pip to 20.1 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.0.2** to **20.1**. <details> <summary>Changelog</summary> ### 20.1 ``` ================= Process ------- - Document that pip 21.0 will drop support for Python 2.7. Features -------- - Add ``pip cache dir`` to show the cache directory. (`7350 <https://github.com/pypa/pip/issues/7350>`_) Bug Fixes --------- - Abort pip cache commands early when cache is disabled. (`8124 <https://github.com/pypa/pip/issues/8124>`_) - Correctly set permissions on metadata files during wheel installation, to permit non-privileged users to read from system site-packages. (`8139 <https://github.com/pypa/pip/issues/8139>`_) ``` ### 20.1b1 ``` =================== Deprecations and Removals ------------------------- - Remove emails from AUTHORS.txt to prevent usage for spamming, and only populate names in AUTHORS.txt at time of release (`5979 <https://github.com/pypa/pip/issues/5979>`_) - Remove deprecated ``--skip-requirements-regex`` option. (`7297 <https://github.com/pypa/pip/issues/7297>`_) - Building of local directories is now done in place, instead of a temporary location containing a copy of the directory tree. (`7555 <https://github.com/pypa/pip/issues/7555>`_) - Remove unused ``tests/scripts/test_all_pip.py`` test script and the ``tests/scripts`` folder. (`7680 <https://github.com/pypa/pip/issues/7680>`_) Features -------- - pip now implements PEP 610, so ``pip freeze`` has better fidelity in presence of distributions installed from Direct URL requirements. (`609 <https://github.com/pypa/pip/issues/609>`_) - Add ``pip cache`` command for inspecting/managing pip's wheel cache. (`6391 <https://github.com/pypa/pip/issues/6391>`_) - Raise error if ``--user`` and ``--target`` are used together in ``pip install`` (`7249 <https://github.com/pypa/pip/issues/7249>`_) - Significantly improve performance when ``--find-links`` points to a very large HTML page. (`7729 <https://github.com/pypa/pip/issues/7729>`_) - Indicate when wheel building is skipped, due to lack of the ``wheel`` package. (`7768 <https://github.com/pypa/pip/issues/7768>`_) - Change default behaviour to always cache responses from trusted-host source. (`7847 <https://github.com/pypa/pip/issues/7847>`_) - An alpha version of a new resolver is available via ``--unstable-feature=resolver``. (`988 <https://github.com/pypa/pip/issues/988>`_) Bug Fixes --------- - Correctly freeze a VCS editable package when it is nested inside another VCS repository. (`3988 <https://github.com/pypa/pip/issues/3988>`_) - Correctly handle ``%2F`` in URL parameters to avoid accidentally unescape them into ``/``. (`6446 <https://github.com/pypa/pip/issues/6446>`_) - Reject VCS URLs with an empty revision. (`7402 <https://github.com/pypa/pip/issues/7402>`_) - Warn when an invalid URL is passed with ``--index-url`` (`7430 <https://github.com/pypa/pip/issues/7430>`_) - Use better mechanism for handling temporary files, when recording metadata about installed files (RECORD) and the installer (INSTALLER). (`7699 <https://github.com/pypa/pip/issues/7699>`_) - Correctly detect global site-packages availability of virtual environments created by PyPA’s virtualenv>=20.0. (`7718 <https://github.com/pypa/pip/issues/7718>`_) - Remove current directory from ``sys.path`` when invoked as ``python -m pip <command>`` (`7731 <https://github.com/pypa/pip/issues/7731>`_) - Stop failing uninstallation, when trying to remove non-existent files. (`7856 <https://github.com/pypa/pip/issues/7856>`_) - Prevent an infinite recursion with ``pip wheel`` when ``$TMPDIR`` is within the source directory. (`7872 <https://github.com/pypa/pip/issues/7872>`_) - Significantly speedup ``pip list --outdated`` by parallelizing index interaction. (`7962 <https://github.com/pypa/pip/issues/7962>`_) - Improve Windows compatibility when detecting writability in folder. (`8013 <https://github.com/pypa/pip/issues/8013>`_) Vendored Libraries ------------------ - Update semi-supported debundling script to reflect that appdirs is vendored. - Add ResolveLib as a vendored dependency. - Upgrade certifi to 2020.04.05.1 - Upgrade contextlib2 to 0.6.0.post1 - Upgrade distro to 1.5.0. - Upgrade idna to 2.9. - Upgrade msgpack to 1.0.0. - Upgrade packaging to 20.3. - Upgrade pep517 to 0.8.2. - Upgrade pyparsing to 2.4.7. - Remove pytoml as a vendored dependency. - Upgrade requests to 2.23.0. - Add toml as a vendored dependency. - Upgrade urllib3 to 1.25.8. Improved Documentation ---------------------- - Emphasize that VCS URLs using git, git+git and git+http are insecure due to lack of authentication and encryption (`1983 <https://github.com/pypa/pip/issues/1983>`_) - Clarify the usage of --no-binary command. (`3191 <https://github.com/pypa/pip/issues/3191>`_) - Clarify the usage of freeze command in the example of Using pip in your program (`7008 <https://github.com/pypa/pip/issues/7008>`_) - Add a "Copyright" page. (`7767 <https://github.com/pypa/pip/issues/7767>`_) - Added example of defining multiple values for options which support them (`7803 <https://github.com/pypa/pip/issues/7803>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <github-bot@pyup.io>

triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Mar 19, 2020

This was referenced Mar 19, 2020

Solving issues related to out-of-tree builds #7555

Closed

Prevent infinite recursion with pip wheel with $TMPDIR in $PWD #7873

Merged

pradyunsg closed this as completed in #7873 Apr 10, 2020

lock bot added the auto-locked Outdated issues that have been locked by automation label May 20, 2020

triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label May 20, 2020

lock bot locked as resolved and limited conversation to collaborators May 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

When $TMPDIR is within $PWD, pip wheel dies on infinite recursion #7872

When $TMPDIR is within $PWD, pip wheel dies on infinite recursion #7872

hroncok commented Mar 19, 2020 •

edited

hroncok commented Mar 19, 2020

hroncok commented Mar 19, 2020

When $TMPDIR is within $PWD, pip wheel dies on infinite recursion #7872

When $TMPDIR is within $PWD, pip wheel dies on infinite recursion #7872

Comments

hroncok commented Mar 19, 2020 • edited

hroncok commented Mar 19, 2020

hroncok commented Mar 19, 2020

hroncok commented Mar 19, 2020 •

edited