pip wheel -e git+https:// does a git export instead of git clone #9273

sbidoul · 2020-12-14T14:02:26Z

Environment

pip version: 20.3
Python version: 3.8 (probably irrelevant)
OS: linux (probably irrelevant)

Description

pip wheel --no-deps -e git+https://github.com/pypa/pip-test-package#egg=pip-test-package used to keep a git clone in the src directory. Now it keeps something there, that does not have a .git directory in it.

Analysis

In progress, but a quick git bissect shows that the issue originates in the same area as #9122.

The text was updated successfully, but these errors were encountered:

sbidoul · 2020-12-14T14:06:04Z

This matters because if the package being built relies on setuptools_scm's git file finder, it does not work anymore.

uranusjr · 2020-12-14T14:32:41Z

Does this only affect wheel, or also install?

sbidoul · 2020-12-14T14:53:57Z

install works fine. The problem comes from here:

pip/src/pip/_internal/operations/prepare.py

Line 573 in 94b89c9

req.update_editable(self.download_dir is None)

which was changed in #8843 (the merge of wheel_download_dir and download_dir). The changed argument is obtain which drives whether pip does a vcs clone or vcs export in update_editable.

Now it does a vcs export in wheel mode too, whereas before it did that in download mode only.

sbidoul · 2020-12-14T15:12:21Z

So this bug has indeed the same origin as #9122, which is that _download_should_save has been replaced by a test on the presence of download_dir, which is not equivalent.

sbidoul · 2020-12-14T15:16:50Z

I suspect this clone vs export issue can also have implications on dependency resolution, for wheel and download alike. Indeed, even in download mode, if the vcs information is not present, setuptools_scm may yield incorrect version information, influencing the resolver outcome.

sbidoul · 2020-12-14T17:09:55Z

So, a radical approach could be to always vcs clone and never vcs export, which would fix this bug, but change slightly how pip download of editable requirements works, as follow.

	20.2.4 src has .git dir	20.2.4 zip has .git dir	20.3.1 src has .git dir	20.3.1 zip has .git dir	proposal src has .git dir	proposal zip has .git dir
pip download git+https://	n/a	yes	n/a	yes	n/a	yes
pip download -e git+https://	no	no	no	no	yes (this is an improvement)	yes (is this change a problem?)
pip wheel git+https://	n/a	n/a	n/a	n/a	n/a	n/a
pip wheel -e git+https://	yes	n/a	no (this bug)	n/a	yes (this bug is fixed)	n/a

What do people think?

sbidoul · 2020-12-23T11:30:34Z

I'm assigning this to the 20.3.4 milestone it this is a regression. #9274 implements the table above. See #9338 for the root cause analysis and discussion about possible approaches.

pradyunsg · 2020-12-23T20:35:52Z

I don't think we should have the .git folder in the zip under any circumstances.

Gah, why is pip's VCS setup so complicated. :(

RonnyPfannschmidt · 2020-12-23T21:36:16Z

This would be slightly less of a issue if git archives / exports had any usable Metadata whatsoever, unfortunately they don't, anything short of a full clone is typically crippled and unusable

pradyunsg · 2020-12-23T22:10:48Z

*made an important edit to my earlier comment, because I retyped the same thing in 3 different ways and picked the worst way to combine them :)

215: Update pip to 21.0.1 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.3.3** to **21.0.1**. <details> <summary>Changelog</summary> ### 21.0.1 ``` =================== Bug Fixes --------- - commands: debug: Use packaging.version.parse to compare between versions. (`9461 <https://github.com/pypa/pip/issues/9461>`_) - New resolver: Download and prepare a distribution only at the last possible moment to avoid unnecessary network access when the same version is already installed locally. (`9516 <https://github.com/pypa/pip/issues/9516>`_) Vendored Libraries ------------------ - Upgrade packaging to 20.9 ``` ### 21.0 ``` ================= Deprecations and Removals ------------------------- - Drop support for Python 2. (`6148 <https://github.com/pypa/pip/issues/6148>`_) - Remove support for legacy wheel cache entries that were created with pip versions older than 20.0. (`7502 <https://github.com/pypa/pip/issues/7502>`_) - Remove support for VCS pseudo URLs editable requirements. It was emitting deprecation warning since version 20.0. (`7554 <https://github.com/pypa/pip/issues/7554>`_) - Modernise the codebase after Python 2. (`8802 <https://github.com/pypa/pip/issues/8802>`_) - Drop support for Python 3.5. (`9189 <https://github.com/pypa/pip/issues/9189>`_) - Remove the VCS export feature that was used only with editable VCS requirements and had correctness issues. (`9338 <https://github.com/pypa/pip/issues/9338>`_) Features -------- - Add ``--ignore-requires-python`` support to pip download. (`1884 <https://github.com/pypa/pip/issues/1884>`_) - New resolver: Error message shown when a wheel contains inconsistent metadata is made more helpful by including both values from the file name and internal metadata. (`9186 <https://github.com/pypa/pip/issues/9186>`_) Bug Fixes --------- - Fix a regression that made ``pip wheel`` do a VCS export instead of a VCS clone for editable requirements. This broke VCS requirements that need the VCS information to build correctly. (`9273 <https://github.com/pypa/pip/issues/9273>`_) - Fix ``pip download`` of editable VCS requirements that need VCS information to build correctly. (`9337 <https://github.com/pypa/pip/issues/9337>`_) Vendored Libraries ------------------ - Upgrade msgpack to 1.0.2. - Upgrade requests to 2.25.1. Improved Documentation ---------------------- - Render the unreleased pip version change notes on the news page in docs. (`9172 <https://github.com/pypa/pip/issues/9172>`_) - Fix broken email link in docs feedback banners. (`9343 <https://github.com/pypa/pip/issues/9343>`_) .. note You should *NOT* be adding new change log entries to this file, this file is managed by towncrier. You *may* edit previous change logs to fix problems like typo corrections or such. To add a new change log entry, please see https://pip.pypa.io/en/latest/development/contributing/#news-entries .. towncrier release notes start ``` ### 20.3.4 ``` =================== Features -------- - ``pip wheel`` now verifies the built wheel contains valid metadata, and can be installed by a subsequent ``pip install``. This can be disabled with ``--no-verify``. (`9206 <https://github.com/pypa/pip/issues/9206>`_) - Improve presentation of XMLRPC errors in pip search. (`9315 <https://github.com/pypa/pip/issues/9315>`_) Bug Fixes --------- - Fixed hanging VCS subprocess calls when the VCS outputs a large amount of data on stderr. Restored logging of VCS errors that was inadvertently removed in pip 20.2. (`8876 <https://github.com/pypa/pip/issues/8876>`_) - Fix error when an existing incompatibility is unable to be applied to a backtracked state. (`9180 <https://github.com/pypa/pip/issues/9180>`_) - New resolver: Discard a faulty distribution, instead of quitting outright. This implementation is taken from 20.2.2, with a fix that always makes the resolver iterate through candidates from indexes lazily, to avoid downloading candidates we do not need. (`9203 <https://github.com/pypa/pip/issues/9203>`_) - New resolver: Discard a source distribution if it fails to generate metadata, instead of quitting outright. This implementation is taken from 20.2.2, with a fix that always makes the resolver iterate through candidates from indexes lazily, to avoid downloading candidates we do not need. (`9246 <https://github.com/pypa/pip/issues/9246>`_) Vendored Libraries ------------------ - Upgrade resolvelib to 0.5.4. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <github-bot@pyup.io>

sbidoul added C: vcs pip's interaction with version control systems like git, svn and bzr C: wheel The wheel format and 'pip wheel' command type: bug A confirmed bug or unintended behavior labels Dec 14, 2020

sbidoul mentioned this issue Dec 14, 2020

Remove VCS export feature #9274

Merged

sbidoul changed the title ~~pip wheel -e git+https:// does not preserve a git clone in src~~ pip wheel -e git+https:// does a git export instead of git clone Dec 14, 2020

This was referenced Dec 23, 2020

pip download of editable VCS requirements fails if requirement needs VCS information to prepare metadata #9337

Closed

pip download behaves differently for editable and non-editable VCS requirements #9338

Closed

sbidoul added this to the 20.3.4 milestone Dec 23, 2020

sbidoul added the C: editable Editable installations label Dec 23, 2020

pradyunsg closed this as completed in #9274 Jan 17, 2021

sbidoul mentioned this issue Jan 24, 2021

Release 20.3.4 #9494

Closed

github-actions bot locked as resolved and limited conversation to collaborators Oct 4, 2021

pip wheel -e git+https:// does a git export instead of git clone #9273

pip wheel -e git+https:// does a git export instead of git clone #9273

sbidoul commented Dec 14, 2020

sbidoul commented Dec 14, 2020

uranusjr commented Dec 14, 2020

sbidoul commented Dec 14, 2020 •

edited

sbidoul commented Dec 14, 2020

sbidoul commented Dec 14, 2020

sbidoul commented Dec 14, 2020 •

edited

sbidoul commented Dec 23, 2020 •

edited

pradyunsg commented Dec 23, 2020 •

edited

RonnyPfannschmidt commented Dec 23, 2020

pradyunsg commented Dec 23, 2020

pip wheel -e git+https:// does a git export instead of git clone #9273

pip wheel -e git+https:// does a git export instead of git clone #9273

Comments

sbidoul commented Dec 14, 2020

sbidoul commented Dec 14, 2020

uranusjr commented Dec 14, 2020

sbidoul commented Dec 14, 2020 • edited

sbidoul commented Dec 14, 2020

sbidoul commented Dec 14, 2020

sbidoul commented Dec 14, 2020 • edited

sbidoul commented Dec 23, 2020 • edited

pradyunsg commented Dec 23, 2020 • edited

RonnyPfannschmidt commented Dec 23, 2020

pradyunsg commented Dec 23, 2020

sbidoul commented Dec 14, 2020 •

edited

sbidoul commented Dec 14, 2020 •

edited

sbidoul commented Dec 23, 2020 •

edited

pradyunsg commented Dec 23, 2020 •

edited