-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pip install [git repo] hangs on clone step on Windows with large repositories #8876
Comments
The only significant change to VCS recently is #8817 but that doesn’t seem that related since you’re just fetching the default branch. Do the stuck repos have submodules or something? |
No, there's no sub-module, or anything exotic in terms of the git structure. It's a fairly vanilla setup, with a .gitignore. The file structure is like this:
|
Same problem for python 3.6 & pip 20.2.4. |
Can folks confirm if this occurs with pip 20.3? |
Confirmed that the same issue occurs with pip 20.3 |
Thanks for the confirmation! Someone needs to provide us with a reproducer here, or contribute the fix, because pip's contributors currently have no way to test this. |
I'll see if I can further isolate the issue. I've done some testing with a few different variables. So far the only variable that has seemed to make a difference is running the command from Ubuntu instead of Windows. It's possible this is an OS dependent thing, but whether that's a purely OS issue, an OS-pip issue, or an OS-git issue, I don't know yet. |
I believe I've isolated the issue down to the way that Git for Windows behaves when working with large repos. For some reason, Git on Windows outputs to I've tested a fix based on I've got a link to the comparison below. What's the process to fully test this and get it merged as a patch on https://github.com/pypa/pip/compare/pypa:20.2.4...Mikuana:v20.2-vcs-bugfix?expand=1 |
@pradyunsg would it be possible to get a branch created from the |
But you won’t be able to get the patch into a new pip version if you don’t create the patch against 20.3 (or master). What are the problems preventing a branch from 20.3 being possible? |
None that I know of. I was planning to do that next. I was just thinking it would be best to patch the minor version where the issue first appeared, then work the change into the rest of the history from there. That way, if someone is doing a large VCS install, it will work in |
Feel free to make a PR against the current I don't see why this issue needs to be treated differently anyway, so I reckon a PR to |
@pradyunsg sounds good. I think there's an case to be made for patching |
PR to |
@pradyunsg Should we reopen this to track the backport to 20.3.4 ? Or is it a better way to track it ? |
215: Update pip to 21.0.1 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.3.3** to **21.0.1**. <details> <summary>Changelog</summary> ### 21.0.1 ``` =================== Bug Fixes --------- - commands: debug: Use packaging.version.parse to compare between versions. (`9461 <https://github.com/pypa/pip/issues/9461>`_) - New resolver: Download and prepare a distribution only at the last possible moment to avoid unnecessary network access when the same version is already installed locally. (`9516 <https://github.com/pypa/pip/issues/9516>`_) Vendored Libraries ------------------ - Upgrade packaging to 20.9 ``` ### 21.0 ``` ================= Deprecations and Removals ------------------------- - Drop support for Python 2. (`6148 <https://github.com/pypa/pip/issues/6148>`_) - Remove support for legacy wheel cache entries that were created with pip versions older than 20.0. (`7502 <https://github.com/pypa/pip/issues/7502>`_) - Remove support for VCS pseudo URLs editable requirements. It was emitting deprecation warning since version 20.0. (`7554 <https://github.com/pypa/pip/issues/7554>`_) - Modernise the codebase after Python 2. (`8802 <https://github.com/pypa/pip/issues/8802>`_) - Drop support for Python 3.5. (`9189 <https://github.com/pypa/pip/issues/9189>`_) - Remove the VCS export feature that was used only with editable VCS requirements and had correctness issues. (`9338 <https://github.com/pypa/pip/issues/9338>`_) Features -------- - Add ``--ignore-requires-python`` support to pip download. (`1884 <https://github.com/pypa/pip/issues/1884>`_) - New resolver: Error message shown when a wheel contains inconsistent metadata is made more helpful by including both values from the file name and internal metadata. (`9186 <https://github.com/pypa/pip/issues/9186>`_) Bug Fixes --------- - Fix a regression that made ``pip wheel`` do a VCS export instead of a VCS clone for editable requirements. This broke VCS requirements that need the VCS information to build correctly. (`9273 <https://github.com/pypa/pip/issues/9273>`_) - Fix ``pip download`` of editable VCS requirements that need VCS information to build correctly. (`9337 <https://github.com/pypa/pip/issues/9337>`_) Vendored Libraries ------------------ - Upgrade msgpack to 1.0.2. - Upgrade requests to 2.25.1. Improved Documentation ---------------------- - Render the unreleased pip version change notes on the news page in docs. (`9172 <https://github.com/pypa/pip/issues/9172>`_) - Fix broken email link in docs feedback banners. (`9343 <https://github.com/pypa/pip/issues/9343>`_) .. note You should *NOT* be adding new change log entries to this file, this file is managed by towncrier. You *may* edit previous change logs to fix problems like typo corrections or such. To add a new change log entry, please see https://pip.pypa.io/en/latest/development/contributing/#news-entries .. towncrier release notes start ``` ### 20.3.4 ``` =================== Features -------- - ``pip wheel`` now verifies the built wheel contains valid metadata, and can be installed by a subsequent ``pip install``. This can be disabled with ``--no-verify``. (`9206 <https://github.com/pypa/pip/issues/9206>`_) - Improve presentation of XMLRPC errors in pip search. (`9315 <https://github.com/pypa/pip/issues/9315>`_) Bug Fixes --------- - Fixed hanging VCS subprocess calls when the VCS outputs a large amount of data on stderr. Restored logging of VCS errors that was inadvertently removed in pip 20.2. (`8876 <https://github.com/pypa/pip/issues/8876>`_) - Fix error when an existing incompatibility is unable to be applied to a backtracked state. (`9180 <https://github.com/pypa/pip/issues/9180>`_) - New resolver: Discard a faulty distribution, instead of quitting outright. This implementation is taken from 20.2.2, with a fix that always makes the resolver iterate through candidates from indexes lazily, to avoid downloading candidates we do not need. (`9203 <https://github.com/pypa/pip/issues/9203>`_) - New resolver: Discard a source distribution if it fails to generate metadata, instead of quitting outright. This implementation is taken from 20.2.2, with a fix that always makes the resolver iterate through candidates from indexes lazily, to avoid downloading candidates we do not need. (`9246 <https://github.com/pypa/pip/issues/9246>`_) Vendored Libraries ------------------ - Upgrade resolvelib to 0.5.4. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <github-bot@pyup.io>
Environment
20.2
3.7
and3.8
Python packages are available as git repositories, hosted in AWS CodeCommit, with authentication handled via AWS CLI integration with git credential helper.
Description
Pip version 20.2 causes VCS pip install from AWS CodeCommit to freeze during the clone step. This does not occur with all our repos, but occurs consistently for the ones with the problem.
My team has a suite of python packages that we develop for our internal use. These packages are available as git repositories, and we install them using the pip VCS integration feature.
pip install git+https://git-codecommit.us-east-2.amazonaws.com/v1/repos/MyDemoRepo
This solution has worked for us, without issue, until we upgraded to pip version 20.2. I have tested that all minor versions between 19.1 and 20.1 work as expected.
On version 20.2, with specific repositories, when the install gets to the clone step it stalls there, and the process never completes. The temporary folder is initialized, and the hidden
.git
repo appears to be populated and in-tact, but the rest of the repo is empty and the process never completes.Adding the verbose switch does not provide any detail as to why the process is hanging. Examination of logs on the side of the remote server suggest that the git command is being processed correctly.
Expected behavior
Using
pip install git
on a CodeCommit repo will complete or raise an error for all repositories.How to Reproduce
Reproducing may be difficult, since the issue only occurs with some of our packages, but we don't know why. Our code is not public, so I can't expose it directly. However, the steps are as follows:
.gitconfig
credential helper to use AWS CLI for CodeCommit URLpip install git+https://
; we're not sure why some work and some don't, so this last step is a little, uninformative...Output
The text was updated successfully, but these errors were encountered: