bug(github-installation): reinstalling package from GitHub branch every time I install another package

[Kristinita]

1. Summary

If I installed md_mermaid package from GitHub branch via Pipenv:

    Pipenv re-install this package every time when I install other packages.

In my opinion, this is undesirable behavior.

2. MCVE

See this configuration on KiraInstallGitHubBranchFork branch of my demo GitLab repository for testing Pipenv.

2.1. Pipfile

• Pipfile:

   [dev-packages]

2.2. Steps to reproduce

pipenv install --dev --verbose
# [INFO] My fork of non-working Python package:
# https://github.com/Kristinita/md_mermaid/tree/KiraMermaidWorking
# [INFO] I install it as recommended Stack Overflow answer:
# https://stackoverflow.com/a/52863110/5951529
# [INFO] I also opened an issue on pypa/pipenv on how to install packages from a specific GitHub branch:
# https://github.com/pypa/pipenv/issues/1524
pipenv install -e git+https://github.com/Kristinita/md_mermaid.git@KiraMermaidWorking#egg=md-mermaid --dev --verbose
# [INFO] Any another pip package instead of pylint and pytest also suitable
pipenv install pylint --dev --verbose
pipenv install pytest --dev --verbose

2.3. Desired behavior

No re-installation md_mermaid when I install pylint and pytest.

2.4. Current behavior

1. Travis — 1, 2
2. AppVeyor — 1, 2

Success!
Updated Pipfile.lock (757963)!
Installing dependencies from Pipfile.lock (757963)...
Writing supplied requirement line to temporary file: '-e git+https://github.com/Kristinita/md_mermaid.git@97a2cfe66aaea98e2a9f1797870a21d3d6b8f3b4#egg=md-mermaid'
Installing 'md-mermaid'
$ C:/Users/appveyor/.virtualenvs/kirapipenv--bEYFGX1/Scripts/python.exe 'C:\Python310-x64\Lib\site-packages\pipenv\patched\pip\__pip-runner__.py' install --verbose --upgrade --no-deps --exists-action=i -r 'c:\users\appveyor\appdata\local\temp\1\pipenv-no8u4qx5-requirements\pipenv-af7fm8zo-requirement.txt' -i https://pypi.org/simple
Using source directory: 'C:\\Users\\appveyor\\.virtualenvs\\kirapipenv--bEYFGX1\\src'
Using pip 22.2.2 from C:\Python310-x64\Lib\site-packages\pipenv\patched\pip (python 3.10)
Obtaining md-mermaid from git+https://github.com/Kristinita/md_mermaid.git@97a2cfe66aaea98e2a9f1797870a21d3d6b8f3b4#egg=md-mermaid (from -r c:\users\appveyor\appdata\local\temp\1\pipenv-no8u4qx5-requirements\pipenv-af7fm8zo-requirement.txt (line 1))
  Skipping because already up-to-date.
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Installing collected packages: md-mermaid
  Attempting uninstall: md-mermaid
    Found existing installation: md-mermaid 0.1.2
    Uninstalling md-mermaid-0.1.2:
      Removing file or directory c:\users\appveyor\.virtualenvs\kirapipenv--beyfgx1\lib\site-packages\md-mermaid.egg-link
      Removing pth entries from c:\users\appveyor\.virtualenvs\kirapipenv--beyfgx1\lib\site-packages\easy-install.pth:
      Removing entry: c:\users\appveyor\.virtualenvs\kirapipenv--beyfgx1\src\md-mermaid
      Successfully uninstalled md-mermaid-0.1.2
  Running setup.py develop for md-mermaid
Successfully installed md-mermaid-0.1.2
To activate this project's virtualenv, run pipenv shell.

After installing pylint or pytest, a reinstallation of md_mermaid is started.

3. Environment

1. Operating system:

   1. Local — Microsoft Windows [Version 10.0.19041.1415]
   2. Travis — Ubuntu 22.04 LTS (Jammy Jellyfish)
   3. AppVeyor — Windows Server 2019

2. Python — 3.10.4, 3.10.5, 3.10.6

3. Pipenv — version 2022.8.19

4. pipenv --support

$ pipenv --support

Pipenv version: '2022.8.19'

Pipenv location: 'C:\\Python310\\lib\\site-packages\\pipenv'

setuptools version: '63.2.0'

Python location: 'C:\\Python310\\python.exe'

OS Name: 'nt'

User pip version: '22.2.2'

user Python installations found:

• 3.10.6: C:\Python310\python.exe
• 3.9.7: C:\Python39\python.exe
• 3.7.4: C:\Python37\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.10.6',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.19041',
 'python_full_version': '3.10.6',
 'python_version': '3.10',
 'sys_platform': 'win32'}

System environment variables:

• ALLUSERSPROFILE
• ANDROID_HOME
• ANSICON
• ANSICON_DEF
• API_KEY_APPVEYOR
• API_KEY_KRISTINITA_SEARCH
• API_KEY_PAGESPEED_INSIGHTS_V5
• API_KEY_YANDEX_STRUCTURED_DATA_VALIDATOR
• APPDATA
• CARGO_HOME
• CHOCOLATEYINSTALL
• CHOCOLATEYLASTPATHUPDATE
• CHOCOLATEYTOOLSLOCATION
• CHOCOLATEY_BIN_ROOT
• CHROME_PATH
• CLASSPATH
• CLINK_DIR
• COMMONPROGRAMFILES
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• COMPUTERNAME
• COMSPEC
• CONEMUANSI
• CONEMUANSILOG
• CONEMUARGS
• CONEMUARGS2
• CONEMUBACKHWND
• CONEMUBASEDIR
• CONEMUBASEDIRSHORT
• CONEMUBUILD
• CONEMUCFGDIR
• CONEMUCONFIG
• CONEMUDIR
• CONEMUDRAWHWND
• CONEMUDRIVE
• CONEMUHOOKS
• CONEMUHWND
• CONEMUISADMIN
• CONEMUPALETTE
• CONEMUPID
• CONEMUSERVERPID
• CONEMUTASK
• CONEMUWORKDIR
• CONEMUWORKDRIVE
• DOTNET_CLI_TELEMETRY_OPTOUT
• DRIVERDATA
• FARADMINMODE
• FARHOME
• FARLANG
• FARLOCALPROFILE
• FARPROFILE
• FPS_BROWSER_APP_PROFILE_STRING
• FPS_BROWSER_USER_PROFILE_STRING
• GEM_HOME
• GEM_PATH
• GITHUB_TOKEN
• GIT_EDITOR
• GIT_TOKEN
• GNUPLOT_LIB
• GOPATH
• GRUNT_PYLINT_SKIP_POSTINSTALL
• GTK_BASEPATH
• HOME
• HOMEDRIVE
• HOMEPATH
• JAR_PDFTK
• JAVA_HOME
• JD2_HOME
• K2PDFOPT_CUSTOM0
• K2PDFOPT_CUSTOM1
• K2PDFOPT_CUSTOM2
• K2PDFOPT_WINPOS
• LESSCHARSET
• LOCALAPPDATA
• LOGONSERVER
• MAGICK_CODER_MODULE_PATH
• MAGICK_CONFIGURE_PATH
• MAGICK_HOME
• MYPATH
• NODE_NO_WARNINGS
• NUMBER_OF_PROCESSORS
• ONEDRIVE
• OPENSSL_CONF
• OS
• PATH
• PATHEXT
• PATH_AUTOHOTKEY
• PATH_GINGERINAS_NOTES
• PATH_KIROMANIA
• PATH_PORTABLEAPPS
• PATH_VIDI_DC
• PERCY_TOKEN
• PIPENV_IGNORE_VIRTUALENVS
• PIPENV_SKIP_LOCK
• PIPENV_VENV_IN_PROJECT
• PROCESSOR_ARCHITECTURE
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMDATA
• PROGRAMFILES
• PROGRAMFILES(X86)
• PROGRAMW6432
• PROJECTS_PATH
• PROMPT
• PSMODULEPATH
• PUBLIC
• PYTHONIOENCODING
• PYTHONPACKAGES
• PYTHONPATH
• RADONFILESENCODING
• RUSTUP_HOME
• RUST_BACKTRACE
• RUST_LOG
• SASHABACKUPSTORAGE
• SASHASECONDBACKUPSTORAGE
• SCOOP
• SCROLLVIEW_PATH
• SESSIONNAME
• SUBLIME
• SUBLIME_COLOR_SCHEME_UNIT_DEBUG
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• TESSDATA_PREFIX
• TMP
• TWINE_PASSWORD
• TWINE_USERNAME
• UD_INSTALL_DIR
• USERDOMAIN
• USERDOMAIN_ROAMINGPROFILE
• USERNAME
• USERPROFILE
• VS140COMNTOOLS
• WINDIR
• XDG_CONFIG_DIR
• _MSYS2_BASH
• _MSYS2_PREFIX
• PIP_SHIMS_BASE_MODULE
• PIP_DISABLE_PIP_VERSION_CHECK
• PIP_PYTHON_PATH
• PYTHONDONTWRITEBYTECODE
• PYTHONFINDER_IGNORE_UNSUPPORTED

PipenvтАУspecific environment variables:

• PIPENV_IGNORE_VIRTUALENVS: 1
• PIPENV_SKIP_LOCK: 1
• PIPENV_VENV_IN_PROJECT: 1

DebugтАУspecific environment variables:

• PATH: C:\Program Files\ConEmu\ConEmu\Scripts;C:\Program Files\ConEmu\ConEmu;C:\Program Files\ImageMagick-7.1.0-Q16-HDRI;;C:\Program Files\Eclipse Adoptium\jre-8.0.342.7-hotspot\bin;c:\program files\graphicsmagick-1.3.36-q8;C:\Python310\Scripts\;C:\Python310\;C:\Python39\Scripts\;C:\Python39\;C:\Program Files\Docker\Docker\Resources\bin;D:\SashaPrograms\;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;D:\Chocolatey\bin;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\gnuplot\bin;C:\Android\android-sdk\tools;C:\Program Files\OpenVPN\bin;C:\Program Files\Calibre2\;C:\Program Files\dotnet\;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin;C:\Users\SashaChernykh\.dotnet\tools;D:\Sublime Text 3 x64;C:\Program Files (x86)\GnuWin32\bin;C:\Users\SashaChernykh\AppData\Local\Pandoc\;D:\Chocolatey\lib\cloneapp\tools;C:\Users\SashaChernykh\.local\bin;D:\Chocolatey\tools\msys64;D:\Chocolatey\lib\mingw\tools\install\mingw64\bin;C:\Users\SashaChernykh\go\bin;C:\Users\SashaChernykh\AppData\Roaming\npm;D:\Chocolatey\tools\BCURRAN3;C:\Program Files\Java\jdk1.8.0_231\bin;C:\Program Files (x86)\GtkSharp\2.12\bin;D:\Chocolatey\lib\mpv.install\tools;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\Program Files (x86)\Universal Extractor;C:\Program Files (x86)\Universal Extractor\bin;C:\ProgramData\ComposerSetup\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Go\bin;C:\Program Files\Git\cmd;C:\Program Files\gs\gs9.56.1\bin;D:\Chocolatey\tools\php81;C:\Program Files\nodejs\;C:\texlive\2022\bin\win32;C:\Program Files\Cloudflare\Cloudflare WARP\;D:\Scoop\apps\rustup\current\.cargo\bin;D:\Scoop\apps\imagemagick\current;D:\Scoop\apps\ruby\current\bin;D:\Scoop\apps\ruby\current\gems\bin;D:\Scoop\apps\ghostscript\current\lib;D:\Scoop\apps\miniconda3\current\scripts;D:\Scoop\apps\miniconda3\current\Library\bin;D:\SashaPrograms\jpdfbookmarks;C:\Program Files\nodejs;D:\SashaPrograms\jbig2;D:\SashaPrograms\Weeny Free PDF Merger;D:\SashaForks\vcpkg\packages\protobuf_x64-windows\tools\protobuf;C:\Program Files\ConEmu;D:\SashaPrograms\ripgrep_all;D:\PortableApps\PortableApps\FirefoxPortable\App\firefox64;C:\Program Files\FreeFileSync;D:\Scoop\shims;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin;C:\Users\SashaChernykh\.dotnet\tools;D:\Sublime Text 3 x64;C:\Program Files (x86)\GnuWin32\bin;D:\Chocolatey\bin;C:\Users\SashaChernykh\AppData\Local\Pandoc\;D:\Chocolatey\lib\cloneapp\tools;C:\Users\SashaChernykh\.local\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl;D:\Chocolatey\lib\mingw\tools\install\mingw64\bin;C:\Users\SashaChernykh\go\bin;C:\Program Files (x86)\DjVuLibre;C:\Program Files\OpenSSL-Win64\bin;C:\Program Files (x86)\Vale\;C:\Users\SashaChernykh\go\bin;C:\Users\SashaChernykh\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Tesseract-OCR;C:\Users\SashaChernykh\AppData\Roaming\npm

---

Contents of Pipfile ('D:\SashaDemoRepositories\KiraPipenv\Pipfile'):

[dev-packages]

Thanks.

[dqkqd]

When new packages arrive. pipenv installs them, checks existing packages to resolve conflict. Then it decides which one should be re-installed using pip.
Since you have editable package, pip will always re-install it. pypa/pip#9169.
So I guess this is expected behavior?

[matteius]

@Kristinita I get what the ask is, but as @dqkqd points out it is likely a pip behavior that may be intentional. That being said, I don't think it makes sense to issue pipenv install <pkg_name> command during the CI flow -- because you loose the ability to provide an already generated lock file which has the prior package hashes that can be validated during the pipenv sync phase, which is more common to use in the CI.

For the case of your example, if you ran pipenv install pytest pylint --dev it would only retrigger the install once.

[Kristinita]

@dqkqd , @matteius

Type: Questions ❓

1. Where can I read why this behavior is done? It’s not at all obvious to me why pipenv reinstall a package every time I install other packages that have nothing to do with the package being reinstalled. I read #9169, #9147, #8711, #9116, but didn’t understand the need for this behavior in cases, as in my example.
2. Where can I see information about this behavior in the pipenv documentation? Intuitively, this isn’t at all obvious, and the user may well think that this is a bug.
3. How can I disable this behavior? Are there any options, command line arguments, environment variables?

Thanks.

[matteius]

@Kristinita

Sorry, pipenv uses pip to perform resolutions and installs but we do not document every single pip behavior in our documentation. Nor can I explain this particular decision very well other than to say there are cases where the file installs need to be reinstalled and it probably is difficult to determine if the package you have installed is different from the one you are going to install in these cases. As far as I know, pip provides no way to disable this behavior. You may want to take up this discussion with pip because should they changed their behavior, then it would be reflected in a future pipenv release. Additionally, if there were such an argument that could be passed to pip, doing so would be made possible by #5283