New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pipenv clean does not clean packages from the virtual env, but some packages from the global env #2849

Closed
ociule opened this Issue Sep 13, 2018 · 12 comments

Comments

Projects
None yet
4 participants
@ociule

ociule commented Sep 13, 2018

Issue description

I am trying to run pipenv clean after removing a package from Pipfile and running pipfile lock

Expected result

That the removed packaged is uninstalled by pipenv clean.

Actual result

Pipenv actually tries to remove 4 packages from the global env: pipenv, virtualenv, virtualenv-clone and certifi. It doesn't actually remove them. Maybe it's actually removing them from another virtualenv ?

Steps to replicate

Create a project, add a package with pipenv install.

Remove it by editing the Pipfile.

Run pipfile lock to update Pipfile.lock

Run pipfile clean to remove the package you edited out of Pipfile.

Pipenv clean --dry-run just lists the names of the 4 packages it tries to remove: pipenv, virtualenv, virtualenv-clone, certifi

I then run pip list and the package I removed from Pipfile is still there in the venv. The 4 packages it tried to remove are not even installed in the venv.

I have tested this on both the latest pip release and pipenv master, on Windows. I've tested on both cmd.exe and git bash. Yeah I know bash is not officially supported, that's why I reproed on cmd.exe

This happens in the venv shell or not, and by running 'pipenv run pipenv clean' too.

Update:
Amusingly, if I do install one of the four packages in the venv, it does uninstall it, which points to the code that gets that list as being faulty. The code that tries to uninstall the packages from that list is at fault for failing silently, though.

I can try to repro on Mac OS later tonight


$ pipenv --support

Pipenv version: '2018.7.1.dev0'

Pipenv location: 'C:\\Users\\safco106950\\AppData\\Roaming\\Python\\Python37\\site-packages\\pipenv'

Python location: 'c:\\users\\safco106950\\appdata\\local\\programs\\python\\python37\\python.exe'

Python installations found:

  • 3.7.0: C:\Users\safco106950\AppData\Local\Programs\Python\Python37\python.exe
  • 3.6.5: C:\Users\safco106950\AppData\Local\Programs\Python\Python36-32\python.exe
  • 3.5: C:\Users\safco106950\AppData\Local\Programs\Python\Python35\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '7',
 'platform_system': 'Windows',
 'platform_version': '6.1.7601',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'win32'}

System environment variables:

  • ALLUSERSPROFILE
  • APPDATA
  • CLASSPATH
  • COMMONPROGRAMFILES
  • COMMONPROGRAMFILES(X86)
  • COMMONPROGRAMW6432
  • COMPUTERNAME
  • COMSPEC
  • DADIR
  • DEFLOGDIR
  • FP_NO_HOST_CHECK
  • HOMEDRIVE
  • HOMEPATH
  • LOCALAPPDATA
  • LOGONSERVER
  • NUMBER_OF_PROCESSORS
  • OS
  • PATH
  • PATHEXT
  • PIPENV_ACTIVE
  • PIP_PYTHON_PATH
  • PROCESSOR_ARCHITECTURE
  • PROCESSOR_IDENTIFIER
  • PROCESSOR_LEVEL
  • PROCESSOR_REVISION
  • PROGRAMDATA
  • PROGRAMFILES
  • PROGRAMFILES(X86)
  • PROGRAMW6432
  • PROMPT
  • PSMODULEPATH
  • PUBLIC
  • PYTHONDONTWRITEBYTECODE
  • QTJAVA
  • SAPLOGON_INI_FILE
  • SESSIONNAME
  • SYSTEMDRIVE
  • SYSTEMROOT
  • TEMP
  • TMP
  • UATDATA
  • USERDNSDOMAIN
  • USERDOMAIN
  • USERDOMAIN_ROAMINGPROFILE
  • USERNAME
  • USERPROFILE
  • VIRTUAL_ENV
  • VSEDEFLOGDIR
  • WINDIR
  • WINDOWS_TRACING_FLAGS
  • WINDOWS_TRACING_LOGFILE

Pipenvûspecific environment variables:

  • PIPENV_ACTIVE: 1

Debugûspecific environment variables:

  • PATH: C:\Users\safco106950\.virtualenvs\jupyter_test-UM4uSYuv\Scripts;C:\Users\safco106950\AppData\Roaming\Python\Python37\Scripts;C:\Users\safco106950\AppData\Local\Programs\Python\Python37\Scripts;C:\users-apps\Oracle\ora1123\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Common Files\Hitachi ID;C:\Program Files (x86)\Microsoft Application Virtualization Client;C:\Program Files (x86)\QuickTime\QTSystem\;C:\Program Files (x86)\Enterprise Vault\EVClient\;C:\Program Files (x86)\CheckPoint\Endpoint Security\Endpoint Common\bin;C:\Users\safco106950\AppData\Local\Programs\Python\Launcher\;C:\Users\safco106950\AppData\Local\Programs\Python\Python36-32\Scripts\;C:\Users\safco106950\AppData\Local\Programs\Python\Python36-32\;C:\Users\safco106950\AppData\Local\atom\bin;C:\Users\safco106950\AppData\Local\Pandoc\
  • VIRTUAL_ENV: C:\Users\safco106950\.virtualenvs\jupyter_test-UM4uSYuv

Contents of Pipfile ('C:\Users\safco106950\work\jupyter_test\Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('C:\Users\safco106950\work\jupyter_test\Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "415dfdcb118dd9bdfef17671cb7dcd78dbd69b6ae7d4f39e8b44e71d60ca72e7"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}
@ociule

This comment has been minimized.

ociule commented Sep 13, 2018

Just thought of adding verbose to clean:

λ pipenv clean -vvv               
Ignoring 'pip'.                   
Ignoring 'setuptools'.            
Uninstalling 'pipenv'…            
Uninstalling 'virtualenv'…        
Uninstalling 'virtualenv-clone'…  
Uninstalling 'certifi'…           

Doesn't help all that much.

@uranusjr

This comment has been minimized.

Member

uranusjr commented Sep 13, 2018

I assume you have global packages enabled in the virtualenv. Pipenv uses pip for removal, and pip just ignores non-local packages when you try to uninstall them, so my guess is nothing is not actually removed, those are just no-op.

Where are those packages actually installed at? Try printing their __file__ attribute in Python.

@ociule

This comment has been minimized.

ociule commented Sep 14, 2018

Those packages are not in the local venv:

(jupyter_test-UM4uSYuv) C:\Users\safco106950\work\jupyter_test>python -c "import
 certifi"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'certifi'

After "exit", going back to global python:

C:\Users\safco106950\work\jupyter_test>python -c "import certifi; print(certifi.
__file__)"
C:\Users\safco106950\AppData\Local\Programs\Python\Python36-32\lib\site-packages
\certifi\__init__.py

As expected

The fact that they're not in the local venv is further shown by the experiment described above, where I install virtualenv in the local venv using pip, then run pipenv clean. This now succeeds in uninstalling virtualenv (and only virtual env, even if it prints the same 4 as always) from the local venv.

@ociule

This comment has been minimized.

ociule commented Sep 14, 2018

The output of "pipenv run pip list" doesn't show any of those 4. There are lots of packages in the venv, whereas my Pipfile is empty. I've tried with a non-empty pipfile, to check it's not caused by that.

techalchemy added a commit that referenced this issue Oct 10, 2018

Properly scope pipenv clean to local env
- Fixes #2849

Signed-off-by: Dan Ryan <dan@danryan.co>
@svenpanne

This comment has been minimized.

svenpanne commented Oct 10, 2018

Just another data point: pipenv clean (from current master) does not only emit tons of messages related to packages outside the virtual environment, it actually removes packages which pipenv install has just installed. I am not sure if this is fixed by #2976 , too, or if this is just another symptom of the current confusion regarding normalized/non-normalized package names. Simple steps to reproduce in isolation:

  1. Create an empty directory and cd into it.
  2. Run PIPENV_VENV_IN_PROJECT=true pipenv install pillow. pillow is happily installed into the virtual environment and it is contained in Pipfile.lock:
{
    "_meta": {
        "hash": {
            "sha256": "522f147da35e9204817c9f43302b2e1ce76b85c6055c7c23422787950b3c459d"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "2.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "pillow": {
            "hashes": [
                "sha256:00203f406818c3f45d47bb8fe7e67d3feddb8dcbbd45a289a1de7dd789226360",
                "sha256:0616f800f348664e694dddb0b0c88d26761dd5e9f34e1ed7b7a7d2da14b40cb7",
                "sha256:1f7908aab90c92ad85af9d2fec5fc79456a89b3adcc26314d2cde0e238bd789e",
                "sha256:2ea3517cd5779843de8a759c2349a3cd8d3893e03ab47053b66d5ec6f8bc4f93",
                "sha256:48a9f0538c91fc136b3a576bee0e7cd174773dc9920b310c21dcb5519722e82c",
                "sha256:5280ebc42641a1283b7b1f2c20e5b936692198b9dd9995527c18b794850be1a8",
                "sha256:5e34e4b5764af65551647f5cc67cf5198c1d05621781d5173b342e5e55bf023b",
                "sha256:63b120421ab85cad909792583f83b6ca3584610c2fe70751e23f606a3c2e87f0",
                "sha256:696b5e0109fe368d0057f484e2e91717b49a03f1e310f857f133a4acec9f91dd",
                "sha256:870ed021a42b1b02b5fe4a739ea735f671a84128c0a666c705db2cb9abd528eb",
                "sha256:916da1c19e4012d06a372127d7140dae894806fad67ef44330e5600d77833581",
                "sha256:9303a289fa0811e1c6abd9ddebfc770556d7c3311cb2b32eff72164ddc49bc64",
                "sha256:9577888ecc0ad7d06c3746afaba339c94d62b59da16f7a5d1cff9e491f23dace",
                "sha256:987e1c94a33c93d9b209315bfda9faa54b8edfce6438a1e93ae866ba20de5956",
                "sha256:99a3bbdbb844f4fb5d6dd59fac836a40749781c1fa63c563bc216c27aef63f60",
                "sha256:99db8dc3097ceafbcff9cb2bff384b974795edeb11d167d391a02c7bfeeb6e16",
                "sha256:a5a96cf49eb580756a44ecf12949e52f211e20bffbf5a95760ac14b1e499cd37",
                "sha256:aa6ca3eb56704cdc0d876fc6047ffd5ee960caad52452fbee0f99908a141a0ae",
                "sha256:aade5e66795c94e4a2b2624affeea8979648d1b0ae3fcee17e74e2c647fc4a8a",
                "sha256:b78905860336c1d292409e3df6ad39cc1f1c7f0964e66844bbc2ebfca434d073",
                "sha256:b92f521cdc4e4a3041cc343625b699f20b0b5f976793fb45681aac1efda565f8",
                "sha256:bfde84bbd6ae5f782206d454b67b7ee8f7f818c29b99fd02bf022fd33bab14cb",
                "sha256:c2b62d3df80e694c0e4a0ed47754c9480521e25642251b3ab1dff050a4e60409",
                "sha256:c5e2be6c263b64f6f7656e23e18a4a9980cffc671442795682e8c4e4f815dd9f",
                "sha256:c99aa3c63104e0818ec566f8ff3942fb7c7a8f35f9912cb63fd8e12318b214b2",
                "sha256:dae06620d3978da346375ebf88b9e2dd7d151335ba668c995aea9ed07af7add4",
                "sha256:db5499d0710823fa4fb88206050d46544e8f0e0136a9a5f5570b026584c8fd74",
                "sha256:f36baafd82119c4a114b9518202f2a983819101dcc14b26e43fc12cbefdce00e",
                "sha256:f52b79c8796d81391ab295b04e520bda6feed54d54931708872e8f9ae9db0ea1",
                "sha256:ff8cff01582fa1a7e533cb97f628531c4014af4b5f38e33cdcfe5eec29b6d888"
            ],
            "index": "pypi",
            "version": "==5.3.0"
        }
    },
    "develop": {}
}
  1. Run PIPENV_VENV_IN_PROJECT=true pipenv run pip list. This yield the expected output:
Package    Version
---------- -------
Pillow     5.3.0  
pip        18.1   
setuptools 40.4.3 
wheel      0.32.1 
  1. Run PIPENV_VENV_IN_PROJECT=true pipenv clean. This gives a loooong output, although it should actually do nothing:
Uninstalling 'virtualenv'…
Uninstalling 'virtualenv-clone'…
Uninstalling 'typing'…
Uninstalling 'pipenv'…
Uninstalling 'certifi'…
Uninstalling 'zope.interface'…
Uninstalling 'wstools'…
Uninstalling 'wrapt'…
Uninstalling 'webencodings'…
Uninstalling 'wadllib'…
Uninstalling 'urllib3'…
Uninstalling 'unity-lens-photos'…
Uninstalling 'ubuntu-dev-tools'…
Uninstalling 'trollius'…
Uninstalling 'subprocess32'…
Uninstalling 'Sphinx'…
Uninstalling 'SOAPpy'…
Uninstalling 'six'…
Uninstalling 'singledispatch'…
Uninstalling 'simplejson'…
Uninstalling 'SecretStorage'…
Uninstalling 'scour'…
Uninstalling 'scandir'…
Uninstalling 'roman'…
Uninstalling 'requests'…
Uninstalling 'PyYAML'…
Uninstalling 'pyxdg'…
Uninstalling 'pyudev'…
Uninstalling 'pytz'…
Uninstalling 'python-djvulibre'…
Uninstalling 'python-debian'…
Uninstalling 'python-dateutil'…
Uninstalling 'python-apt'…
Uninstalling 'pytest'…
Uninstalling 'Pyro'…
Uninstalling 'pyparsing'…
Uninstalling 'pyOpenSSL'…
Uninstalling 'PyICU'…
Uninstalling 'pygobject'…
Uninstalling 'Pygments'…
Uninstalling 'pyflakes'…
Uninstalling 'pyenchant'…
Uninstalling 'pydns'…
Uninstalling 'pycrypto'…
Uninstalling 'pycodestyle'…
Uninstalling 'pycairo'…
Uninstalling 'py'…
Uninstalling 'polib'…
Uninstalling 'pluggy'…
Uninstalling 'Pillow'…
Uninstalling 'pep8'…
Uninstalling 'pathlib2'…
Uninstalling 'olefile'…
Uninstalling 'oauth'…
Uninstalling 'numpy'…
Uninstalling 'neovim'…
Uninstalling 'msgpack'…
Uninstalling 'mercurial'…
Uninstalling 'mccabe'…
Uninstalling 'matplotlib'…
Uninstalling 'MarkupSafe'…
Uninstalling 'lxml'…
Uninstalling 'lazy-object-proxy'…
Uninstalling 'lazr.uri'…
Uninstalling 'lazr.restfulclient'…
Uninstalling 'launchpadlib'…
Uninstalling 'keyrings.alt'…
Uninstalling 'keyring'…
Uninstalling 'Jinja2'…
Uninstalling 'ipaddress'…
Uninstalling 'imagesize'…
Uninstalling 'idna'…
Uninstalling 'httplib2'…
Uninstalling 'html5lib'…
Uninstalling 'greenlet'…
Uninstalling 'futures'…
Uninstalling 'functools32'…
Uninstalling 'funcsigs'…
Uninstalling 'flake8'…
Uninstalling 'enum34'…
Uninstalling 'docutils'…
Uninstalling 'distro-info'…
Uninstalling 'defusedxml'…
Uninstalling 'defer'…
Uninstalling 'cycler'…
Uninstalling 'cryptography'…
Uninstalling 'configparser'…
Uninstalling 'configobj'…
Uninstalling 'compizconfig-python'…
Uninstalling 'chardet'…
Uninstalling 'ccsm'…
Uninstalling 'bzr'…
Uninstalling 'bzr-builddeb'…
Uninstalling 'beautifulsoup4'…
Uninstalling 'backports.functools-lru-cache'…
Uninstalling 'Babel'…
Uninstalling 'attrs'…
Uninstalling 'astroid'…
Uninstalling 'asn1crypto'…
Uninstalling 'alabaster'…

Apart from the confusing lines mentioning global packages (which seem to be no-ops), it acutally uninstalls Pillow from the virtual environment, ignoring Pipfile.lock.

In a nutshell: pipenv clean in its current state is totally broken. :-(

@ociule

This comment has been minimized.

ociule commented Oct 10, 2018

Everything that svenpanne described is consistent with my understanding of the bug.

@ociule

This comment has been minimized.

ociule commented Oct 10, 2018

This bug makes using pipenv clean dangerous on continuous integration tools - previously it was a part of our solution for reusing CI python environments for reducing setup times.

@svenpanne

This comment has been minimized.

svenpanne commented Oct 10, 2018

This bug makes using pipenv clean dangerous on continuous integration tools - previously it was a part of our solution for reusing CI python environments for reducing setup times.

This is exactly our scenario here, too: Do a pipenv install --dev followed by a pipenv clean before a CI run.

@techalchemy

This comment has been minimized.

Member

techalchemy commented Oct 10, 2018

For sure. No fix has been merged yet but I would like to get one in. In the mean time you might want to try https://GitHub.com/sarugaku/mork or https://nox.readthedocs.io/en/stable/ depending on your needs. Nox has a nicer API but I’m guessing uses subprocess calls

@svenpanne

This comment has been minimized.

svenpanne commented Oct 11, 2018

OK, using the current master of pipenv, things have improved: All those 'Uninstalling...' messages for global packages are gone, but pipenv clean still wants to remove Pillow. I guess this is related to the current confusion regarding (non-)normalized package names, because doing the same for e.g. six works fine (i.e. it doesn't get removed).

I'm a bit confused by now: Is the naming confusion bug a pipenv bug or a pip bug? Should I open a new issue for the incorrect removal of Pillow? If yes, in which project?

@techalchemy

This comment has been minimized.

Member

techalchemy commented Oct 17, 2018

Ah yes please, open it here with output included @svenpanne — thanks and sorry for the trouble!

@svenpanne

This comment has been minimized.

svenpanne commented Oct 17, 2018

Ah yes please, open it here with output included @svenpanne — thanks and sorry for the trouble!

I just reinstalled pipenv from master (8212a21), and pipenv clean seems to work fine now, so from my POV there is no need for a separate issue anymore.

Can we get a new pipenv release soon? All recent versions release so far had one or more serious bugs making them unusable in our CI and daily work... :-(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment