Old dependency from install_requires not removed from env after removal from manifest

[glenjamin]

After removing a dependency from install_requires in my setup.py, and running pipenv update, the removed dependency is still present in the environment.

---

Please run $ python -m pipenv.help, and paste the results here.

Pipenv version: '11.8.0'

Pipenv location: '/usr/local/Cellar/pipenv/11.8.0/libexec/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/11.8.0/libexec/bin/python'

Other Python installations in PATH:

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/local/bin/python3.6m

• 3.6: /usr/local/bin/python3.6

• 2.7.14: /usr/local/bin/python

• 2.7.10: /usr/bin/python

• 2.7.14: /usr/local/bin/python2

• 3.6.4: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.4.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST '
                     '2017; root:xnu-4570.41.2~1/RELEASE_X86_64',
 'python_full_version': '3.6.4',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

---

Expected result

I would expect that the dependency is removed from the environment, as nothing depends on it anymore.

Actual result

See detailed steps to replicate below:

Steps to replicate

Create the following simple setup.py

from setuptools import setup

setup(
    name='pipenvtest',
    version='0.1.0',
    description='testing',
    license='MIT',
    install_requires=[
        'plain_obj ~= 0.1.2',
    ]
)

Create a new pipenv by installing the current directory as editable
pipenv install -e .

Check the dependency graph for the env, all looks good:

> pipenv graph
pipenvtest==0.1.0
  - plain-obj [required: ~=0.1.2, installed: 0.1.2]

Update setup.py to comment out the one dependency - this simulates removing a dependency

    install_requires=[
        # 'plain_obj ~= 0.1.2',
    ]

Run pipenv update to update the lockfile and environment

> pipenv update
Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (86895b)!
Installing dependencies from Pipfile.lock (86895b)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
To activate this project's virtualenv, run the following:
 $ pipenv shell
All dependencies are now up-to-date!

Run pipenv graph to view the environment

> pipenv graph
pipenvtest==0.1.0
plain-obj==0.1.2

plain-obj is still there - although it is no longer present in the lockfile.

[glenjamin]

I've just spotted the pipenv clean command - not sure how I overlooked this before.

Is there anywhere I can read up on why clean is a distinct separate step rather than part of sync?

[techalchemy]

Yes. Search the issues for the thoroughly outlined discussion about the sync and lock commands. You will find #1463 which outlines the command and everything it does. Nothing that pipenv does is secretly destructive. If you want to destroy your environment you will have to tell pipenv that explicitly.

Pipenv is not magic. It doesn’t confirm that the accumulated resolved dependencies are all that is installed. setup.py is an install directive, so presence means installation. Absence does not mean uninstallation.

[glenjamin]

Ok, that mostly makes sense.

The word sync implies to me that it would synchronise my environment against my lockfile - although to be fair the help text "Installs all packages specified in Pipfile.lock." doesn't say this. The current semantics appear to be more of an ensure. (I'm not suggesting it be renamed).

You say that nothing pipenv does is secretly destructive - but does that mean sync will also refuse to downgrade a dependency that was previously installed if the lockfile has changed?

It would be useful to future users if the documentation contained an overview of the commands and the scenarios they're expected to be used.

The same behaviour is seen if a dependency is removed from the Pipfile and lock + sync are run. I think it is non-obvious that the environment isn't fully synchronised until also running the clean command.

[techalchemy]

If re-resolving dependencies results in a lower version being put in the lockfile, sync installs that one. That’s the whole point of pipenv. Once it’s out of the lockfile, if you didn’t call clean or uninstall, pipenv isn’t managing it. We don’t assume we are managing every package in the environment all the time (see —system or note that pipenv can be installed and used inside a virtualenv).

So does pipenv refuse to downgrade something that was explicitly resolved to a lower version? No. That is not a secret action. It is a strict pin in the lockfile. Things in the lockfile always get installed in the environment. Just imagine as if we iterate over the list of things we determined need to be installed and for each of those we pip install it (because this is exactly what happens).

It may not have been obvious to you, but I would be interested in seeing an example of dependency management software that behaves the way you expect pipenv to behave.

[tsiq-oliver]

@techalchemy - In the sense of "deleting things that are no longer specified?". NPM 5 in the JS world, or Gradle/Maven in the Java world.

[techalchemy]

@tsiq-oliverc can you link an example of how that works? I'm not sure that makes sense here given the above use cases but I want to be sure (for instance what happens if the user is working in the global namespace with the npm usecase?)

[glenjamin]

We don’t assume we are managing every package in the environment all the time (see —system or note that pipenv can be installed and used inside a virtualenv).

This is the piece of information which I was missing makes everything else fall into place. Because pipenv had created my virtualenv I was under the impression that it was fully managing it. I wasn't aware of the options to use it on the system or another existing virtualenv.

---

npm and yarn are usually working on a local node_modules folder, so they assume they're allowed to have full control. When you work on the global node_modules, a lockfile isn't normally used so this behaviour doesn't apply.

The Bundler behaviour is also similar, although because it works by assembling a complicated load path pointing to centrally stored depdendencies the comparison isn't quite the same - removing things from your lockfile removes them from your bundle - but the files themselves stay intact. I believe this is pretty much what maven and gradle do too.

I wonder if it would be possible for pipenv to mark the virtualenvs it creates in some way as "fully managed", which could default to the clean behaviour when doing a sync. This might be a bit too magic, so perhaps including a section about managed vs shared environments (or whatever the correct term should be), and having that section make some references to the clean behaviour would be a good step forwards here.

I see that the sync command is much newer than i'd first realised, so I can see why it is only lightly mentioned in the documentation so far.

[tsiq-oliver]

@techalchemy - Basically what @glenjamin said. NPM/Yarn's global thing is a bit of a special snowflake, which one generally tries to avoid if trying to construct a repo with reproducible build results. And you get this behaviour by default in the Java world (with Maven/Gradle/etc.) - the Gradle cache is not affected by removing deps in your build files, but the classpath exposed to your compiler/app is. Coming from these worlds, the current behaviour of sync is highly unintuitive to me 😭

More generally, ignoring the global/system issue, the idea of a virtualenv that's not fully managed is going to lead to all sorts of issues - there are a bunch of beneficial invariants that one has to give up. Is that really a tax that Pipenv wants to endure?

[techalchemy]

Our documentation definitely needs a good update.

We can consider removing dependencies, but I feel like we did discuss it at some point. I will try to find the discussion. I get the usability perspective....will need to consider the possible consequences

[uranusjr]

@tsiq-oliverc I don’t object your point, just want to add a note.

[T]he Gradle cache is not affected by removing deps in your build files, but the classpath exposed to your compiler/app is.

Unfortunately this is not how Python (and Node, for this particular matter) packages work. If it’s installed, it is visible for import. That is probably why NPM and Yarn don’t do it by default as well.

[glenjamin]

I think there are two related discussion points here.

1. should pipenv have a command or a mode which “fully manages” an environment.

2. if a mode existed, how would you know when to use it?

As a point of comparison, npm < 5 didn’t use a lock file by default, and has install and prune commands which are similar to what pipenv has now.

However, whenever a lockfile is present, which is “always” in npm 5, the install command with no additional arguments will perform a sync which removes packages no longer in the lockfile.

[tsiq-oliver]

@uranusjr - Indeed. In practice that's achieved by leaving things in the Gradle cache, and dynamically constructing a classpath pointing to each "available" lib in the cache.

I suppose a big difference is that Gradle doesn't have to build dependencies from source; once you download a Jar/Pom into the cache you're done. Though naively it doesn't strike me as impossible for Python deps to be cached post-build, as an optimisation for a world where sync does a "destructive" sync. (Though that's probably in the realm of Pip rather than Pipenv.)

[uranusjr]

@tsiq-oliverc Python already does. I don’t know when it started, but modern pip versions build a wheel when you install from source. That wheel (binary format analogous to Java’s jar) is cached locally, so you don’t have to download/build it again next time you want to install the same package. Wheels, however, generally depends on the Python interpreter’s version (unless it’s pure Python, but in which case the build overhead is minimal anyway), and the cache is global, so the probability of cache misses are much higher. It also does not cache everything (e.g. VCS installations), leaving Pipenv in a strange place.

[uranusjr]

No use beating the dead horse though. We should decide whether to go this route based solely on whether it is easy enough to do in Python. How and why Java does this is not really relevant.