Upgrading package via pip does not remove old files from site-packages

[jt1]

• Pip version: 9.0.1
• Python version: 3.5.2
• Operating system: Ubuntu 16.04.3 LTS

Description:

I build wheel from package which contains two files: a.py and b.py and install it via pip install
Then I remove a.py file and build wheel once again. New wheel contains only b.py file so everything is ok.
When I run try to do update via pip file a.py is not removed from site-packages. Even uninstalling package via pip uninstall leaves a.py file in site-packages

Is there a way to upgrade package and remove old-package files from site-packages?

What I've run:

(foo) ~/foo$ tree
.
├── foo
│   ├── a.py
│   ├── b.py
│   └── __init__.py
└── setup.py

1 directory, 4 files
(foo) ~/foo$ cat setup.py 
from distutils.core import setup

setup(
    name='Foo',
    version='0.1',
    packages=['foo',],
)
(foo) ~/foo$ pip wheel .
Processing /home/qba/foo
Building wheels for collected packages: Foo
  Running setup.py bdist_wheel for Foo ... done
  Stored in directory: /home/qba/foo
Successfully built Foo
(foo) ~/foo$ tree
.
├── foo
│   ├── a.py
│   ├── b.py
│   └── __init__.py
├── Foo-0.1-py3-none-any.whl
└── setup.py

1 directory, 5 files
(foo) ~/foo$ pip install --no-index --ignore-installed --use-wheel --upgrade -v --find-links=file:///home/qba/foo/ foo
Ignoring indexes: https://pypi.python.org/simple
Collecting foo
  0 location(s) to search for versions of foo:
  Skipping link file:///home/qba/foo/ (from -f); not a file
  Skipping link file:///home/qba/foo/setup.py; unsupported archive format: .py
  Skipping link file:///home/qba/foo/foo; not a file
  Found link file:///home/qba/foo/Foo-0.1-py3-none-any.whl, version: 0.1
  Local files found: /home/qba/foo/Foo-0.1-py3-none-any.whl
  Using version 0.1 (newest of versions: 0.1)
Installing collected packages: foo

Successfully installed foo-0.1
Cleaning up...
(foo) ~/foo$ ls /home/qba/.virtualenvs/foo/lib/python3.5/site-packages/foo/
__pycache__  a.py  b.py  __init__.py
(foo) ~/foo$ rm Foo-0.1-py3-none-any.whl 
(foo) ~/foo$ tree
.
├── foo
│   ├── a.py
│   ├── b.py
│   └── __init__.py
└── setup.py

1 directory, 4 files
(foo) ~/foo$ rm foo/a.py 
(foo) ~/foo$ pip wheel .
Processing /home/qba/foo
Building wheels for collected packages: Foo
  Running setup.py bdist_wheel for Foo ... done
  Stored in directory: /home/qba/foo
Successfully built Foo
(foo) ~/foo$ pip install --no-index --ignore-installed --use-wheel --upgrade -v --find-links=file:///home/qba/foo/ foo
Ignoring indexes: https://pypi.python.org/simple
Collecting foo
  0 location(s) to search for versions of foo:
  Skipping link file:///home/qba/foo/ (from -f); not a file
  Skipping link file:///home/qba/foo/setup.py; unsupported archive format: .py
  Skipping link file:///home/qba/foo/foo; not a file
  Found link file:///home/qba/foo/Foo-0.1-py3-none-any.whl, version: 0.1
  Local files found: /home/qba/foo/Foo-0.1-py3-none-any.whl
  Using version 0.1 (newest of versions: 0.1)
Installing collected packages: foo

Successfully installed foo-0.1
Cleaning up...
(foo) ~/foo$ ls /home/qba/.virtualenvs/foo/lib/python3.5/site-packages/foo/
__pycache__  a.py  b.py  __init__.py

[benoit-pierre]

You're using --ignore-installed, that's your issue. IMHO, never a good idea.

[pfmoore]

Explaining in a bit more detail, --ignore-installed does exactly that - it ignores the existing foo package. It doesn't uninstall what's already there, it just installs your modified wheel "over the top" of what's already present. So it'll overwrite a.py and ignore b.py. It'll also overwrite the package metadata, so there's no longer any indication that b.py was ever owned by the foo package, and if you now uninstall foo, b.py will be left behind, and the only way for you to remove it is manually.

[benoit-pierre]

It woud be great if pip would never overwrite an existing file when trying to install something: if the target file already exists, throw an error. And --ignore-installed would do the sensible thing: uninstall an existing version if one already exists in the target location. Unfortunately, that's not how it works...

[jt1]

I expected replacing --ignore-installed with --force-reinstall will do the job but I was wrong. BTW I triad almost all flag combinations for pip install and none removed the file :/

is there really no other method than removing this file manually?

[pfmoore]

Not now you've used --ignore-installed. That broke your system, effectively. I would have expected a simple pip install --upgrade to work as long as you increased your package's version number. If you're changing the files in your package, and not changing the version number, pip's pretty much guaranteed to get confused, because you're lying to it (saying it's the same version of your package when it isn't).

[benoit-pierre]

Once your install as been messed by the use of --ignore-installed, there's no way out of having to manually cleanup leftover files; which may include metadata (e.g. .dist-info entries).

[jt1]

Replacing --ignore-installed with --force-reinstall from the beginning of the installing/upgrading process solved the problem. Thanks for help.

[mukundjalan]

I have never used --ignore-installed but still I am facing the same problem. It so happens that --force-reinstall doesn't work for me. Also using --no-cache-dir did not help. I manually deleted the files in site-packages which are deleted in the latest version of the package, but running pip install -U brings back the files. I have verified that tar.gz file of released package does not have those files. Any pointers on what I could be missing here?

[benoit-pierre]

@mukundjalan: it would help if you provided the exact command used (included the packages being installed).

[mukundjalan]

It is an internal package within our organisation. Lets say named a2z. Following were the commands I tried:

pip uninstall a2z
pip install -U a2z
pip install -U a2z --no-cache-dir --no-deps --force-reinstall

The package contains release_page.xml files in multiple folders which are used as templates for cms page generation. There were certain files in version 2.4.1.

Later we removed those files from few folders and created different files in some other folder and released 2.4.2. I have verified that a2z-2.4.2.tar.gz package does not contain the deleted files. But when package is updated, it replaces the modified files and adds new files successfully but it brings back the old files from version 2.4.1 even if they are manually deleted from site-packages

[benoit-pierre]

I don't see how pip could bring back those old files unless they are present in the new package. You mention those files are not in the source archive, are you sure that's what is being used for installation (and not some other binary package, like a wheel)?

[mukundjalan]

Thanks @benoit-pierre. I checked the packages and found that a2z-2.4.2.tar.gz does not contain those files but a2z-2.4.2-py2-none-any.whl does contain those files. How can I make sure while building the package that the whl package does not contain the deleted files?

[benoit-pierre]

Build it from a clean work tree? As in during a CI build for example.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.