Package with pyproject.toml cannot be installed with pip install -e

[erzoe]

OS version

Debian 11

Python version

Python 3.9.2

Pip version

pip 21.3.1

Guide link

https://packaging.python.org/en/latest/tutorials/packaging-projects/

Problem description

I have created an example project as explained in Packaging Python Projects with the following pyproject.toml file:

[build-system]
requires = [
    "setuptools>=42",
    "wheel"
]
build-backend = "setuptools.build_meta"

and setup.cfg:

[metadata]
name = example-pkg-erzo
version = file: src/example_package/version.txt
author = Example Author
author_email = author@example.com
description = A small example package
long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/pypa/sampleproject
project_urls =
    Bug Tracker = https://github.com/pypa/sampleproject/issues
classifiers =
    Programming Language :: Python :: 3
    License :: OSI Approved :: MIT License
    Operating System :: OS Independent

[options]
package_dir =
    = src
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src

[options.entry_points]
console_scripts =
    example-program = example_package.main:main
    hello-world = example_package.greet:main

Also I have added a setup.py because I need to include data files:

#!/usr/bin/env python3

import setuptools

setuptools.setup(package_data={"":["*.txt", "doc/*"]})

When I am trying to install the package with pip install -e . I am getting the error message listed in the next section.

Removing pyproject.toml resolves the problem. But does it cause other problems that I am missing right now?

Error message

Defaulting to user installation because normal site-packages is not writeable
Obtaining file://.../example_package_git
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Installing collected packages: example-pkg-erzo
  Running setup.py develop for example-pkg-erzo
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'.../example_package_git/setup.py'"'"'; __file__='"'"'.../example_package_git/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix=
         cwd: .../example_package_git/
    Complete output (22 lines):
    running develop
    /tmp/pip-build-env-807d9_g2/overlay/lib/python3.9/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    WARNING: The user site-packages directory is disabled.
    /tmp/pip-build-env-807d9_g2/overlay/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    error: can't create or remove files in install directory
    
    The following error occurred while trying to add or remove files in the
    installation directory:
    
        [Errno 13] Permission denied: '/usr/lib/python3.9/site-packages'
    
    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /usr/lib/python3.9/site-packages/
    
    This directory does not currently exist.  Please create it and try again, or
    choose a different installation directory (using the -d or --install-dir
    option).
    
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'.../example_package_git/setup.py'"'"'; __file__='"'"'.../example_package_git/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix= Check the logs for full command output.

[bhrutledge]

@erzoe Please don't run the sudo command that was suggested. I think the "permission denied" error is a symptom of your environment not being properly configured. I'm guessing you're trying to install into the system Python environment, which is leading to the error.

In general, it's best to run pip install from within a virtual environment; see https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/ for an overview.

To troubleshoot, can you share the output of the following commands?

which python

which pip

[erzoe]

Thank you for your fast reply. Here is the output of the requested commands:

$ which python
/usr/bin/python

$ which pip
/usr/bin/pip

[bhrutledge]

Yeah, that pip install is trying to install to the "system" Python environment, i.e. the one that's used by your OS, hence the permissions error. Projects should generally be installed in a virtual environment, esp. when you're using editable mode.

[erzoe]

Sorry, but I don't understand why I should install it in a virtual environment. Virtual environments are great for testing that the software works with other versions of python and other packages but this is the version that I want to use always (1) because I want to find the bugs that I am adding asap and (2) because I need the functionality that I am adding asap.

I guess I could install it in a virtual environment and start that virtual environment when I boot up my computer but then what's the point in the virtual environment?

[bhrutledge]

@erzoe I think the main issue with installing packages in a global Python environment is that there could be conflicts between their dependencies. For example, if package-a requires package-c==2.0, and package-b requires package-c==3.0, then installing package-a and package-b into the same Python environment will cause a conflict.

Also, the system Python environment is carefully constructed by your OS distributor (e.g. Debian) to provide the Python packages used by the OS and its packaged tools. So, it's best to heed the "permission denied" errors and leave that alone.

Your idea to create a "personal" virtual environment would address the second issue, but only partially address the first issue, because conflicts could still arise. An excellent tool that solves both issues is pipx, which handles creating isolated virtual environments and putting any command line scripts on your $PATH.

Further reading:

https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/
https://snarky.ca/a-quick-and-dirty-guide-on-how-to-install-packages-for-python/
https://snarky.ca/why-you-should-use-python-m-pip/

[erzoe]

Thank you for the information. Pipx seems to work fine. Thanks to your links I have also found this other interesting article on What to do when you botch a release on PyPI which might come in handy someday (well, hopefully not).

Nevertheless I am still going without a pyproject.toml file to avoid that error message because—as you have pointed out yourself—it tempts users to run pip with sudo which is in my experience more dangerous than installing a package outside of a virtual environment. But I am now showing how to install the program using pipx in my readme and have added a note that installing python packages outside of a virtual environment is discouraged and a link to your comment.

Also, it seems inconsistent if editable mode installs fail where normal installs work just fine.

[henryiii]

Pip is supposed to apply --user automatically if sudo doesn't work - you seem to have a recent pip version, so I'm not sure why it's not doing that - maybe a bug? pyproject.toml is the future of building, and is useful for other things too, so I'd not recommend avoiding it.

[henryiii]

Ahh, it could be because of the editable mode install. Pip might assume any user who knows enough to use an editable install will also know to use --user or a virtualenv? Even if it works, you should never make an editable install into a global site packages.

Technically, system python distributions are not intended to have any pip installs in them - those are for system installs only (apt, apk, yum, etc). Even upgrading pip is unsupported.

[bhrutledge]

@erzoe I agree with @henryiii: you should be using pyproject.toml, and I would avoid using -e unless you're actively developing the project, within a virtual environment.

More useful reading: https://snarky.ca/what-the-heck-is-pyproject-toml/

[erzoe]

@henryiii no, --user does not make a difference. And the first line says:

Defaulting to user installation because normal site-packages is not writeable

so pip recognizes correctly that --user should be used here.
I think the interesting part is this line:

WARNING: The user site-packages directory is disabled.

[erzoe]

It's not that I am opposed to using pyproject.toml, I would like to use that standard. My problem is that misleading error message which pip gives.

I looked at another project where I knew that this error message did not occur and noticed the difference was the existence of the pyproject.toml file. That is why I removed it from my project, it was the easiest way around this error message.

When I started using pip I, I did not know about --user yet and pip did not use it automatically yet. And when it prompted me with an error similar to this one I just tried again with sudo. And by doing that I broke the package manager of my operating system. That is why I don't want anyone to see this error message. So that they don't repeat my silly mistake.

On the other hand I have never experienced any problems with installing packages outside of virtual environments. (Although it's probably fair to mention that I have started using virtual environments after I learned about them and I knew that I would only need the package in a limited context and for a limited time, like for a university project. Maybe that has saved my some trouble.)

If pip gave a concise error message, one or two lines explaining that you should run that command inside of a virtual environment that would be ok. But with the error as it is I want to make sure that none of my users is accidentally running into it.

[erzoe]

It seems there is the possibility to avoid that error message while still having a pyproject.toml by adding the following lines in setup.py before calling setuptools.setup:

import site
import os

site.ENABLE_USER_SITE = os.geteuid() != 0

I have found that variable in this issue.

[henryiii]

none of my users

Are your users downloading your git source, then running pip install -e .? If so, then whatever docs directed them to do that should also mention what to do if they run into this sort of problem, and warn against sudo. If you are worried about "normal" users that do pip install package from pypi, that can't be editable.

If you want to suggest a better error message for pip, that's probably something you could request (check first to make sure no one already made the request).

tried again with sudo. And by doing that I broke the package manager

That's the point of sudo. It means you know what you are doing and take responsibility for breaking stuff.

The user site-packages directory is disabled.

Yep, that's why. You should never install to the system site packages, so if user site packages are disabled, you have a completely locked down system, no installs allowed.

System Pythons were not intended for users to mess with. They were intended to run system scripts and support system packages. Feel free to ask the packagers, they will tell you the one and only way to distribute Python packages is to make apt/yum/etc. packages, and pip should not be used. 🤦 That's why virtual environments are useful - they give you something you can edit, with an easy revert (delete the venv) and they manage to share the binary part of Python (so they are easy to setup).

I'd think your OS disabling user site packages is a mistake, you could look into why that was done. It's obviously pushing people toward sudo installs, which is much worse.

[henryiii]

seems there is the possibility to avoid that error message

I'd guess this would install to the user location, but since it's globally disabled, no one would be able to use it after it's installed?

[merwok]

Debian does not disable user site-packages, I am sure there is something more to understand why this happens.

[erzoe]

Are your users downloading your git source, then running pip install -e .?

Most users will just run pip install (or pipx install if they read my readme) but if somebody wants to contribute then that would be the way to go.

If you want to suggest a better error message for pip, that's probably something you could request (check first to make sure no one already made the request).

So this is the wrong place to do that? Then I'll take a look at the pip repository.

but since it's globally disabled, no one would be able to use it after it's installed?

no, running the program is possible.

[henryiii]

I assume you can run things in .local/bin, because it's in your path, but probably not import the package from a vanilla Python interpreter.

if somebody wants to contribute then that would be the way to go.

Then I'd include this in the contributor documentation, and I think that's fine. If someone knows enough to start running pip install -e . on their own, and doesn't bother to look at the documentation, then there's not much you need to do. A slightly nicer message from pip might help.

[layday]

Your contributors should create a virtual environment. The documentation should recommend that they do so.

That setuptools attempts a root install after a user install fails is definitely not something that should happen. I'd have suggested to report it to setuptools but everything installation-related in setuptools is deprecated and the only time you'd be exposed to setuptools performing an install is through -e.

[erzoe]

@henryiii

$ python3 -m example_package

and

$ python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import example_package

work, too

[henryiii]

Another idea is to use nox (or tox) for contributors; you can control the experience, and users get a virtual environment created for them. See https://scikit-hep.org/developer/tasks for some examples. You pay a little setup cost but get free environment management and reproducibility.

That's weird, so user site packages are supposed to be enabled in Debian, and seem to be enabled in Debian, but are being reported as not enabled.

[henryiii]

Have you considered trying PyPA/Flit? :)

[erzoe]

@layday yes, I am showing in my readme how to install with pipx and am saying that "It is discouraged to install python packages outside of a virtual environment."

I'd have suggested to report it to setuptools but everything installation-related in setuptools is deprecated and the only time you'd be exposed to setuptools performing an install is through -e.

Thank you very much for your comment. That explains the inconsistent behavior. If it's deprecated anyway I guess it does not make much sense to open a new issue.

[erzoe]

Another idea is to use nox (or tox) for contributors
Have you considered trying PyPA/Flit? :)

Huh, a lot of stuff to look into. This is my first time publishing something on PyPI so there are a lot of new things to learn. Haven't done much else this week ^^"

No, thus far I have only ever used pip, switched to pipx today, but I'll take a look at them eventually.
Thanks for the recommendations.

[henryiii]

There's a guide for flit at https://scikit-hep.org/developer/pep621. I've tried to get something like this into packaging.python.org in pypa/packaging.python.org#1031 - but a few people don't want that in until setuptools supports PEP 621 configuration, and they don't want to push Flit, even if it avoids these sorts of problems for new users. :'(

[merwok]

No, thus far I have only ever used pip,

pip is an installer, but the suggestion to use flit is about changing your build tool (from setuptools) 🙂

[erzoe]

ok, so pipx works fine for running the code but breaks the tests because obviously the imports don't work anymore.
So I had to introduce tox, too, which automatically creates the required virtual environment(s) and installs all required packages inside of them. Luckily setting it up with tox-quickstart was pretty easy (after I figured out that unittest is part of the standard library and therefore specifying it as a dependency causes tox to crash, stupid me).

With this simple tox.ini file

[tox]
envlist = py3

[testenv]
commands =
    python -m unittest discover -s tests

I can run my tests simply by calling tox where I was using a tests/*.py before.

I think I've finally got everything running now. Thank you for your help!

[henryiii]

I'd highly recommend pytest over unittest - unittest mostly exists for Python's own test suite, but pretty much everyone else likely should use pytest. You don't even have to change your tests at first, you can just run unittest with pytest and get nicer features and reporting. See https://henryiii.github.io/level-up-your-python/notebooks/3.1%20PyTest.html :)

Yes, Nox or Tox are ideal for this sort of testing and tasks, though. :)

[erzoe]

thanks for the tip, that looks good

[kynan]

FYI, the relevant bug tracking the user site editable installation error is pypa/setuptools#3019 and the fix (pypa/setuptools#3151) is released in setuptools 62.0.0