Poetry with virtualenvs.create false installs packages into the wrong place on Ubuntu

[amcinnes]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 20.04
• Poetry version: 1.2.0
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/amcinnes/8c61ad88b37a7740dd93efa976f85ed7

Issue

To reproduce the issue:

docker run --rm -it ubuntu:20.04@sha256:af5efa9c28de78b754777af9b4d850112cad01899a5d37d2617bb94dc63a49aa
apt-get update && apt-get -y install python3-pip
python3 -m pip install poetry==1.2.0
poetry config virtualenvs.create false
poetry init -n
poetry add awscli
aws --version

Expected behaviour (seen with poetry 1.1.14): the aws command runs successfully.

# aws --version
aws-cli/1.25.70 Python/3.8.10 Linux/5.19.7-arch1-1 botocore/1.27.69

Observed behaviour (with poetry 1.2.0): the aws command fails.

# aws --version
Traceback (most recent call last):
  File "/usr/bin/aws", line 19, in <module>
    import awscli.clidriver
ModuleNotFoundError: No module named 'awscli'

This seems to be because with poetry 1.2.0, awscli has been installed into /usr/lib/python3.8/site-packages/awscli which is not in the Python path.

[neersighted]

root@7f452a8de1d3:/# python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
>>> import sys
>>> print(sys.path)
['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/root/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
>>> import sysconfig
>>> print(sysconfig.get_paths())
{'stdlib': '/usr/lib/python3.8', 'platstdlib': '/usr/lib/python3.8', 'purelib': '/usr/lib/python3.8/site-packages', 'platlib': '/usr/lib/python3.8/site-packages', 'include': '/usr/include/python3.8', 'platinclude': '/usr/include/python3.8', 'scripts': '/usr/bin', 'data': '/usr'}

Looks like this is the old Debian patching of distutils without patching sysconfig acting up again, combined with our apparent failure to make use of our modified detection code:

poetry/src/poetry/utils/env.py

Lines 193 to 222 in c4b2253

	GET_PATHS_FOR_GENERIC_ENVS = """\
	# We can't use sysconfig.get_paths() because
	# on some distributions it does not return the proper paths
	# (those used by pip for instance). We go through distutils
	# to get the proper ones.
	import json
	import site
	import sysconfig
	from distutils.command.install import SCHEME_KEYS
	from distutils.core import Distribution
	d = Distribution()
	d.parse_config_files()
	obj = d.get_command_obj("install", create=True)
	obj.finalize_options()
	paths = sysconfig.get_paths().copy()
	for key in SCHEME_KEYS:
	if key == "headers":
	# headers is not a path returned by sysconfig.get_paths()
	continue
	paths[key] = getattr(obj, f"install_{key}")
	if site.check_enableusersite() and hasattr(obj, "install_usersite"):
	paths["usersite"] = getattr(obj, "install_usersite")
	paths["userbase"] = getattr(obj, "install_userbase")
	print(json.dumps(paths))

If I had to guess, it's because we always treat the target environment as a VirtualEnv instead of a GenericEnv and thus don't invoke that script.

Honestly, this is such an edge case (and mostly the result of bad Debian packaging) that I'm not sure there's much value in fixing it -- it's easily resolved if you make use of a self-compiled Python or virtual environments (there is no reason to avoid them in a container, and doing so exposes you to all sorts of sharp edges like this).

This feels like another instance of #6398 -- I would highly suggest looking at the pattern here #6397 (comment) for something that is as ergonomic, but unlikely to be broken by distro packaging decisions.

[neersighted]

Some more reading on this issue:

mesonbuild/meson#8739
https://github.com/gramineproject/graphene/blob/6ba10c60b94a066d516bb9121c31280c0ba69468/Scripts/get-python-platlib.py
https://sources.debian.org/src/python3.7/3.7.3-2+deb10u3/debian/patches/distutils-install-layout.diff/

I also suppose I'll ping @abn as historically he's the one interested in trying to paper over these bad decisions.

[tigris]

This feels like another instance of #6398

Thanks @neersighted - this is indeed an instance of that. The context here from our perspective is that the OS package for awscli is quite old and lagging, which admittedly is not your concern at all.

But, that led to us wanting to manage certain OS level dependencies via python packaging rather than OS level packaging. At the time that seemed like a reasonable thing to do.

We were originally just doing something simple like python3 -m pip install awscli==1.2.3 or whatever. This came with it's own set of problems. So then we wanted to use better python dependency management tools so that all dependencies were locked/manageable rather than just the top level one, which had an added benefit of people able to use other dependency management tools such as renovate.

I hope that adds some context around situations where managing OS level dependencies via a python dependency management tool might be desirable. Perhaps the path forward for us is to manipulate the default $PATH rather than try and shoehorn global executables into /usr/loca/bin/?

[dimbleby]

Perhaps the path forward for us is to manipulate the default $PATH

a sensible pattern is, pipx-style, to install awscli or whatever into a virtual environment but then put a symlink in your path pointing at the executable in that environment.

[neersighted]

I'd definitely say that this is one of the rare cases where 'I know what I'm doing and the global environment is the best solution' is the case... Certainly, MRs to fix it are welcome since this is functionality we should have -- it's just functionality that only comes up during installs to the global environment on 4+ year old Debian releases 😆

If anyone would like advice on working on this, please feel free to reach out -- otherwise I'll try to look at it, but the timeline on that may be a while.

That being said, this is a bit of a square peg round hole situation, and ultimately pipx or something else probably is a more natural fit.

[Mattwmaster58]

I'm running into this with a simpler setup again with virtualenvs = false. Dependencies aren't being installed into the right location, so the system python can't find them.

root@3d366f729e32:/app# python -m site
sys.path = [
    '/app',
    '/usr/lib/python38.zip',
    '/usr/lib/python3.8',
    '/usr/lib/python3.8/lib-dynload',
    '/root/.local/lib/python3.8/site-packages',
    '/usr/local/lib/python3.8/dist-packages',
    '/usr/lib/python3/dist-packages',
]
USER_BASE: '/root/.local' (exists)
USER_SITE: '/root/.local/lib/python3.8/site-packages' (exists)
ENABLE_USER_SITE: True
# (after a poetry install including flask)
root@851269ec027a:/app# find / | grep flask
/usr/lib/python3.8/site-packages/flask
...

I guess for the time being we will start using venvs

[neersighted]

@Mattwmaster58 Please report what OS version and Python package version you are encountering this with.

[manojatlas]

Facing this with python 3.8 on Ubuntu 20.04. on docker image mcr.microsoft.com/playwright/python:v1.24.0-focal

I had to replace to pip install poetry==1.1.14 to get this working expected. but with current poetry version the issue persist.

[Mattwmaster58]

@neersighted @manojatlas is my exact case where I'm seeing the problem. I can share a minimum docker file if you'd like

[neersighted]

@neersighted @manojatlas is my exact case where I'm seeing the problem. I can share a minimum docker file if you'd like

Does that mean you're on the same image (or at least OS version)?

[Mattwmaster58]

Same image yes

[neersighted]

Okay, thanks for the info -- in the mean time, is there a reason that you cannot use virtual environments (like @amcinnes)? There are myriad reasons to prefer them, from being insulated from the packaging choices of the distro, to avoiding mixing unexpected Python code into your environment, to being able to copy them across multi-stage container builds.

[Mattwmaster58]

No, not any substantial reason. We ended up that it to get around this bug.

[manojatlas]

The reason I wanted to disable virtual environment was that inside the docker image Playwright is install in system environment and I use poetry then the dependencies will be installed in the virtual environment. so it wont work as cant find packages.

So for this I had to disable the virtual environment and install all dependencies in the system.

@Mattwmaster58 : what alternative way you've used? also if you can share your minimum docker file would be great.

[Mattwmaster58]

@manojatlas nothing special really, I can share if you want.

Playwright images don't have client libraries preinstalled, only necessary libs and binaries to run the browsers it supports. Installing in a venv does not prevent you from using those preinstalled browsers.

[manojatlas]

Maybe out of topic question: ok in that case I have to activate virtual environment when I want to execute tests?

[Mattwmaster58]

Yes, or use poetry run

…

On Mon, Sep 12, 2022, 10:30 PM manojatlas ***@***.***> wrote: ok in that case I have to activate virtual environment when I want to execute tests? — Reply to this email directly, view it on GitHub <#6459 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGI56LIXAADXU5BVLHBOPB3V577OBANCNFSM6AAAAAAQILJ47U> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mstandley-tempus]

I was trying to find a bug report opened against Ubuntu describing this behavior. I think it's this one, but I'd appreciate it if someone else can confirm: https://bugs.launchpad.net/ubuntu/+source/python3-defaults/+bug/1408092

The biggest reason I'm doubting myself is that this one has been open since 2015, but no apparent progress.

[neersighted]

We do have code to work around this, but it is just not being triggered properly -- so there is a Poetry bug, but Poetry would not have to do anything special if the distro patches were applied properly/consistently (trying to get this information out of distutils is fraught).

[amy-langley]

Honestly, this is such an edge case (and mostly the result of bad Debian packaging) that I'm not sure there's much value in fixing it -- it's easily resolved if you make use of a self-compiled Python or virtual environments (there is no reason to avoid them in a container, and doing so exposes you to all sorts of sharp edges like this).

I see that the conversation has moved on quite a bit since this comment about the bug being an edge case, but I just wanted to push back on it a little bit: not only are Ubuntu and Debian a considerable portion of the total Linux space on their own, but also the official Python docker containers are built from Debian bullseye and buster (unless we want to use alpine). I would think poetry would hope to work out of the box with the official Python images.

Edit: not to say that it isn't annoying that this issue has persisted for so long in those distros, but it really does seem like something poetry needs to be resilient to.

[neersighted]

The official Python images are just fine as they use a Python.org build of Python and not a patched/hacked distro-supplied Python. The edge case is mostly installing to system site-packages using a Debian-derived Python, which is definitely a thing people do, but is often ill-advised and will always be prone to sharp edges due to deviation from the standard Python ecosystem.

[neersighted]

It's not so much that Poetry isn't resilient to it -- it's that Debian makes undocumented and invasive adjustments to code that isn't meant to be exposed/portable to any external tooling like Poetry, and we have to constantly hit a moving target that is not documented anywhere. It's hard to be truly robust and resilient when Debian doesn't make their patches in a way that is stable or consumable.

If the linked launchpad issue is ever fixed this will just work -- but as long as Debian monkeypatches distutils and doesn't properly set sysconfig paths (like specified in the Python Packager's documentation) this will remain a moving target as we have to introspect unexported internals of a deprecated module.

[beeb]

Adding to the argument by saying that I'd very much like to use official tensorflow docker images and they use system python. When adding my custom packages to the default install through poetry and virtualenvs.create=false it results in the bug described here.

Using a virtual environment would defeat the purpose of using an image with everything pre-installed (including jupyter notebook).

[fsonntag]

It's also causing issues with the Tensorflow images.
And of course I don't want a virtual environment there, since my Docker image will blow up with two TF installations. Plus, it is quite hacky to access the python executable from the created virtual environment in a script then.

[georgeseifada]

+1 for this issue. We shouldn't need to use a virtual environment inside a Docker image. I'm facing the issue when using an ubuntu:22.04 base image with poetry==1.3.1. Making the setting poetry config virtualenvs.create false and then installing packages via poetry install results in packages being install in the wrong location. This was not an issue in poetry==1.1.14.

[ncoish]

I'm running into this issue as well on Fedora 36

[amang95]

Any update on the fix for this bug?

[bVdCreations]

Same issue for me

[mcsheehan]

Plus one for this - it's not just aws cli it's a problem for us deploying our docker containers - we want our installed [tool.poetry.scripts] to be runnable at system level without having to prepend with poetry run.

[neersighted]

In a container, you can use ENV VIRTUAL_ENV=/path/to/env PATH=/path/to/env/bin:$PATH to easily activate a virtual environment. Combine that with python -m venv /path/to/env and the fact that Poetry will install into an already-activated environment instead of managing its own, and you have a much better solution.

[fsonntag]

@neersighted Thanks for the feedback :)

Of course there are workarounds, but it was working very seamlessly before. I think also using the term "virtual environment" is kind of misleading when one just wants to use the system Python environment. So the expecation of poetry config virtualenvs.create false would be that it's using the actual system Python environment. Otherwise it might be even better to deprecate this option, so that it's clear.

(Finding out the correct path of the system Python environment is also not super easy 😄)

[neersighted]

virtualenvs.create false does use the system environment -- the issue is more that Debian/Ubuntu partially Python to use opinionated paths and our detection of that broke.

Anyway, you should still at the very least use virtual environments to keep Poetry and your project from mixing in a container, or Poetry will happily uninstall parts of itself when they intersect with your project.

[fsonntag]

our detection of that broke.

Is it hard to fix that? If you can give some pointers, maybe I can try?

[dimbleby]

looks as though debian do patch sysconfig from python 3.10, so there should be light at the end of this messy tunnel. eg on my ubuntu 22.04 vm:

>>> sysconfig.get_paths()
{'stdlib': '/usr/lib/python3.10', 'platstdlib': '/usr/lib/python3.10', 'purelib': '/usr/local/lib/python3.10/dist-packages', 'platlib': '/usr/local/lib/python3.10/dist-packages', 'include': '/usr/include/python3.10', 'platinclude': '/usr/include/python3.10', 'scripts': '/usr/local/bin', 'data': '/usr/local'}

NB the above contains dist-packages, not site-packages.

with python 3.7 near enough eol, that only leaves debian-based distributions of python 3.8 and 3.9 as problematic. That's still a non-zero number of users: but hopefully a dwindling number, for many of whom "use a newer distribution / python" should be a reasonable answer.

[dimbleby]

#7766 implicitly makes this wontfix: if on debian, use a virtual environment or python >= 3.10

[legioz]

FROM python:3.11.3-slim-bullseye
...
RUN pip install -U poetry
RUN poetry config virtualenvs.create false
RUN poetry install --no-root --without dev --no-interaction --no-ansi

The first line gets ignored in Docker...

If I list configs inside the container I get "venv.create = true"...

[dimbleby]

the above is both irrelevant in this issue, and untrue.

FROM python:3.11.3-slim-bullseye
RUN pip install -U poetry
RUN poetry config virtualenvs.create false
CMD ["/usr/local/bin/poetry"]

$ docker run -it --rm foo:latest poetry config --list | grep virtualenvs.create
virtualenvs.create = false

[legioz]

@dimbleby
Running on MacOS M1 with compose to build.

[dimbleby]

Still irrelevant to this issue (you are mixing users)

[a7744hsc]

Here is a tmp solution I used to set poetry env as the default python interpreter in docker.

#base image
FROM some_python_image

RUN pip install poetry

#copy source and install requirement
COPY ../pyproject.toml poetry.lock /app/
WORKDIR /app

# Their is a bug with poetry config virtualenvs.create false https://github.com/python-poetry/poetry/issues/6459
# Below is a workaround to set poetry env as default env
RUN poetry install --no-interaction --no-ansi
RUN POETRY_ENV_PATH=$(poetry env info --path) && ln -s $POETRY_ENV_PATH /mypyenv
ENV PATH="/mypyenv/bin:$PATH"

#start command
CMD ["python", "main.py"]