pyproject.toml option equivalent to "--no-root"

[techdragon]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Can we add an option to the pyproject.toml or poetry.toml that defaults poetry to using the install --no-root behaviour by default, the fact I'm constantly having to run poetry install --no-root almost every time (I honestly cant recall a time I haven't used this... so for me its close to always) rather than poetry install makes this feel like something I should be able to configure.

[finswimmer]

Just out of interest: What's your use case for the no-root option? Because I couldn't imagine one for me until now.

[techdragon]

tox and I just generally prefer to avoid adding the application itself to the development virtualenv. It broke things for me more often than not when I first started using poetry and I've stubbornly avoided it ever since.

[finswimmer]

Hello @techdragon,

could you provide a small example, that shows the problems you have?

fin swimmer

[techdragon]

@finswimmer I took a few days to ponder, and I honestly cant remember the exact problems I was having which isn't a great starting point for producing some examples of them. I could try swapping a random sample of my projects to the normal default flow of not using "--no-root", but I really don't want to break anything, and the alternative of making new some random new projects and trying to find problems, is awfully close to trying to prove a negative.

[anaulin]

I would love to have this feature as well. The types of projects I work on typically have no use for installing the project itself in my environment, and I'd love to be able to skip this useless (for me) part of the installation process, without having to type --no-root every time.

(It also, to me, feels a little odd to have this self-installation behavior be the default behavior, even if in my case it is just harmless overhead.)

[egnor]

I think the root install is an issue mostly because it's a surprising special case and thus impacts usability.

I think for many people the local-package install breaks people's mental models of how virtualenv "normally" works. Classically, the working directory is code under development, and the environment (system install and/or virtualenv) contains imported copies of dependency code not under development. The surprise is compounded by the use of "editable"/"develop" mode, which is a somewhat oddball installation mode -- and not particularly indicated unless you read the documentation very closely. The "poetry install" command is mostly pitched as "install my dependencies", not "install my dependencies and also 'build' myself".

To be clear, this isn't super terrible. The virtualenv is project-local, so you're not polluting a shared system image with development code. Editable/develop mode mostly works to give you a Pythonically fast edit/run cycle. But again it is surprising, and the surprise can lead to things being broken when people don't understand the magic that has taken place under the hood.

Specifically, if the project doesn't follow standard layout (e.g. "this is just a couple quick command line scripts, I don't want to make a whole module directory") or otherwise has some unusual install-time actions, then the root install can break stuff -- I've run into this myself. This can be hard to debug.

Furthermore, the fact that "poetry install" (re-)installs the local package every time it's run, even if literally nothing has changed, feels weird, because the rest of "poetry install" is very nice about doing nothing when nothing needs doing.

Finally, last and least, you get these weird .egg-info files around, which are annoying litter.

So, in summary -- the biggest problem with root install (and thus, the biggest reason people want to opt out of it with a project configuration flag) is one of user surprise. Perhaps simply doing a better job of describing the intended development cycle (both in the docs and in runtime help and output) would help the problem. As it stands, when introducing poetry to people, I always have to give two big "gotcha" warnings -- one is the silly python2/python3 installer thing, and this is the other one. If I don't, they almost always trip over it at some point.

[finswimmer]

Hello,

it seems we have very different expectations about that :) For me, I still have not found any use-case, where I need to use --no-root.

Let's assume a simple project like this:

mypackage
├── bin
│   ├── __init__.py
│   └── cli.py
├── lib
│   ├── __init__.py
│   └── somelib.py
└── __init__.py

How do you import somelib into cli.py, which will also be your running script, when the package itself is not installed? Yes, you can do this by adding a path to PYTHONPATH or sys.path. But being able to avoid this, is one of the reasons why packages exists.

fin swimmer

[egnor]

In that particular case (and this may be more common than you'd think), I'd just put cli.py and sharedcode.py in the toplevel, not even in a package. You can run "./cli.py", which can "import sharedcode" and everything is very simple and doesn't require any poking into subdirectories.

Nobody publishes anything like this into pypi, of course, but for quick Python scripts it's fairly common.

[finswimmer]

You mean like this?

myscript
├── cli.py
├── pyproject.toml
└── sharedcode.py

Then just do so. As poetry cannot find a root package it will not install any :)

fin swimmer

[anaulin]

The bigger point that some of us have tried to make in this thread is that the behavior of installing a root package by default is surprising and unusual in the context many Python projects and practitioners, and it would be good to have a way to disable that behavior at the project level.

@finswimmer, what I'm hearing from your comments is that Poetry wants to be an opinionated tool in this respect, and not necessarily a tool that's geared towards a broad audience of Python practitioners. If that's the case, it would be helpful to be explicit about this in the documentation, to avoid further discussions like this one, which seems to have devolved into a "well, it works for me" in a way that does not seem productive. I would also suggest closing off this issue with that clarification, to avoid any other hopeful souls like myself getting their hopes up and wasting their time upvoting the issue.

[techdragon]

I opened this knowing it should be fine for most people, otherwise someone would probably have opened it before me. I'm simply seeking a way to turn this off when necessary.

I'm not suggesting we make something users have to think about or suggesting we add new capabilities to poetry...

• The defaults as they are are fine
• This capability already exists and is exposed via a command line flag

This issue is about adding a matching option into pyproject.toml equivalent to using --no-root on the command line, so that anyone using that command line flag can stop typing it on the command line over and over again.

[finswimmer]

Hello @anaulin,

@finswimmer, what I'm hearing from your comments is that Poetry wants to be an opinionated tool in this respect, and not necessarily a tool that's geared towards a broad audience of Python practitioners.

of course poetry is opinionated, as any other tool. As stated in the docs, its focus is to be " a tool for dependency management and packaging in Python." Saying this, it's of course possible to use it not only for this and as the project moves forward other aspect might be included.

I would also suggest closing off this issue with that clarification, to avoid any other hopeful souls like myself getting their hopes up and wasting their time upvoting the issue.

I don't think we should close this now. We appreciate user input about how they use poetry and what they are missing. But we don't want to blow up poetry with any feature that is requested. So it's necessary do understand why someone likes to have a feature. Maybe this can be solved already by another, more elegant way, maybe this is something this is out of scope for poetry, maybe we think "good idea, let's do it".

fin swimmer

[egnor]

@finswimmer , with all due respect, I note that you are focusing on use cases and "what is possible" and "what is not possible". I would like to gently encourage you to also consider the feedback about "what is confusing" and "what is surprising".

Note that confusion and surprise can be addressed in various ways, not just by adding or adjusting features. The way the tool communicates, the way configuration is done, and the way the documentation is written can all contribute to reducing confusion and surprise.

Finally, it seems strange that there are "command line only" flags and "configuration file only" options. Is there a reason poetry doesn't have a unified configuration system that would have avoided this discussion entirely (allowing any flag default to be changed in the configuration file, and allowing any configuration option to be changed on the command line)?

[techdragon]

Finally, it seems strange that there are "command line only" flags and "configuration file only" options. Is there a reason poetry doesn't have a unified configuration system that would have avoided this discussion entirely (allowing any flag default to be changed in the configuration file, and allowing any configuration option to be changed on the command line)?

This is the only thing I’m keen to focus on. I feel like all the other things are distracting from the simple request I opened to add a config file option to reflect this one command line flag so it was “unified” and available in both places.

Simple as that. I’d have written the PR myself if I wasn’t busy.

[nextmat]

I've run into a lot of friction with this as well. To @anaulin's point, this behavior tends to confuse other folks when I introduce poetry into a project and I feel like it is a hindrance for broader adoption.

A per-project config option to set this makes a lot of sense to me since you generally either want the behavior or don't and it is unlikely you will be switching back and forth.

Is there active resistance to a PR for this or is it more a matter of someone taking the initiative to start drafting a PR?

[mikenerone]

There is an extremely common misunderstanding of virtualenvs that I think is what has lead to this request. A virtualenv is, by design, independent of (almost) everything outside of it by default. You should have no dependence on your modules being importable simply because they happen to be in your current directory. What could be more environment-sensitive than that? :P This has harmful effects such as false successes where your app seems to work during dev because you have a package in your current dir, but fails when you deploy it because you failed to ensure that that package is included in your build artifact.

Consider that virtualenvs don't need an associated "project directory" at all. E.g. when you use pipx to install a tool, it creates a private virtualenv deep in the bowels of your home directory and installs that tool (and its dependencies) therein. As per usual, nothing in that virtualenv relies on your current directory. That's what being a "virtual environment" means.

You need to think of a virtualenv as being isolated and independent. You should have a true install (even if it's an editable one) of your project so that you can be (more) confident that it's working the way it will in the real world. I strongly suggest two measures to help you adjust your mental model:

1. Always keep your virtualenvs and your project directory completely separate from each other. This is what Poetry does by default (and I suggest leaving virtualenvs.in-project = false for this reason) - it has a completely separate subdir in your home directory to hold virtualenvs. Outside the Poetry ecosystem, this is exactly the same thing that pyenv does (if you're not aware, it's a very common tool for managing parallel installation of multiple Python versions as well as virtualenvs). In fact, I recommend using pyenv for ad-hoc virtualenvs in general.
2. Use a src layout for your project directory (where your source files are in src subdirectory) - and don't make a habit of changing into it. This way your project modules are not in your current directory, and if they're importable, you know it's because your project installs correctly. Side note: poetry new --src <pkgname> creates a src-style layout for you.

Poetry, pyenv, and other virtualenv management tools are typically designed around this best practice. This is why installing the project is the default behavior, and it's how it can be the case that while some can't imagine why it's the default, others who have a strong understanding of virtualenvs scratch their heads as to why you could possibly want it any other way. IMO it's a bad idea to add a configuration explicitly for the purpose of easing a bad practice.

[egnor]

With all due respect, @mikenerone, I disagree a bit with your analysis. (I would like to gently point out that your tone lands as a bit condescending, which I presume you don't intend; you strongly imply that a lack of technical proficiency is the only reason for disagreement on this issue. Please consider alternative ways to highlight assumption mismatches; thank you!)

First, you seem to be asserting that always-run-from-venv is the One True Development Workflow (if using a venv at all), or at least the Definitely Best Development Workflow. It's certainly not a bad workflow, your advice is a quite reasonable default for anyone developing a Python package that will be shared via package distribution systems (pypi, etc), and "test like you fly" is generally good practice for sure. However, not everyone is developing a Python package that will be shared (some of us mainly use Poetry to consistently manage dependencies, not to build a package), and even for those that do, there are plenty of other workflows that can also work and verify installability.

Second, even if that workflow is generally good practice, there is still a surprise factor being discussed, which is somewhat orthogonal. For example, the description of "poetry install" says: "The install command reads the pyproject.toml file from the current project, resolves the dependencies, and installs them." This implies that it installs dependencies and says nothing about installing the current package. The fact that it also does that is surprising and disconcerting and breaks people's mental model that Poetry is managing dependencies, since it's also stuffing the contents of the current directory into the venv.

In any case, you may be quite right that differing assumptions from different user communities are what make this thread weirdly contentious. (One wouldn't normally expect that a request for a command line option also be settable from a configuration file would be in any way controversial!!) If that is accurate, then that observation may show a path to resolution, namely (1) decide whether it is in fact desirable for Poetry to strongly center on this One True Way, (2) if so, make sure this is made more clear in Poetry's documentation and operational communication -- for example:

• the description of what "poetry install" does (currently just talks about dependencies)
• the discussion of supported directory layouts (currently quite neutral)
• the default behavior of "poetry new" (currently --src is not the default)
• general introductory discussion of Poetry's role in development workflow (currently doesn't mention this at all)

As a final remark, it's probably best if we please try to take on board that the people asking for this are highlighting a spot of confusion, surprise and dismay -- I'd prefer if we figure out where that comes from and how to resolve it (which may not be by implementing the feature request as originally proposed), rather than assert that it is unnecessary or ill informed? Many of us are willing to offer PRs, but of course don't want to do something that runs against the grain of the design intent.

[mikenerone]

Indeed, no disparagement is intended. If I feel it's a misunderstanding, I also acknowledge that it's a super common one (almost always one of the first corrections with new hires), and also that I had the same misunderstanding myself at one time. If there's a fault, it falls with documentation regarding virtualenvs, not the developer community at large. In any case, though, it's certainly not a "special case" as you said.

I do agree that something should be done about the confusion. A bit of explanation about it in the Poetry documentation to provide some guidance might go a long way toward addressing the issue.

[anaulin]

It'd be nice to pause the mutual 'splaining of the Real Good Way to Do Python ™️ , and instead refocus on @nextmat 's very practical, and reasonable, question:

Is there active resistance to a PR for this or is it more a matter of someone taking the initiative to start drafting a PR?

[mikenerone]

@anaulin Hmm, a single comment that attempted to explain the reason behind that resistance, directly addressing that question, followed by an exchange about what else to do about it in light of that resistance? That's what you want paused?

[egnor]

(I also think answering @nextmat 's question would be nice! And then we can unpause the discussion of Larger Changes. "Pause" doesn't have to mean "stop" after all.)

[mikenerone]

Well I stated very clearly that for me the answer is yes, there's active resistance (not that I get to make the decision or anything). That's why I don't really get why @anaulin's criticism about not answering that referenced my comment. If you're upset that it's been too long since a response from Poetry devs, don't point at me about it.

[egnor]

(OK, who's the decider? I'm actually unfamiliar with the governance of Poetry. Would it be @finswimmer ?)

[mikenerone]

@finswimmer, @sdispater, and @abn

[anaulin]

@mikenerone seems you felt personally called out. Sorry you feel hurt, that wasn't my intention. I was referring to the several comments on this thread that have happened both before and after you showed up, both by you and others, about what constitutes or does not constitute the "standard" or "right" or "better" way to do Python. I'm sure you are aware that you are not the only participant on this thread.

I did not read your comment as a response to the direct question, but rather as a detailed explanation of why some folks don't like the idea of allowing folks to set an existing Poetry option via pyproject.toml in addition to command-line. I don't even know if you are or are not involved with Poetry directly, @mikenerone. 🤷‍♀️

[mikenerone]

np, clean slate then. But let's be open to points of debate. It's async communication, and we have to live with the concurrency. :)

[abn]

First off, the team appreciates the community's interest in the project, and we are really happy that the project is useful in varied scenarios. Some scenarios are more surprising than others. :)

As for the original request, am I understanding this correct that the intent is to be able to configure per command cli option in a project's pyproject.toml? Or, is the intent to have a per-user per-project configuration of default values/options, ie. poetry config --local command.install.no-root true?

If it is the former, I do not think that this will be something Poetry will implement. This is not something that, imho, belong in a pyproject.toml file. While a project maintainer would like to manage their environment a certain way, contributors may prefer another. There is a distinction between a "project" option and a "local tool" option I feel. What is requested here, to me at least, feels is more a "local tool" option.

Obviously there are many ways to achieve the same results.

From what I can gather there are two separate use cases being discussed here that would (?) benefit from being able to not install the root project.

1. Projects that rely on Python's PYTHONPATH behaviour.
2. Projects that simply use Poetry for either dependency management or virtual environment management.

Am I missing something here?

Note: For topics other than this, please create isolated issues.

[sinoroc]

If I may... I feel like what's missing from this conversation is concrete examples of how poetry install (without --no-root) breaks (1.) people's expectations and/or (2.) their development environments.

1. It is a bit confusing to read that to some users the fact that poetry install does indeed install the project is surprising behavior. It might be helpful to expand on that. Why is it surprising? What do people's workflows usually look like? Surprising compared to what, exactly? Are you more used to pipenv for example, or other tools?

Indeed the first sentence of poetrys' documentation on the install command is a bit misleading, and could be improved. But later on it definitely says: "By default poetry will install your project's package everytime you run install". Which is better. (Although from my point of view, the "as editable" bit is missing.)

2. On the other hand, it seems like for some projects, installing it in the development environment (even as editable) is problematic or even a no go. Why is that? What actually breaks or prevents people to work on their project?

• Is it an issue with the installation? Maybe the project is not meant to be installed at all? How do you run it in production then? Is the editable aspect of the installation problematic?
• Is it an issue with the build? Maybe it takes too long or it has side effects that we don't want in a development environment? Are you maybe working with Cython? Or something that doesn't integrate well with poetry's build backend?
• Is it an issue with the packaging? Maybe the project doesn't need a name, a version, etc. Maybe it doesn't really have actual modules and packages and so on.
• Is it an issue with the imports? Some confusion with what the top level packages should be? Does the project rely on sys.path or PYTHONPATH modifications?

Maybe in some of the cases presented in this thread, there is a concrete fix that could be applied before reaching for --no-root.

---

Or, is the intent to have a per-user per-project configuration of default values/options, ie. poetry config --local command.install.no-root true?

This suggestion looks like a good compromise to me. That would mean that the setting would land in a poetry.toml file right next to pyproject.toml, is that right?

[nextmat]

Where I run into this are projects where I'm using poetry for dependency management and virtual environment management but don't want to explicitly package anything. This is a pretty common scenario in application development.

As a point of clarification, the friction for me is really in two places right now:

1. I forget to use --no-root constantly because I'm not running install every day and I also use tools for similar purposes in other languages where I generally am not using flags routinely. This is not a big deal, easy to clean up and just kind of an annoyance.
2. Project collaborators who use python less frequently run into this issue, get confused and I have to explain to them why this happens and why we can't currently fix it. Since I appreciate many other aspects of poetry and am likely the one introducing it, this can get a bit old.

Either a poetry config option or a pyproject.toml option would be fine with me. If something will be added to the filesystem my preference would be a modification to pyproject.toml since part of the promise of the file is the reduction of the ever-growing expansion of tool-specific config files.

Thanks again for Poetry. In general I find it dramatically better than the alternatives and look forward to resolving this in one way or another so we can just go on with using it. 😄

[sinoroc]

@nextmat Thanks. What happens when you forget to use --no-root?

[nextmat]

@sinoroc I just get a <my-project>.egg-info directory that I don't need and I remove it.

[techdragon]

I'd argue this option actually belongs in both poetry.toml and pyproject.toml since a developer wanting to control their personal environment should use poetry.toml as this is their local config for their environments, however if a project were to for some reason require this option, then it makes sense for them to have this in the pyproject.toml that will be checked into git and shared with other developers.

One reason why a project might require this option is the popular testing tool Tox. Currently tox and poetry don't play very well together, the current best practice recommends using --no-root in the tox configuration files. I wont reiterate the entire history of these other issues, but #1941 does a good job of summarising.

Another is Docker builds. #2760 and #1879 cover various issues in Docker environments where --no-root has come up as either the solution or a potential solution.

This is doubly problematic in the case of a web based project deployed using docker and tested with tox, where there is potentially (I'd argue this is quite likely) no scenario where the developers actually want to "install" the code they are writing and then the test setup will cause problems, in this situation there is no reason to ever install "the package" at all and forcing this as default behaviour only causes problems by constantly requiring users of poetry to use the --no-root option to avoid problems. I don't come across many Flask or Django tutorials that even suggest a user define their code as an installable package at all so many web based projects are not written with installation as a package in mind. It was a Django package where I was testing using Tox that I had the most issues with poetry. Compounded with issues related to namespaced packages and having parts of the namespaced package in different pythonpath locations I got very frustrated with having to constantly use --no-root and opened this issue.

There are definitely legitimate uses for having a config file option for always doing the same thing as --no-root and it likely belongs in pyproject.toml but once theres an option for this I don't see any reason why it shouldn't also be a personal preference option on poetry.toml, but I'm less bothered by that not being available than I am at the lack of a project specific option for when this is necessary.

[sinoroc]

As far as I know (from a previous discussion with @abn), Poetry's maintainers have on their todo list to revamp parts of the code base responsible for the handling of the configuration (which I believe includes the poetry.toml file, poetry config, the CLI flags, the environment variables, etc.). I have no idea how high or low the priority is for this task.

Most likely this feature request (--no-root in a configuration file) we are discussing here, would be addressed during or after this rewrite of the config handling.

---

If some of you feel like pitching in to get this feature released sooner, I guess looking into the whole config subsystem is not a bad idea. Get in touch with the maintainers and ask what their vision is for the rewrite then try to get things moving forward.

[nextmat]

Thanks for the context, @sinoroc.

In the interim, here is a short shell function that solves this issue, in case others find it useful:

poet() {
  if [ "$1" = "install" ]; then
    poetry "$@" --no-root
  else
    poetry "$@"
  fi
}

[kikyous]

please make --no-root default

[johnthagen]

This would be very useful for a variety of situations where Python applications are designed around the convention of running from a directory rather than being packaged into a wheel at all.

For example, the Django startproject command generates a structure that looks like:

mysite/
    manage.py
    mysite/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

And is run like:

python manage.py runserver

Django applications are typically deployed in this way rather than building a wheel.

FastAPI conventions are similar:

main.py

uvicorn main:app --reload

In both cases, it's normal to run uvicorn, Gunicorn, uWSGI etc from the command line and point them at your application source directory than it is to install an entrypoint through a wheel.

I think that the tension/disagreement is one between libraries and (non-wheel distributed) applications. I think that it's fine that Poetry defaults to targeting wheel-packaged libraries rather than applications, but I also think there is a large user base of source-directory based applications for which being able to explicitly set no_root = true or similar would both remove the need to type something on the CLI often and also be more declarative.

[johnthagen]

While not exactly the same, Rust's cargo new command has both a --bin and --lib flag to control whether the created project is treated as a binary (application) or library (package).

I think having a similar distinction for Poetry would make it more natural to use for non-wheel distributed applications.

[johnthagen]

Another use case is for something like a standalone MkDocs project. You may want to use Poetry to lock the dependencies used to build the docs, but there aren't actually any Python files in your project (just Markdown files) so if you run poetry install (forgetting --no-root) you get an error that looks like:

  ValueError
  docs is not a package.
  at /usr/local/lib/python3.9/site-packages/poetry/core/masonry/utils/package_include.py:80 in check_elements
      76│                 self._package = root.name
      77│                 self._elements = sorted(list(root.glob("**/*")))  # type: List[Path]
      78│ 
      79│                 if not self.is_stub_only() and not self.has_modules():
    → 80│                     raise ValueError("{} is not a package.".format(root.name))
      81│ 
      82│                 self._is_package = True
      83│             else:
      84│                 self._package = root.stem

In these cases, having a way to annotate that this project has no root and allow the simple poetry install to work, would be very useful.

[neersighted]

Duplicate #1132.

@johnthagen's comments pretty much sum up the state of the art in both the Python and broader packaging ecosystems -- a non-package/Pipenv-like mode is the real thing being asked for here.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.