pip install -e . equivalent?

[kootenpv]

We cannot run pip install -e . because there is no setup.py. What would be the way to achieve this using poetry?

[sdispater]

No, this is not possible right now. And I don't think I will add an equivalent since it always seemed like a hack to me. But I might be wrong.

Why do you need pip install -e .? So I can see if there is a "poetry way" to do what you want.

[kootenpv]

I often depend on some local packages, for which I want to be able to change some code and not have to reinstall.

[sdispater]

I see.

The thing is poetry is a dependency/project management tool and is not intended for such use cases.

However, if you really want to to this, you can build your project poetry build and take the setup.py file from the generated tarball and then you should be able to do pip install -e .

[kootenpv]

Ah sorry, with local I mostly meant "developing local packages". I do believe poetry is also for helping with that? It sounds like barely any work to automate that part and create e.g. the poetry develop command.

[sdispater]

Well, poetry can help you develop local packages by orchestrating your dependencies, sure. But it is not meant to support the use case of having only local packages that depend on each other since this is not the best practice that poetry tries to endorse.

However, if you are willing to make a Pull Request, I would gladly take a look at it.

[winni2k]

I might be off the mark here, but in my experience, I usually use pip install -e . in order to allow my acceptance tests to see the python project I am testing. Is that what we are talking about here?

[sdispater]

I don't see why you would need pip install -e . for your tests to work. If you are inside a virtualenv your dependencies should be picked up by your tests.

In poetry that would be:

poetry install  # install your dependencies
poetry run pytest tests/

[kootenpv]

I'm not sure, but when you share a module in git, with other dev members, it's nice when it is updated automatically on pull. Or is this automatically managed in pycharm or something? I use emacs without virtualenvs but just simply different python versions.

[winni2k]

In Pycharm you set the environment by choosing a python interpreter in your directory tree. Packages are discovered from that AFAIK.

[winni2k]

I guess this is a bit of a style question, but I like to build a wheel for local packages and use pip to install them into my venv. Is that the workflow that is intended by poetry?

[sdispater]

@winni2k Not at the moment, no. For now, poetry assumes each dependency is located in a remote repository (PyPI or a private one) or is a VCS dependency, and does not support local packages.

That might change in the future, however but I can't give you an ETA.

I am trying first to support the most common use cases, like installing from PyPI before diving into more specific workflow, especially if they are not the recommended way of managing packages.

[kootenpv]

It now also does not allow me to use it in the interactive interpreter :/

I tried using poetry build, then copy out the setup.py to the root level but that doesn't work. I'm very disappointed :(

Until this becomes important, I wish there'd be a nice hack. I'm wondering though how do you develop - how do you avoid having to constantly reinstall?

[sdispater]

@kootenpv I just tested and copying the setup.py file works.

I don't know how you develop, but I never need pip install -e .. I either upload my packages to PyPI or a private index, or I install from a git repository.

[woutervh]

I don't know how you develop, but I never need pip install -e .. I either upload my packages to PyPI or a private index, or I install from a git repository

Ugh, that is not developing, that is installing a finished package.

These are roughly equivalent, in your src dir:

pip install -e .
python setup.py develop

It's called "installing a package in development mode". They are the starting point of beginning development. If you are not familiar with that, then I honestly do not understand how you do development.

I know two ways of doing development without it:

export PYTHONPATH=
or in your code:
import sys
sys.path.append(....)

Are you recommending this? (I hope not)

[sdispater]

No, I don't modify sys.path(). And please don't be condescending.

So, if I develop a library (or an application for that matter) I put myself in my project directory. At this point I have access to my module: I can import it in my tests to test it.

You can test this yourself: clone the poetry repository place yourself in it install the dependencies with poetry install and execute poetry run pytest tests/ and everything will work. You do not need pip install -e .

If you need it in another project, and that is why you install it in develop mode, the you have dependencies between project and in this case I recommend to use a private index and release preversions or installing from git repository and updating when needed.

[merwok]

There are many communities of Python users, using different categories of development tools. Depending on practices (e.g. code in src dir vs code at repo root) or tools (e.g. Django runserver vs. Pyramid pserve), you can rely on «current dir is sys.path[0]» or you always install in develop/editable mode.

[merwok]

Another use case is when you’re trying to fix something in a library used by your project. From your project’s virtualenv, you run pip install -e path/to/library so that every change you make in the library repo is directly picked up in your project where you are testing the results.

[sdispater]

@merwok I understand your point, but most of the time you don't need it, in my opinion.

Regarding you last example, this will be supported in the next feature release since it will add the ability to install from a directory.

I know that there is a lot of different workflows out there, but poetry can't support them all. It tries to enforce some of them to help make the dependency management and packaging in Python easier and for that it has to be somewhat opinionated.

I am not saying it will never make it into poetry but since I am the sole developer I have to focus on a workflow that will suit most cases to develop applications and libraries.

[kootenpv]

@sdispater Thanks for your way of developing (private index / from git), I wasn't aware of another way.

I can agree that under some conditions it is more proper and might lead to fewer bugs!

In my case, I do a lot of data science and "live" in Interactive python. It's very convenient when I can just do pip install -e . once and I'm just able to change the code so that in another module it will be picked up and see the results quickly (rather than having to run any install command again). That is the step before writing tests.

I understand you are hesitant from your perspective as it does not seem like a really clean use case, and it looks like the solution would have to be a bit hacky?

Still, I hope you could guide us and be accepting of a solution that might help a few (and not bother others)!

Currently the only way I see is something like:

1. poetry build
2. find the build package (?)
3. extract it
4. move the setup.py a level up
5. find the python version (?) used by poetry and use it like python setup.py develop

Does this sound like the best way?

[ipmb]

I'll chime in and say I also use pip install -e . when doing local development. It is helpful for a couple reasons:

1. It puts my project on the Python path. I don't have to depend on being in a specific directory for things to work which feels hacky to me.
2. It sets up scripts and entry points I wouldn't have otherwise.

[sdispater]

@kootenpv Don't get me wrong, I can understand why you would use pip install -e, especially if you want to debug a dependency.

And, like I said, directory dependencies will be introduced in the next feature release so, with them, you will be able to reproduce this behavior.

[moigagoo]

@sdispater

You can test this yourself: clone the poetry repository place yourself in it install the dependencies with poetry install and execute poetry run pytest tests/ and everything will work. You do not need pip install -e .

Hi! Could you please clarify how you run tests with pytest without installing poetry in the virtualenv? I can't import my package from pytest tests because it's not installed and resides one level higher than the tests.

It seems like poetry has the same source layout, but for some reason the imports work (although the tests fail). Am I missing something?

UPD: Even the pytest docs say that you can only run tests against an installed version of your package under this layout: https://docs.pytest.org/en/latest/goodpractices.html#tests-outside-application-code

[moigagoo]

Sorted this one out. Should've put an empty __init__.py file in tests.

[blueyed]

poetry build and take the setup.py

What do you think about some option to generate the setup.py file directly?

[bersace]

Is it related to #47 ?

If i summarize it, the need is that a poetry managed project should be installable as editable. e.g. i want to debug that library used by my project, i git clone it and poetry install -e the path ?

[bersace]

Or maybe the goal is just to have project console scripts installed in virtualenv ?

[sdispater]

@bersace If you want to test you console scripts, it's possible via the poetry script my-script. For example for poetry that would be poetry script poetry.

Also, I understand the use case of pip install -e path/to/some/dependency/ to allow debugging a dependency since you won't have to reinstall each time you make a modification. That's why it will be possible in the next feature release.

However, I do not think pip install -e . (or python setup.py develop) is that useful and as such introducing an equivalent in poetry is not something I plan on adding at the moment.

Now, that being said, if someone can give me a compelling use case where it's necessary, I will gladly change my view on the subject.

[kootenpv]

I think I get what you mean: you want to enable the option for a package (A) to list a development version of another package (B).

I perhaps see a problem though.

I believe you're saying you could list B as a requirement in A in the new version by listing pip install -e path/to/some/dependency/ in A's poetry file (or some similar syntax, maybe just relative path)?

Isn't the problem then that if both are poetry packages, A will not be able to pip install -e B since poetry does not generate a setup.py for B?

[sdispater]

@kootenpv Exactly.

I think it would take the form:

[tool.poetry.dependencies]
dependency = { "path" = "relative/path/to/folder/" }

And with the equivalent of -e:

[tool.poetry.dependencies]
dependency = { "path" = "relative/path/to/folder/", develop = true }

Regarding the issue when the dependency is also a poetry package: at the moment, there won't be another solution but to generate a temporary setup.py to pip install it. I am not a fan of it but for now it will do, at least until the ability to specify a build backend in pyproject.toml is supported (see https://www.python.org/dev/peps/pep-0517/).

[adam-grant-hendry]

@adam-grant-hendry IMO pip install -e <3p-pkg> from within poetry shell, given new enough pip.

By itself, this does not update the lock file nor the pyproject.toml file.

[NeilGirdhar]

@adam-grant-hendry Why do you want to change pyproject.toml?

Usually, "installing a third-party package in development mode", it's because you're working on that third party package locally. You shouldn't be commiting local paths to pyproject.toml for other people in your project to see.

In an ideal world, there would be a local version of pyproject.toml, say pyproject_local.toml, that would contain local overrides of various options. This file would not be checked in to the repository. However, no such file exists, so I think @dimaqq has the best solution we have today.

[adam-grant-hendry]

@NeilGirdhar To manage dependencies. The package, though development, is required by the project. Users must be able to depend on it, but not accidentally download the actual third-party package. poetry will hash the contents of my package and yield a different SHA than the third party-package since I will have edited it.

Local paths are necessary since, if you see from previous posts, only path= inline tables in the toml accept develop=true argument, which is required for poetry v1.1+.

I appreciate your and @dimaqq opinions, but I do not agree they are solutions nor the best existing solution.

[NeilGirdhar]

@adam-grant-hendry

Users must be able to depend on it, but not accidentally download the actual third-party package

You keep your third-part dependency in pyproject.toml, but users can't use your local paths. Those are local to you. That's why your local paths don't belong in the committed version of the shared file pyproject.toml.

, if you see from previous posts, only path= inline tables in the toml accept develop=true argument, which is required for poetry v1.1+.

Yes, I know about this feature. The problem with this solution is you must be very careful not to accidentally commit pyproject.toml. The advantage of your solution is it goes over the dependency management logic and ensures consistency. Personally, I find pip install -e to be more convenient, but I understand the attraction to the other solution.

If I'm still not understanding you, could you flesh out the workflow you're trying to accomplish?

[adam-grant-hendry]

@NeilGirdhar I can flesh out my workflow. Is there a place we can take this offline? This issue post is getting long and I don't want to distract from the main issue.

[NeilGirdhar]

@adam-grant-hendry I think the best thing is to start a new issue or discussion on poetry so that interested people can follow along?

[adam-grant-hendry]

@NeilGirdhar I'm fine with that. Let's continue on the Discussions tab.

[NeilGirdhar]

@adam-grant-hendry I'm not sure.

[adam-grant-hendry]

@NeilGirdhar I've created a new Discussion titled "RE: Issue #34 - pip install -e . equivalent?" (#5276). Please feel free to join me there.

[adam-grant-hendry]

It's not about adding commands to poetry. This issue was about extending the backend to support editable installs using a package installer like pip, which it now does.

@NeilGirdhar
This is a distinction without a difference. It reads the same as "this issue isn't about adding a feature. it's about adding a feature."

There is a reason why I said what I am saying is "in simple terms"

[NeilGirdhar]

This is a distinction without a difference. It reads the same as "this issue isn't about adding a feature. it's about adding a feature."

There is a difference:

Adding commands to poetry means adding to the front end, i.e., things you do on the command line poetry add, poetry install, etc. Extending the backend is completely different. These are not commands that the user does. These are hooks that are accessed by the installer.

The reason that the backend has to change is that you need to be able to install a poetry project in editable mode in an ordinary (non-poetry) virtual environment. The front-end won't help you there.

[golgor]

Another possible reason to be using a setup.py is when handling with system drivers. I was working with a python project that needed control of the I2C-bus to communicate (i.e. i2c-dev).

I set everything up with adding my own user to the correct group (i2c), and everything was working when running the script outside of Poetry (running via Pyenv). When I tried to run it via poetry run I only got errors regarding permissions. I double-checked the user in the virtual environment, and everything seemed fine but still couldn't get it to work.

I didn't spend a lot of time on it as I found the workaround in here regarding moving the setup.py from the tarball and installing my package as editable install via pip to work fine. Might even be there is already some workaround in Poetry?

If that is not the case, this might not be a reason for making changes. It is however a possible solution if you are working closer with system drivers and are having issues with permissions.

[adam-grant-hendry]

@golgor Are you looking for help? If so, I'm happy to help where I can on a Discussion. Let me know. You could provide some more info about your pyproject.toml and dig deeper.

[golgor]

@adam-grant-hendry As mentioned I got it to work, but it might not be a good solution. I will write in the discussion, thanks for reaching out!

Edit:
I tested it again just now with using poetry run <script> and it was actually working. I did some trouble-shooting again and it seems that the problem might actually be in the implementation of the terminal in VS Code. If I run poetry run <script> in my system terminal, everything works great, but if I run the same command in the internal VS Code terminal I get permission problem (even though the user is correct).

I.e. this doesn't seem to be a problem for Poetry but rather some settings regarding the terminal setup of VS Code. I guess it doesn't make sense to write anything in the discussion then. I will look into this elsewhere. Thanks a lot though for reaching and offer help.

[thautwarm]

Cross-language project here. Poetry troubles a lot.

[neersighted]

Hey all, this issue has been conflated and confused many times, and the issue it was meant to address has been long since solved. Most of the conversation relates to workflow issues, X-Y questions, or possible feature requests.

For those stumbling onto this issue, check out @finswimmer's summary, which I will re-state here:

• When you poetry install, the root package(s) (the ones that live in your repo) are installed as editable into your venv.
• Path dependencies with develop = true in pyproject.toml are installed as editable.
• Poetry 1.2 betas add a poetry add --editable that sets the develop flag in pyproject.toml.

poetry-core also supports PEP 660 in the stable and beta versions, which allows for a Poetry-managed project to be installed in editable mode by any PEP 517 frontend (e.g. Poetry itself, pip, etc) that also supports PEP 660.

If you want to discuss workflow issues/improvements, please head over to the Discussions tab or join the Discord server.

If you're looking for a feature/enhancement that can streamline your workflow related to this, please do a good look to make sure there are no duplicate issues first (or ask on Discussions/Discord if you're not sure) -- then open an issue with your suggested feature/design.

I'll be locking this issue as it's considered long-solved and further comments are just sending a ton of notifications and blurring the issues under discussion.

[neersighted]

An addendum from @mraleson on Discord:

If you try to install a poetry package locally using pip in editable mode pip install --editable ./ and get an error like ERROR: File "setup.py" not found. Directory cannot be installed in editable mode. You must make sure your pip version is > 21.3 and your poetry project specifies poetry-core version > 1.0.0:

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

In essence, what I wrote about above only applies to contemporary versions of poetry-core and pip.