How can I install a local version of my package with poetry? non setup.py file

[SHxKM]

I know that poetry is supposed to replace setup.py, but how can I develop a package on my local machine when using poetry, I've read that the equivalent of pip install -e . in poetry is simply poetry install from the root directory of the project, but even though poetry does state it finished installing my package, I get import errors everywhere.

I would ideally want to install my package system-wide, or a virtualenv of my choice, so I can run it no after going into poetry shell, but even that isn't working.

Edit: one thing I noticed, is that my package is (re?) installed every time I run poetry install..

[SHxKM]

I figured out the problem. Because poetry is directory based, then I can't import my local package in a file in another directory (I guess the virtualenv isn't activated there). Is there a way around this?

[jacebrowning]

I can't import my local package in a file in another directory

Can you clarify what you're trying to do? Either of the following should work:

$ poetry install
$ poetry run python
>>> import my_package

$ poetry install
$ poetry run python my_script.py

[SHxKM]

I'm talking about being able to import my package inside a file outside of my package directory.

# ~/Desktop/test_file.py
from mypackage.decorators import deco

In Terminal:

$ cd Desktop
$ ./test_file.py

This throws an ImportError because the file is outside of my package's folder. It also throws an error in VSCode for example, because I'm using it's not using poetry's own virtualenv.

With pip and setup.py I can simply do pip install -e . from within my package's directory, and then use it liberally anywhere, whether working in an IDE or the Terminal.

[djetelina]

The problem is: when you do pip install -e, you're using your system pip, however poetry is installing your package within the virtualenv, so if you don't want to use your virtuelnv for the test_file, you might do poetry config settings.virtualenvs.create false, but it is not recommended, it's even discouraged.

[melMass]

14 downvotes... people are crazy

[jaklan]

@melMass no, people are not crazy, but they are enough experienced to know this answer doesn't have any sense - 1) is completely unrelated to the question 2) is simply wrong.

[melMass]

@melMass no, people are not crazy, but they are enough experienced to know this answer doesn't have any sense - 1) is completely unrelated to the question 2) is simply wrong.

I misread the first part, my bad!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[merwok]

To answer the original question:

If I understand correctly, poetry is a development tool. It works in a virtualenv (one it creates or one that is managed externally, depending on your options) and provides commands (including install) to work on a project in an isolated fashion, because that’s what we want when developping.

To have a project globally installable, we shouldn’t reach for a dev tool but for an installation tool: pip! Thanks to PEP 517 integration, you can use pip install path/to/project/with/poetry/config and get your tool installed globally (using any usual pip options such as --user or --target to control where things end up).

Hope this helps!

[dash-samuel]

Thank you that worked for me!

[MestreLion]

To have a project globally installable, we shouldn’t reach for a dev tool but for an installation tool: pip! Thanks to PEP 517 integration, you can use pip install path/to/project/with/poetry/config and get your tool installed globally (using any usual pip options such as --user or --target to control where things end up).

Problem is, pip install -e requires setup.py, as stated by this error message:

01:15:27 rodrigo@desktop ~/work/minecraft/nbtlib master $ pip3 install --user -e .
ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/rodrigo/work/minecraft/nbtlib
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

So, if a given library uses Poetry, how can I clone its github repo and install it locally to use in my projects? (in editable mode, so I can also make changes to the library itself)?

[rmvanwyk]

So, if a given library uses Poetry, how can I clone its github repo and install it locally to use in my projects? (in editable mode, so I can also make changes to the library itself)?

I think what may help you is the poetry build command. This will create a wheel and a tar file and place it in the my_pacakge/dist directory which you can then pip install into your environment of choice.
ex) pip install my_package-version.tar.gz
It won't be in editable mode so you'll have to rebuild the package whenever you make changes but I think this solves your problem quoted above. AFAIK, there is no direct equivalent of pip install -e for poetry, as poetry install simply installs the dependencies defined in pyproject.toml.

[cgarciae]

Did this behavior change? I just tried and the package was installed locally with poetry install.

[jaklan]

Did this behavior change? I just tried and the package was installed locally with poetry install.

The issue is not about a local development of a package, but about installing the package in the editable mode in another project, which normally would be handled via pip install -e /path/to/package. Now there's no such an equivalent, there are only dirty workarounds like the one mentioned by @rmvanwyk. @sdispater such issues are a huge blocker to use poetry for a serious development.

[cgarciae]

@jaklan I see, thanks for the explanation. So I am guessing that using

my_package = { path = "my/package/path" }

Would not install it in editable mode?

BTW: is editable mode just putting a symlink in site-packages?

[jaklan]

@cgarciae Yeah, that would work:

Path dependencies pointing to a local directory will be installed in editable mode (i.e. setuptools "develop mode"). It means that changes in the local directory will be reflected directly in environment.

But... only when you also use poetry for that second project ;) Which is not always possible, especially with some legacy projects or e.g. Data Science projects, which are often based on conda envs and files (which is not a problem normally, because in environment.yml I can just use pip install -e some/path).

Ad. BTW - here's a detailed explanation:
https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode

[merwok]

Editable mode can be a symlink in site-packages (flit does that), or a pth file (setuptools method), and a dist-info directory. Both the import system and the packaging metadata systems need to find what they’re looking for.

[hay]

Maybe this issue should be renamed to something like 'Poetry should support installing local packages in editable mode'?

What works for me is indeed just symlinking the local library folder to site-packages in a venv. It definitely is not very pretty, but it works for me. Here are all the commands needed to set up a venv, install a local package, and then replacing it with the symlinked version.

    # I'm assuming your package is called 'my-lib' and your folder where
    # you want to test is called 'my-code'

    # Create venv first
    cd ~/my-code
    virtualenv env
    source env/bin/activate

    # Now install the local library
    pip install ~/my-lib

    # And delete the directory, assuming you're using python 3.6
    rm lib/python3.6/site-packages/my-lib

    # Now add a symlink
    ln -s ~/my-lib ~/lib/python3.6/site-packages/my-lib

[rtbs-dev]

@jaklan The data science worflow is exactly how I found this issue. Building a toolkit that will likely serve as a jupyter-centric assist within conda env. rather than require jupyter as a hard dependency, my plan was to have a notebooks top-level folder with an environment.yml file that allows a reader to reproduce locally.

This conda env would depend on a pypi-hosted version of my package eventually, but in development I'm used to satisfying that dependency via a local development installation (pip intstall -e ., etc).

Loving poetry so far but this is a bit of a roadblock to clean conda-centric dev.

[rosejn]

I struggled with this, and then learned that you can poetry add with a relative path. So inside of poetry project A, you can do $ poetry add ../other_project_b And then it will be available in your project A.

…

On Mon, Jun 22, 2020 at 1:57 PM Neil ***@***.***> wrote: @finswimmer <https://github.com/finswimmer> What do you propose users of poetry do to achieve editable installs? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/python-poetry/poetry/issues/1135#issuecomment-647739207>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAI53QVDBPAJPR7OHWM6L3RX6ZTHANCNFSM4HQQRHQQ> .

[evisong]

In my case, I forked an open source python lib and made some changes locally, that lib is also based on Poetry. I installed it via absolute path in my own project:

poetry add /path/to/python/lib

It works well as long as I'm not going to commit it to any git repo.

[jaklan]

I struggled with this, and then learned that you can poetry add with a relative path. So inside of poetry project A, you can do $ poetry add ../other_project_b And then it will be available in your project A.

Thanks for repeating exactly the same what @finswimmer wrote 2 comments above and not understanding the question at all - we're talking here about:

Installing a poetry managed project - or better: any project managed by pyproject.toml instead of setup.py - in development mode by pip

@NeilGirdhar it seems @rmvanwyk workaround with poetry build is one way (but still - it's not editable mode), another - generating setup.py from pyproject.toml with dephell, so pip install -e would work.

[NeilGirdhar]

@jaklan Right, I'm asking about editable mode. Generating setup.py is unfortunate since the whole point is to just rely on modern pyproject.toml. finswimmer might be right that this is pip's problem, but it doesn't seem that pip is going to fix this. Therefore, I would prefer if poetry simply added a develop command or editable flag to the install command. I currently do this in zsh:

function pyinstall {
    local package=${PWD##*/}
    ln -s -f $PWD/$package `python -c 'import site; print(site.getsitepackages()[0])'`/$package
    echo Installed $package
}
function pyuninstall {
    local package=${PWD##*/}
    rm -r `python -c 'import site; print(site.getsitepackages()[0])'`/$package
    echo Uninstalled $package
}

I'm sure it could be better.

[jaklan]

@NeilGirdhar I fully agree. The topic on pip side seems to be frozen, so we're completely stuck at the moment...

There was also some discussion here, but it's also dead for months... I really don't understand the pip core team approach to that, very important after all, issue.

[thmsklngr]

Hi,

just stumbled over this ticket. I had about the same problem, I wasn't able to install a package in editable mode within an environment created with poetry.
I used setup.py and setup.cfg to define all options for my package. Using it with pyproject.toml might be quite ok, but I miss somehow a good documentation for it. Additionally the support for data files is quite poor. So that's why I go for setup.py.

Anyway, whenever I ran poetry run python setup.py bdist_wheel, everything is fine. Also the installation in a different virtual environment worked pretty well. But stepping into the virtual Python environment (poetry shell) and running pip install -e . ends up with the error ModuleNotFoundError: No module named 'setuptools'.

My installed modules in the Poetry virtual environment:

Package     Version
----------- -------
autopep8    1.5.4
flake8      3.8.4
mccabe      0.6.1
pip         20.2.3
pycodestyle 2.6.0
pyflakes    2.2.0
setuptools  50.3.0
toml        0.10.1
wheel       0.35.1

setuptools is there, but why the heck it's not usable by setup.py? Any ideas?

I solved it by removing pyproject.toml, delete poetry.lock and created a normal virtual environment using venv

Regards, Thomas

[NeilGirdhar]

Update for people struggling with this. Editable installs seem to work for me now provided you do the following:

• Add the build system to your pyproject.toml:

[build-system]
requires = ["setuptools", "poetry_core>=1.0"]
build-backend = "poetry.core.masonry.api"

(This should fix @mosesontheweb's error)

• This step is no longer necessary. Produce a setup.py:

poetry build --format sdist
tar -xvf dist/*-`poetry version -s`.tar.gz -O '*/setup.py' > setup.py

• Now, pip install -e . should work!

However, installing a project in editable mode doesn't help you make it available to another poetry virtual environment. I don't think the proposed solution above of adding a relative path dependency is good because you might end up committing that dependency, which is probably not what you want. Edit: If it is what you want, poetry recently added an editable install command for that purpose.

The best approach I've found is to enter a poetry shell and from there pip install -e path_to_other_library. That way, poetry run commands within your shell will work.

Given that we have a workaround, I've come to accept that @finswimmer is correct that this needs to be solved at pip's end. The corresponding issue is pypa/pip#6950. They decided that they shouldn't decide and deferred to the Python people here: https://discuss.python.org/t/specification-of-editable-installation/1564/40 and pypa/packaging-problems#256 (comment). That discussion seems to be frozen. Edit: PEP 660 addresses this issue.

[phillipuniverse]

Thanks for the workaround, works great!!

I personally lumped this into a Makefile target:

setuppy:
	@poetry build --format sdist
	@tar --wildcards -xvf dist/*-`poetry version -s`.tar.gz -O '*/setup.py' > setup.py

Note that I also used --wildcards so that this worked on linux as well. To get cross-compatibility with MacOS, need to brew install gnu-tar.

The best approach I've found is to enter a poetry shell and from there pip install -e path_to_other_library. That way, poetry run commands within your shell will work.

The one-liner for this is, from the consuming project using Poetry, do poetry run pip install -e ../path/to/other/library.

[ekhaydarov]

im not so sure this is an issue with pip? I dont want to start a debate about pipenv, just anecdotal experience that i have managed to install editables fine with with that tool. I am trying your suggestion on how to install editable but I am still failing to do so with

  Build backend does not support editables, falling back to setup.py egg_info.

It installs fine otherwise. The project isnt even run by poetry or pyproject.toml. its a setup.py file.

I think theres a bit more to dig through in poetry before we go escalating it all the way up to python

[cglacet]

@finswimmer 's answer makes me wonder why there is no way to pin an answer. In threads like this one that would really be helpful.

[sinoroc]

@cglacet GitHub has a (beta) feature called discussion. I wonder if poetry is eligible. That would maybe a better fit for such questions that are not bugs.

https://github.blog/2020-05-06-new-from-satellite-2020-github-codespaces-github-discussions-securing-code-in-private-repositories-and-more/#discussions

[max-sixty]

We have it enabled if you want to check it out — it's pretty good!
https://github.com/pydata/xarray/discussions

[sinoroc]

https://github.com/python-poetry/poetry/discussions

I don't know if we are idiots, or blind, or if someone just played a trick on us... :D

[cglacet]

Wow, someone converted this issue to a discussion, nice.

[qdx]

To add to the discussion:
Using poetry add ../my/project seems to make a copy of the source file to your virtual env.
Which means, if you make updates in ../my/project, the change won't be reflected before you run poetry remove <project name> + poetry add ../my/project.

To get around that limitation, poetry document here already provided the answer to set develop = true. To make it work, what I did was:

1. poetry remove <project name>
2. Edit pyproject.toml and add <project name> = {path = "../my/project", develop = true}
3. poetry install

Now poetry will generate a link to the directory, so you can directly make changes in that repo. Hope this can help someone facing the same challenge as me.

[labdmitriy]

@qdx thank you for the answer, I've checked and didn't find duplicated packages in pyproject.toml.
But I've found that I can install my local package with the command "python setup.py develop", as I understand this is only right way to install the project in editable mode if it contains pyproject.toml file.

[revolter]

I'm receiving this instead:

> poetry install
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

  SolverProblemError

  Because python-telegram-bot (13.7 /Users/revolt/Development/GitHub/revolter/python-telegram-bot) depends on cachetools (4.2.2) which doesn't match any versions, python-telegram-bot is forbidden.
  So, because dexrobot depends on python-telegram-bot (13.7 ../python-telegram-bot), version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

[revolter]

I had to also run poetry update <project name> as a fourth step.

[MatanRubin]

This worked perfectly for me with poetry 1.1.7. I think this should be marked as the accepted answer instead of the one that is currently accepted.

[williscool]

To add to the discussion:
Using poetry add ../my/project seems to make a copy of the source file to your virtual env.
Which means, if you make updates in ../my/project, the change won't be reflected before you run poetry remove <project name> + poetry add ../my/project.

To get around that limitation, poetry document here already provided the answer to set develop = true. To make it work, what I did was:

1. poetry remove <project name>
2. Edit pyproject.toml and add <project name> = {path = "../my/project", develop = true}
3. poetry install

Now poetry will generate a link to the directory, so you can directly make changes in that repo. Hope this can help someone facing the same challenge as me.

In order to get this to work for me...

I had to MAKE SURE

[tool.poetry]
name = "backend"
version = "0.1.0"
description = ""
authors = ["William Harris <wharris@upscalews.com>"]

packages = [
    { include = "app", from = "./app" },
]

[tool.poetry.dependencies]

... blah blah more packageks...
app = {path = "./app", develop = true }

the path ./app for the app package in tool.poetry.dependencies and tool.poetry

[Okroshiashvili]

I wanted to install the existing package with the existing setup.py file in the poetry environment. I went through every message here and nothing works except one.

If you already have setup.py file and want to install the package in editable mode do the following:

in pyproject.toml file add the following:

[build-system]
requires = ["setuptools", "poetry_core>=1.0"]
build-backend = "poetry.core.masonry.api"

Then, just run pip install -e .

Only that worked for me

[NeilGirdhar]

I think I covered this in my comment 😄

[AndreiPuchko]

Recipe for vscode
I'm developing a package that depends on my other packages and I want to edit everything on the fly.
So what i did:

1. added my dependencies in the usual way, for example:

poetry add zzgui

2. renamed added packages folders inside site-packages

zzgui -> zzgui-

3. changes made in launch.json

    "env": {
        "PYTHONPATH": "../zzgui:../zzdb"
    }

4. changes made in settings.json:

"python.autoComplete.extraPaths": ["../zzgui", "../zzdb"]

It is not very clean, but at least i can do what i want :)

[TomekPro]

Is seems that it is enough to add

[tool.poetry]
# ...
packages = [
    { include = "my_package" },

Then import my_package works and the library is editable.

Documentation: https://python-poetry.org/docs/pyproject/#packages

Nothing like this is not needed:

[tool.poetry.dependencies]
# directory
my-package = { path = "../my-package/", develop = true }

[injuryholmes]

I've tried this

[tool.poetry]
# ...
packages = [
    { include = "my_package" },
]

It do works, then you can import the project from anywhere，and it's editable, just like pip install -e .

[noymer]

From pip v21.3,
pip supports editable install with pyproject.toml project.

Support editable installs for projects that have a pyproject.toml and use a build backend that supports PEP 660. (#8212)

https://pip.pypa.io/en/stable/news/#id254

[andrewrgarcia]

why not run the following as a solution?:

poetry build
pip install dist/*.tar.gz 

[merwok]

poetry is supposed to be a tool that handles anything

[justinTM]

it's supposed to be better than pip. the fact it can't do a lot of what pip can... hush go away issue close #wontfix lmao

[Secrus]

Poetry targets slightly different use cases than pip. It's not (and never was) meant as a pip replacement