Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip install -e . equivalent? #34

Closed
kootenpv opened this issue Apr 3, 2018 · 57 comments

Comments

@kootenpv
Copy link

commented Apr 3, 2018

We cannot run pip install -e . because there is no setup.py. What would be the way to achieve this using poetry?

@sdispater

This comment has been minimized.

Copy link
Owner

commented Apr 3, 2018

No, this is not possible right now. And I don't think I will add an equivalent since it always seemed like a hack to me. But I might be wrong.

Why do you need pip install -e .? So I can see if there is a "poetry way" to do what you want.

@kootenpv

This comment has been minimized.

Copy link
Author

commented Apr 4, 2018

I often depend on some local packages, for which I want to be able to change some code and not have to reinstall.

@sdispater

This comment has been minimized.

Copy link
Owner

commented Apr 4, 2018

I see.

The thing is poetry is a dependency/project management tool and is not intended for such use cases.

However, if you really want to to this, you can build your project poetry build and take the setup.py file from the generated tarball and then you should be able to do pip install -e .

@kootenpv

This comment has been minimized.

Copy link
Author

commented Apr 4, 2018

Ah sorry, with local I mostly meant "developing local packages". I do believe poetry is also for helping with that? It sounds like barely any work to automate that part and create e.g. the poetry develop command.

@sdispater

This comment has been minimized.

Copy link
Owner

commented Apr 4, 2018

Well, poetry can help you develop local packages by orchestrating your dependencies, sure. But it is not meant to support the use case of having only local packages that depend on each other since this is not the best practice that poetry tries to endorse.

However, if you are willing to make a Pull Request, I would gladly take a look at it.

@winni2k

This comment has been minimized.

Copy link

commented Apr 5, 2018

I might be off the mark here, but in my experience, I usually use pip install -e . in order to allow my acceptance tests to see the python project I am testing. Is that what we are talking about here?

@sdispater

This comment has been minimized.

Copy link
Owner

commented Apr 5, 2018

I don't see why you would need pip install -e . for your tests to work. If you are inside a virtualenv your dependencies should be picked up by your tests.

In poetry that would be:

poetry install  # install your dependencies
poetry run pytest tests/
@kootenpv

This comment has been minimized.

Copy link
Author

commented Apr 5, 2018

I'm not sure, but when you share a module in git, with other dev members, it's nice when it is updated automatically on pull. Or is this automatically managed in pycharm or something? I use emacs without virtualenvs but just simply different python versions.

@winni2k

This comment has been minimized.

Copy link

commented Apr 6, 2018

In Pycharm you set the environment by choosing a python interpreter in your directory tree. Packages are discovered from that AFAIK.

@winni2k

This comment has been minimized.

Copy link

commented Apr 6, 2018

I guess this is a bit of a style question, but I like to build a wheel for local packages and use pip to install them into my venv. Is that the workflow that is intended by poetry?

@sdispater

This comment has been minimized.

Copy link
Owner

commented Apr 6, 2018

@winni2k Not at the moment, no. For now, poetry assumes each dependency is located in a remote repository (PyPI or a private one) or is a VCS dependency, and does not support local packages.

That might change in the future, however but I can't give you an ETA.

I am trying first to support the most common use cases, like installing from PyPI before diving into more specific workflow, especially if they are not the recommended way of managing packages.

@kootenpv

This comment has been minimized.

Copy link
Author

commented Apr 6, 2018

It now also does not allow me to use it in the interactive interpreter :/

I tried using poetry build, then copy out the setup.py to the root level but that doesn't work. I'm very disappointed :(

Until this becomes important, I wish there'd be a nice hack. I'm wondering though how do you develop - how do you avoid having to constantly reinstall?

@sdispater

This comment has been minimized.

Copy link
Owner

commented Apr 6, 2018

@kootenpv I just tested and copying the setup.py file works.

I don't know how you develop, but I never need pip install -e .. I either upload my packages to PyPI or a private index, or I install from a git repository.

@WouterVH

This comment has been minimized.

Copy link

commented Apr 18, 2018

I don't know how you develop, but I never need pip install -e .. I either upload my packages to PyPI or a private index, or I install from a git repository

Ugh, that is not developing, that is installing a finished package.

These are roughly equivalent, in your src dir:

pip install -e .
python setup.py develop

It's called "installing a package in development mode". They are the starting point of beginning development. If you are not familiar with that, then I honestly do not understand how you do development.

I know two ways of doing development without it:

export PYTHONPATH=
or in your code:
import sys
sys.path.append(....)

Are you recommending this? (I hope not)

@sdispater

This comment has been minimized.

Copy link
Owner

commented Apr 18, 2018

No, I don't modify sys.path(). And please don't be condescending.

So, if I develop a library (or an application for that matter) I put myself in my project directory. At this point I have access to my module: I can import it in my tests to test it.

You can test this yourself: clone the poetry repository place yourself in it install the dependencies with poetry install and execute poetry run pytest tests/ and everything will work. You do not need pip install -e .

If you need it in another project, and that is why you install it in develop mode, the you have dependencies between project and in this case I recommend to use a private index and release preversions or installing from git repository and updating when needed.

@merwok

This comment has been minimized.

Copy link

commented Apr 18, 2018

There are many communities of Python users, using different categories of development tools. Depending on practices (e.g. code in src dir vs code at repo root) or tools (e.g. Django runserver vs. Pyramid pserve), you can rely on «current dir is sys.path[0]» or you always install in develop/editable mode.

@merwok

This comment has been minimized.

Copy link

commented Apr 18, 2018

Another use case is when you’re trying to fix something in a library used by your project. From your project’s virtualenv, you run pip install -e path/to/library so that every change you make in the library repo is directly picked up in your project where you are testing the results.

@sdispater

This comment has been minimized.

Copy link
Owner

commented Apr 18, 2018

@merwok I understand your point, but most of the time you don't need it, in my opinion.

Regarding you last example, this will be supported in the next feature release since it will add the ability to install from a directory.

I know that there is a lot of different workflows out there, but poetry can't support them all. It tries to enforce some of them to help make the dependency management and packaging in Python easier and for that it has to be somewhat opinionated.

I am not saying it will never make it into poetry but since I am the sole developer I have to focus on a workflow that will suit most cases to develop applications and libraries.

@kootenpv

This comment has been minimized.

Copy link
Author

commented Apr 19, 2018

@sdispater Thanks for your way of developing (private index / from git), I wasn't aware of another way.

I can agree that under some conditions it is more proper and might lead to fewer bugs!

In my case, I do a lot of data science and "live" in Interactive python. It's very convenient when I can just do pip install -e . once and I'm just able to change the code so that in another module it will be picked up and see the results quickly (rather than having to run any install command again). That is the step before writing tests.

I understand you are hesitant from your perspective as it does not seem like a really clean use case, and it looks like the solution would have to be a bit hacky?

Still, I hope you could guide us and be accepting of a solution that might help a few (and not bother others)!

Currently the only way I see is something like:

  1. poetry build
  2. find the build package (?)
  3. extract it
  4. move the setup.py a level up
  5. find the python version (?) used by poetry and use it like python setup.py develop

Does this sound like the best way?

@ipmb

This comment has been minimized.

Copy link

commented Apr 19, 2018

I'll chime in and say I also use pip install -e . when doing local development. It is helpful for a couple reasons:

  1. It puts my project on the Python path. I don't have to depend on being in a specific directory for things to work which feels hacky to me.
  2. It sets up scripts and entry points I wouldn't have otherwise.
@sdispater

This comment has been minimized.

Copy link
Owner

commented Apr 19, 2018

@kootenpv Don't get me wrong, I can understand why you would use pip install -e, especially if you want to debug a dependency.

And, like I said, directory dependencies will be introduced in the next feature release so, with them, you will be able to reproduce this behavior.

@moigagoo

This comment has been minimized.

Copy link
Contributor

commented Apr 22, 2018

@sdispater

You can test this yourself: clone the poetry repository place yourself in it install the dependencies with poetry install and execute poetry run pytest tests/ and everything will work. You do not need pip install -e .

Hi! Could you please clarify how you run tests with pytest without installing poetry in the virtualenv? I can't import my package from pytest tests because it's not installed and resides one level higher than the tests.

It seems like poetry has the same source layout, but for some reason the imports work (although the tests fail). Am I missing something?

UPD: Even the pytest docs say that you can only run tests against an installed version of your package under this layout: https://docs.pytest.org/en/latest/goodpractices.html#tests-outside-application-code

@moigagoo

This comment has been minimized.

Copy link
Contributor

commented Apr 22, 2018

Sorted this one out. Should've put an empty __init__.py file in tests.

@blueyed

This comment has been minimized.

Copy link
Contributor

commented Apr 22, 2018

poetry build and take the setup.py

What do you think about some option to generate the setup.py file directly?

@bersace

This comment has been minimized.

Copy link

commented Apr 24, 2018

Is it related to #47 ?

If i summarize it, the need is that a poetry managed project should be installable as editable. e.g. i want to debug that library used by my project, i git clone it and poetry install -e the path ?

@bersace

This comment has been minimized.

Copy link

commented Apr 24, 2018

Or maybe the goal is just to have project console scripts installed in virtualenv ?

@sdispater

This comment has been minimized.

Copy link
Owner

commented May 2, 2018

@bersace If you want to test you console scripts, it's possible via the poetry script my-script. For example for poetry that would be poetry script poetry.

Also, I understand the use case of pip install -e path/to/some/dependency/ to allow debugging a dependency since you won't have to reinstall each time you make a modification. That's why it will be possible in the next feature release.

However, I do not think pip install -e . (or python setup.py develop) is that useful and as such introducing an equivalent in poetry is not something I plan on adding at the moment.

Now, that being said, if someone can give me a compelling use case where it's necessary, I will gladly change my view on the subject.

@kootenpv

This comment has been minimized.

Copy link
Author

commented May 2, 2018

I think I get what you mean: you want to enable the option for a package (A) to list a development version of another package (B).

I perhaps see a problem though.

I believe you're saying you could list B as a requirement in A in the new version by listing pip install -e path/to/some/dependency/ in A's poetry file (or some similar syntax, maybe just relative path)?

Isn't the problem then that if both are poetry packages, A will not be able to pip install -e B since poetry does not generate a setup.py for B?

@sdispater

This comment has been minimized.

Copy link
Owner

commented May 2, 2018

@kootenpv Exactly.

I think it would take the form:

[tool.poetry.dependencies]
dependency = { "path" = "relative/path/to/folder/" }

And with the equivalent of -e:

[tool.poetry.dependencies]
dependency = { "path" = "relative/path/to/folder/", develop = true }

Regarding the issue when the dependency is also a poetry package: at the moment, there won't be another solution but to generate a temporary setup.py to pip install it. I am not a fan of it but for now it will do, at least until the ability to specify a build backend in pyproject.toml is supported (see https://www.python.org/dev/peps/pep-0517/).

@kootenpv

This comment has been minimized.

Copy link
Author

commented May 7, 2018

@sdispater So I just tested it:

pip install -e .
pip instal my_package

In my case it looks like it was installing it again, but the binding shows it is still using the develop version.

It looks like pip does not care about version when you just do pip install my_package (we knew that, but also not with editable/dev already installed). Whenever you write pip install my_package==1.0.0, it will install it from pip if it doesn't have it locally.

In case you are talking about whether it remove the editable mode status in the file when you poetry install, I guess you wouldn't want that, but maybe a warning that an editable package is being linked?

As for being allowed to publish to pypi - I think that should not be allowed. Or maybe an input You have an editable package X, are you sure want to publish [y/N]?

@radix

This comment has been minimized.

Copy link
Contributor

commented May 7, 2018

I think #47 (path dependencies) should be able to solve this use case now?

@sdispater

This comment has been minimized.

Copy link
Owner

commented May 7, 2018

@radix #47 is a first step towards it but path dependencies cannot be installed in editable mode at the moment. This is something planned for the next feature release.

@mmerickel

This comment has been minimized.

Copy link

commented May 7, 2018

@sdispater For the editable project mode from point 2 above, I suppose it's important to ask whether poetry has goals of providing a PEP 517 compliant build system for projects in the future? Presumably if that were the case then this is solved via pip install -e <path> on the path containing the pyproject.toml which would tell poetry to install itself in editable mode. Similarly a pip install <path> (not editable) is effectively equivalent to poetry build && pip install dist/*.whl in a future when you can upload a project created with poetry to PyPI without needing to generate a fake setup.py. I know this is futuristic but it helps me understand where poetry is going and whether it even makes sense for poetry to support the develop mode - although I suppose it could support poetry develop and in the future embrace the pip integration. This seems like a pretty logical step for poetry as it already has ambitions of making projects redistributable via poetry build and poetry publish.

@sdispater

This comment has been minimized.

Copy link
Owner

commented May 9, 2018

@mmerickel Yes, I intend to support a build system. I am not sure it will be integrated directly into poetry since I don't want people to have to depend on the whole poetry's codebase to build a project. So it will likely be another package whose sole purpose will be to build a project from a poetrypyproject.toml file which is more or less the purpose of the masonry package inside poetry so it might be extracted from there.

@moigagoo

This comment has been minimized.

Copy link
Contributor

commented May 23, 2018

With the introduction of directory dependencies, I thought poetry add --dev --path . mypackage would be possible and would completely solve this issue. This would install the package itself in the virtual environment from the current directory.

However, you can't do that right now (sample taken from my project Cliar):

$ poetry add --dev --path . cliar
[InvalidProjectFile]
[dev-dependencies.cliar] {'path': '.'} is not valid under any of the given schemas

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--allow-prereleases] [--dry-run] [--] <name> (<name>)…

(The line cliar = {path = "."} is added to pyproject.toml though.)

@sdispater Is it an issue to report or a design decision?

P.S. My two cents on the topic, and why I need poetry add --dev --path . cliar to work. Like your own Cleo, Cliar is a package to create CLIs. Because the purpose of the tool is to generate a commandline interface from Python code, I'm testing it this way:

  • Create a .py file with the interface definition. This file obviously needs to be able to import cliar in order to create the CLI to test.
  • In test files, I'm calling the .py files with subprocess.run like so: run('python clitest.py arg1 arg2', shell=True).
  • I'm checking the output with Pytest's capfd fixture.

It would be really nice to put all the tests in tests folder, put the sample .py files in a subfolder, and discover them in the tests using pytest-datadir plugin.

However, I currently cannot do that. Whereas test_*.py files can import cliar, the sample files can't since the cliar package is not installed.

The workaround I use now is place the tests and the sample in the root dir. But I'd rather keep my tests in an isolated directory.

If there is a way to solve my problem without poetry add --dev --path . cliar I'm not seeing, please advise.

Thanks!

@sdispater

This comment has been minimized.

Copy link
Owner

commented May 28, 2018

Release 0.10.0 adds a new develop command which does something similar to pip install -e ..

The install command now supports a --develop option which installs path dependencies in development/editable mode. 

@kootenpv

This comment has been minimized.

Copy link
Author

commented May 28, 2018

Thanks a lot for the nice surprise!

A couple of observations:

  • the --develop option is missing from the CLI command install (good: badusage is thrown, and poetry develop documentation exists)
  • I thought I can add a folder containing a local package and install it using poetry install --develop, but that's not how it works?
  • poetry develop allowed me to install the "current" package, and import it, but only if I'm in that folder (e.g. in ~/mypackage. You cannot import mypackage from the example location ~/otherpackage)

Maybe I'm misunderstanding the usage?

@sdispater

This comment has been minimized.

Copy link
Owner

commented May 28, 2018

@kootenpv You must pass values to the --develop option. Let's say you have my-pkg1  and my-pkg2 that are path dependencies, you would do:

poetry install --develop my-pkg1 --develop --my-pkg2

poetry develop installs the package in development mode for the project's virtualenv, so that it's available when executing poetry run. If you manage your virtualenvs yourself then it will be installed in the current activated virtualenv.

@moigagoo

This comment has been minimized.

Copy link
Contributor

commented May 29, 2018

@sdispater Sorry, I still don't quite understand why I can't import my package after running poetry develop. When I run poetry develop, the package gets installed in the current venv, right? So, if I import it from a script in a project's subfolder and run the script with poetry run foo/bar.py, it should work, since poetry run uses the same venv.

It doesn't work however. Is it a bug or expected behavior?

P.S. Congrats on the big release and thanks a ton for your work!

@sdispater

This comment has been minimized.

Copy link
Owner

commented May 30, 2018

@moigagoo Yes it should work. So if it's not it's likely a bug.

@moigagoo

This comment has been minimized.

Copy link
Contributor

commented May 30, 2018

@sdispater made a separate issue #162

I think this one can be closed, since the equivalent of pip install -e . does exist now.

@p5a0u9l

This comment has been minimized.

Copy link

commented Oct 20, 2018

So, this issue is marked "closed" and yet it seems the command mentioned above develop is now deprecated. has pip install -e . survived through some other mechanism? or is there another thread discussing this?

@jacebrowning

This comment has been minimized.

Copy link
Contributor

commented Oct 20, 2018

@p5a0u9l The behavior on installing the package in an editable mode (poetry develop) is now part of poetry install.

@phiresky

This comment has been minimized.

Copy link

commented Dec 10, 2018

Still don't understand how pip install -e . is supposed to be solved:

[tool.poetry]
name = "cooli"
...

I have these files:

a/b.py

import c.x
print(c.x.test)

c/x.py

test = "hi"

I tried multiple things:

poetry install --develop cooli
poetry install --develop .
poetry add cooli --path . (just shows AssertionError)
poetry add .

With pipenv this works completely fine. With poetry, how is this supposed to work?

I always just get

Traceback (most recent call last):
  File "a/b.py", line 1, in <module>
    import c.x
ModuleNotFoundError: No module named 'c'
@jgirardet

This comment has been minimized.

Copy link
Contributor

commented Dec 10, 2018

just do poetry install it will do pip install -e .

@phiresky

This comment has been minimized.

Copy link

commented Dec 10, 2018

Apparently not - full example here: https://github.com/phiresky/poetry-install-self-issue

pipenv install && pipenv run python a/b.py works

poetry install && poetry run python a/b.py throws ModuleNotFoundError: No module named 'c'

@radix

This comment has been minimized.

Copy link
Contributor

commented Dec 10, 2018

Do you have the necessary __init__.py files?

@phiresky

This comment has been minimized.

Copy link

commented Dec 10, 2018

__init__.py files aren't required anymore (as of python 3.3). But even if i create empty __init__.py files in /, /a, /c, it still doesn't work with poetry and does with pipenv.

@sdispater

This comment has been minimized.

Copy link
Owner

commented Dec 10, 2018

@phiresky Poetry won't find your packages automatically since the name of the project is different. You need to specify them explicitely as documented here: https://poetry.eustace.io/docs/pyproject/#packages

@phiresky

This comment has been minimized.

Copy link

commented Dec 10, 2018

Thank you! That works :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.