Unable to install direct path dependencies as editable

[mbway]

Description

pip install -e does not install path dependencies as editable. This is OK as a default but there doesn't seem to be any way to configure this behaviour.

I have seen this issue and this stack overflow post where others have wanted this behaviour (or similar behaviour) as well. poetry install does work as expected but poetry may not always be available.

In install_req_from_req_string() where the InstallRequirement for is constructed from the path requirement string I see that there is no way to have the editable attribute set.

Expected behavior

I expected that pip install -e would install all the local/path dependency packages in editable mode like poetry install does.

pip version

23.0.1

Python version

3.11

OS

Ubuntu 20.04.5 LTS

How to Reproduce

• create two packages, p1 and p2
• add p2 as a path dependency of p1: p2 = {path = "../p2", develop = true}.
  • here using poetry but I don't think it matters
• run pip install -e p1

Output

p1 is installed in editable mode but p2 is not. Even if p2 is installed in editable mode beforehand it will be re-installed as non-editable.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

1. I don't know what a "path dependency" is. That may be something poetry-specific.
2. Pip only installs the package(s) specified with -e in editable mode. This is by design.

Even if p2 is installed in editable mode beforehand it will be re-installed as non-editable.

Pip shouldn't reinstall anything it does not need to. If it does need to reinstall, then yes, it will install in "normal" (non-editable) mode - there's no way using standard metadata to say that a dependency must be in editable mode (that's the thing I presume is a Poetry specific feature), so that's all that pip can do.

So I don't think there's a pip issue here, but if you believe there is, you'll need to provide a reproducible example demonstrating the problem without using poetry-specific features.

[mbway]

path dependency: perhaps I didn't use the proper terminology but it ends up being a direct reference to the package directory. i.e. p2 @ file:///some/path

poetry does have the develop key in it's dependency table which specifies whether the package should be installed in editable/develop mode when installed with poetry install but I understand that this is non-standard and not something pip has access to.

I see that when installing with pip install, no metadata is attached to the requirement that pip could use to decide on its own to install in editable mode but for my use case and I imagine many others, having a flag like --all-local-editable or something would be all that is required (install all packages found on the local machine in editable mode).

[mbway]

The problem I'm having is that if I have multiple projects which depend on each other, regardless of the build system I'm using I would like to install them in editable mode, in which case installing just the root package editable and the rest non-editable is undesirable.

[pfmoore]

OK, so that's not something pip supports currently. I don't know whether there is sufficient demand for such a feature to justify it (as far as I know, this is the first time anyone has ever requested it) but if someone were to demonstrate that it's a sufficiently useful feature to justify the maintenance costs, and was interested enough to contribute a PR, I guess it's something we could consider.

Until there's a PR, I don't think there's much benefit in discussing details of the design of such a feature, though, so I'm going to add an "awaiting PR" label here.

[sbidoul]

@mbway sorry I had not seen this issue before your PR.

Have you considered declaring dependencies by name instead of path, and installing with pip install -e ./p1 -e ./p2.
Regular installation can then be done with a constraints file declaring the local directory paths.

Otherwise, I tend to think this feature request is a special case of the "dependency override" mechanism that is discussed elsewhere, and we may want to consider it in that broader context.

[mbway]

Have you considered declaring dependencies by name instead of path

you mean in the project specify mylibrary = "*" instead of specifying where mylibrary is then in the requirements file specifying -e /path/to/myproject -e /path/to/mylibrary.
I suppose that works as a workaround but isn't as convenient when I just want to re-install a single project with pip install -e . in which case it will look for mylibrary in pypi right?

And my PR explains another situation involving this approach to monorepo management where using path requirements is much easier to manage than custom installation procedures for each project.

So unless I'm missing something, using named requirements isn't a good substitute

[sbidoul]

Another alternative could be to install everything, and then use pip install -e /path/to/mylibrary --no-deps for the subprojects you want editable?

[mbway]

This still seems like a manual workaround for a missing feature. I mean I could just create a custom tool (and I have done for the time being) but I think this is a feature missing in pip

[pfmoore]

I'm still not convinced this is worth adding. I get that you need this functionality, but we have to balance wider issues like maintainability and how many people would benefit. This feels more like a "workflow support" feature rather than a lower level "installer" feature. You mention that poetry supports this - maybe the answer should simply be that poetry is the right workflow tool for you? I don't understand what you mean by "poetry may not always be available" - it's absolutely reasonable to require that people working on your code use the project standard workflow tool, so it seems to me that your response to someone saying "I don't have poetry" should just be "you need to follow the instructions for setting up your development environment, and then you will have poetry".

[mbway]

I don't mind if you don't think this feature is a good addition to pip. I opened the issue because I saw it as a missing feature that I thought would be useful and wanted to contribute.
I think that the feature does have uses and the fact that poetry install has the option to install editable requirements as editable shows that it is at least a somewhat established workflow.

In terms of poetry 'not being available' I didn't want to get into the details but for the CI system I have set up there would be several drawbacks to using poetry install

• the credentials to the private index would need to be written to a toml config file and XDG_CONFIG_PATH would have to be set to configure poetry to use it, whereas pip is able to authenticate using environment variables
• pip is configured to fetch from specific indexes globally whereas poetry has to be configured per-project
• poetry has some annoying rules about where and when it creates virtual environments and what interpreter it uses. I have found just creating environments myself and installing with pip to be less hassle
• pip has fewer dependencies and so is faster to install and is already available when creating a virtual environment
• some packages are not using poetry as they are not pure python (eg maturin packages with rust code)

by the way I'm not asking for solutions for the situation above. I have already found one. I presented it only as a motivating example for where this feature could be useful

[IterableTrucks]

I have a similar issue like @mbway . My package(say package_a) has a optional dependency(package_b) which is laid out as a git submodule:

package_a/
  package_a/
  package_b/(git submodule)
  pyproject.toml

pyproject.toml:

[project]
name = "package_a"
# ...
[project.optional-dependencies]
all=["package_b @ file://{path to package_a}/package_b]

I want to install both package_a and package_b in editable mode during development of the project, so I have to execute pip install -e package_b -e . . But the annoying thing is that package_b also has submodule dependencies like package_a (say package_c and package_d). So every time I have to execute pip install -e package_b/package_c -e package_b/package_d -e package_b -e . which is quite verbose.

I tried to add an optional dependency in pyprojec.toml of package_a like below:
dev-all=["-e package_b[dev-all] @ file://{path to package_a}/package_b"]
and similar content in pyproject.toml of package_b.
But pip raise exception on command pip install -e .[dev-all] :

raise InvalidRequirement(str(e)) from e
      setuptools.extern.packaging.requirements.InvalidRequirement: Expected package name at the start of dependency specifier

If pip allows editable option in dependency specification just like it does in the requirement file when executing pip install -r requirements.txt , there's no need to add an --editable-requirements option as @mbway suggests. I think it is a better choice than editable-requirements because you can control which dependencies to be installed in editable mode. Take package_a as an example, if I do not want to install package_c and package_d in editable mode, I can replace dev-all with all in dependecy spec:
dev-all=["-e package_b[all] @ file://{path to package_a}/package_b"]
Then package_c and package_d will still be copied and installed in site-packages directory of current Python environment.

Similar issue searched on Stackoverflow: pyproject-toml-listing-an-editable-package-as-a-dependency-for-an-editable-packa

[mbway]

I agree that the package itself specifying that a requirement should be installed in editable mode would be nice (which I think is what you are suggesting?).

The reason I went with the --editable-requirements route is because it is a minimal change (requires only about 15 lines of actual additional logic) that requires no substantial changes to the pip internals.

Adding additional metedata to track whether a requirement should be editable, and potentially adding additional syntax for specifying requirements that isn't covered by PEP 440 seems much less likely to get off the ground, especially since the initial response is that there is no issue with pip's current capabilities.