Local dependency and packaging #464

nhuray · 2021-05-18T11:18:21Z

Is your feature request related to a problem? Please describe.

Hi Dear community

I'm struggling to build and package an app depending on a local dependency using PDM.

So maybe it's because I'm quite new to python (long time I didn't code in that language), or because I did not understand some concepts related to PEP 582, so please excuse me if my question is obvious (or stupid).

So I want to bundle an AWS Lambda function (in Python) as a Docker image (let's name it A) following the documentation provided.

That function depends on a library (local dependency named B) which itself can bring some external dependencies (C, D, E).

So basically the dependency tree is something like:

├── A  (local)
│   └── B. (local)
│       ├── C (external)
│       ├── D (external)
│       └── E (external)

Add B as a local dependency into A project

When I add to project A the local dependency B running pdm add -e ../relative/path/to/B I have:

an absolute path in the pyproject.toml :

dependencies = [
    "-e file:///Absolute/path/to/B#egg=B",
]

(This looks an issue to me because the pyproject.toml is not portable anymore)

a relative path in the pdm.lock :

name = "B"
sections = ["default"]
version = "0.0.1"
editable = true
path = "../relative/path/to/B"

a directory __pypackages__ loooking like that:

├── __pypackages__
│   └── 3.8
│       ├── bin
│       ├── include
│       └── lib
│           ├── B.egg-link
│           ├── easy-install.pth
│           └── A.egg-link

Install A

When I run pdm install I have:

[CHANGE IN 1.5.0]: dev-dependencies are included by default and can be excluded with `--prod` option
Synchronizing working set with lock file: 0 to add, 1 to update, 0 to remove

  ✔ Update B 0.0.1 -> 0.0.1 successful
Installing the project as an editable package...
  ✔ Update A 0.0.1 -> 0.0.1 successful

🎉 All complete!

Describe the solution you'd like

What I expect when I run pdm install in the A project is to resolve the local and external dependencies then install them to the __pypackages__ directory.

When I build the Docker image like that, I just need to copy the __pypackages__/3.8/lib directory.

So how do we do that ?

The text was updated successfully, but these errors were encountered:

pawamoy · 2021-05-18T17:59:16Z

The issue is that you are installing this local package in editable-mode (-e). Would it be possible for you to publish that local package to a PyPI index so you can depend on it normally?

nhuray · 2021-05-18T18:10:43Z

Sure but that's not exactly the philosophy when you use monorepo...

My goal is to build independent packages using local dependencies from the same monorepo without publishing those artifacts to a registry (PyPi or local python registry).

Something also discussed in the Poetry community: python-poetry/poetry#936

pawamoy · 2021-05-18T18:16:45Z

Maybe it would work if you specify the local package without -e, though you'd lose the editable benefits, requiring the re-run pdm install each time this local package changes.

nhuray · 2021-05-18T21:06:50Z

Unfortunately I tried and it's failing:

Install A 0.0.1 failed
Traceback (most recent call last):
  File "/usr/local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/core.py", line 184, in main
    return Core().main(args)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/core.py", line 145, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/core.py", line 140, in main
    f(options.project, options)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/cli/commands/add.py", line 35, in handle
    actions.do_add(
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/cli/actions.py", line 182, in do_add
    do_sync(project, sections=(section,), default=False)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/cli/actions.py", line 128, in do_sync
    handler.synchronize()
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/installers/synchronizers.py", line 329, in synchronize
    handlers[self_action](self_key)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/installers/synchronizers.py", line 148, in install_candidate
    installer.install(can)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/installers/installers.py", line 43, in install
    candidate.get_metadata(allow_all_wheels=False, raising=True)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/models/candidates.py", line 203, in get_metadata
    sdist = get_sdist(built)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/models/candidates.py", line 38, in get_sdist
    return EggInfoDistribution(egg_info) if egg_info else None
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/distlib/database.py", line 878, in __init__
    metadata = self._get_metadata(path)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/distlib/database.py", line 955, in _get_metadata
    requires = parse_requires_path(req_path)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/distlib/database.py", line 926, in parse_requires_path
    reqs = parse_requires_data(fp.read())
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/distlib/database.py", line 903, in parse_requires_data
    r = parse_requirement(line)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/distlib/util.py", line 193, in parse_requirement
    raise SyntaxError('Invalid URL: %s' % uri)
SyntaxError: Invalid URL: file:///Users/nicolas/Workspace/docparser/libs/B

The PDM version is 1.5.3

frostming · 2021-05-19T00:37:28Z

You should define your pyproject.toml as follows:

dependencies = [
    "B @ file:///${PROJECT_ROOT}/relative/path/to/B"
]

${PROJECT_ROOT} is a built-in variable you should keep as is.

AlexAegis · 2021-09-23T21:07:27Z

I took me a while to find this solution, I feel for one this variable should be better documented around dependency handling, and should be utilized when adding local dependencies with pdm add instead of the absolute path.

matejetz · 2022-04-14T09:34:34Z

This seems to not be working anymore.

dependencies = [
    "B @ file:///${PROJECT_ROOT}/../relative/path/to/B"
]

runs into [KeyError]:('B', '0.0.1', 'file:///${PROJECT_ROOT}/../relative/path/to/B', False)
with some debugging it looks like this occurs because the key tuple in the candidates dictionary has the resolved path

frostming · 2022-04-15T13:31:22Z

You can't refer to a package beyond the project root. Simon Matejetz ***@***.***> 于2022年4月14日周四 17:34写道：

…

This seems to not be working anymore. dependencies = [ "B @ file:///${PROJECT_ROOT}/../relative/path/to/B" ] runs into [KeyError]:('B', '0.0.1', 'file:///${PROJECT_ROOT}/../relative/path/to/B', False) with some debugging it looks like this occurs because the key tuple in the candidates dictionary has the resolved path — Reply to this email directly, view it on GitHub <#464 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD4UNXT6ASWWRDRMRKJ4AXDVE7RDJANCNFSM45CKAULQ> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

erikreppel · 2022-09-01T01:38:26Z

@frostming this seems like a fairly arbitrary limitation, allowing it would very much help monorepos with conflicting dependencies between apps where a single project root isnt viable. Poetry supports this with the following syntax:

[tool.poetry.dependencies]
python = "^3.9"
internalpkg = { path = "../internalpkg" }

jdoiro3 · 2022-10-06T15:13:14Z

When I use this method, run pdm build and then try to pip install the built wheel file I get the below. Is this expected? Also, let me know if I should open a new issue for this.

Processing ./dist/project--py3-none-any.whl
Processing /${PROJECT_ROOT}/local-package
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/${PROJECT_ROOT}/local-package'

frostming · 2022-10-07T03:00:17Z

${PROJECT_ROOT} is there for the convenience of project management, any local paths are not allowed in the wheel metadata. You must replace it with a published package name before you package and upload it to PyPI.

adriangb · 2022-10-07T03:03:41Z

Could there be a new type of dependency that causes the projects to be “merged” into one for the purposes of building a distribution instead of including one as a path dependency (which like you say doesn’t work)?

frostming · 2022-10-07T03:15:24Z

@adriangb why not put the local path dependencies under dev-dependencies which will not go into the wheel metadata?

adriangb · 2022-10-07T04:44:03Z

I don't think that works recursively right? That is, if A depends on B depends on C if I ask pdm to install A it will install A and B (because B is a dev dep of A) but not C.

jdoiro3 · 2022-10-07T12:11:05Z

@frostming this makes sense now. I'm still learning Python packaging and appreciate you taking the time to answer.

JohnHardy · 2022-10-10T21:06:28Z

@frostming We are using PDM for some projects that are not published to a repo, but build to an exe instead. Do you have a similar use case @erikreppel?

frostming · 2022-10-11T01:32:29Z

I don't think that works recursively right? That is, if A depends on B depends on C if I ask pdm to install A it will install A and B (because B is a dev dep of A) but not C.

No, it doesn't

adriangb · 2022-10-11T03:35:03Z

Fair enough. Would you accept a feature request for it (I can open an issue in more detail)? Poetry supports it and together with dependency groups it makes for a nice simple monorepo solution: https://github.com/adriangb/python-monorepo. If not, no worries.

pawamoy · 2022-10-11T08:48:15Z

@adriangb, here an alternative solution: create a "common-dev-deps" project/wheel that depends (production dependencies) on all your common development dependencies. Then dev-depend on this project/wheel in your different monorepo projects. This way they will stay in sync. Would that work for you? 🤔

adriangb · 2022-10-11T14:21:38Z

No, monorepo projects can depend on each other

pawamoy · 2022-10-11T14:35:36Z

OK, it seems I didn't quite understand your use-case 🙂
So, you have projects in a monorepo depending on each other using local paths for convenience, and you'd also like to build distributions for each project? I think previously it was possible to both use a public dependency as well as its editable version. PDM would use the local, editable one when installing deps locally, and the public version when building wheels. Editable dependencies are not allowed anymore in dependencies (only in dev-deps).

adriangb · 2022-10-11T15:10:00Z

So, you have projects in a monorepo depending on each other using local paths for convenience, and you'd also like to build distributions for each project?

Not exactly. I want to have local deps (e.g. lib, app (depends on lib) and cli (depends on cli) and build artifacts from them. For my personal use case I just need to build container images, so even if this only worked with pdm install ... that'd be fine for me. But building wheels (e.g. to publish lib as a public library) is also a pretty common use case, just not one that I have. For that use case doing the "path dependency locally, published version in wheel" makes sense, but again that's not my use case.

0xallie · 2022-10-20T16:24:08Z

${PROJECT_ROOT} is there for the convenience of project management, any local paths are not allowed in the wheel metadata. You must replace it with a published package name before you package and upload it to PyPI.

It seems like it's also impossible to use pip install on the pyproject.toml if ${PROJECT_ROOT} is present. I'm not trying to build or publish a package. Couldn't it be automatically resolved by pdm-pep517 in this case?

txchen · 2023-04-14T02:19:59Z

@adriangb Thanks for your git repo: https://github.com/adriangb/python-monorepo it is really helpful.
I am also trying to figure out how to set up monorepo for a python project, the pdm example in your repo seems not working. Have you figured out how to make it work?

adriangb · 2023-04-14T02:21:28Z

I gave up on PDM (because of #464 (comment)) and used Poetry. Particularly with python-poetry/poetry#6845 merged Poetry is shaping up pretty nicely to handle Python monorepos.

frostming · 2023-04-14T02:21:49Z

@txchen Here is one https://github.com/pdm-project/pdm-example-monorepo I set up recently

adriangb · 2023-04-14T02:23:27Z

That's cool @frostming ! I'll update my example repo.

adriangb · 2023-04-14T02:25:56Z

Could you clarify a couple of things?

Was this changed at some point?
How does pkg-first know how to locate pkg-core without some sort of of path (I would have expected something like ../pkg-core somewhere). Or is PDM doing some special handling because this is a monorepo?
Are there any docs for this in general?

frostming · 2023-04-14T02:30:26Z

Was this changed at some point?

No, dev-dependencies in the sub-packages are not visible by the resolvers since it won't be built into the package metadata.

How does pkg-first know how to locate pkg-core without some sort of of path

Because you resolve and install the dependencies in the top level project only¹ and it specifies editable relative path dependencies, which will take precedence and override the normal dependencies with the same name in the sub packages. This is the main rule of how monorepo work in PDM.

Are there any docs for this in general?

There is a simple one in the docs on the main branch: https://pdm.fming.dev/dev/usage/advanced/#use-pdm-to-manage-a-monorepo

It depends on your workflow, but I assume most people using monorepo don't resolve dependencies for sub packages independently. ↩

adriangb · 2023-04-14T02:33:49Z

No, dev-dependencies are not visible by the resolvers since it won't be built into the package metadata.

Just to make sure I'm understanding, is there any way to install pkg-first and have pkg-core be installed automatically? With the Poetry setup that works out of the box.

It depends on your workflow, but I assume most people using monorepo don't resolve dependencies for sub packages independently

👍🏻 I think you are right. The only situation where that makes sense is if you have subprojects with incompatible dependencies, but then you'd never be able to install the entire project at once, which defeats most of the purpose of a monorepo IMO.

frostming · 2023-04-14T02:42:08Z

is there any way to install pkg-first and have pkg-core be installed automatically?

pkg-first depends on pkg-core as you can see in pkg-first/pyproject.toml so it will work fine if both pkg-core and pkg-first are uploaded to PyPI. However when developing we want to install the local version of pkg-core and its relative path should also be written to the root pyproject.toml. We are planning on a workspace support so users don't need to specify the relative path dependencies themselves, you can regard it as a shortcut only, PDM has all the necessary abilities in the latest release.

The only situation where that makes sense is if you have subprojects with incompatible dependencies, but then you'd never be able to install the entire project at once, which defeats most of the purpose of a monorepo IMO.

Fortunately, 2.5.0 is also released with multiple lockfile support. You can create a lockfile for each incompatible dependency (sub)set and run tasks against it.

oleksii-symon-corva-ai · 2023-10-09T10:49:43Z

For those interested - working example of PDM monorepo https://github.com/Symas1/pdm-monorepo/tree/main
@frostming @adriangb

nhuray added the ⭐ enhancement Improvements for existing features label May 18, 2021

frostming closed this as completed May 25, 2021

BrutalSimplicity mentioned this issue Sep 12, 2022

Failure of Editable Installs from Relative Paths outside Project Root #1381

Closed

frostming mentioned this issue Oct 13, 2022

fix: Fix the support for relative paths from outside the project #1432

Merged

2 tasks

sarayourfriend mentioned this issue Apr 9, 2024

Implementation Plan: Switch Python package management away from Pipenv WordPress/openverse#286

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Local dependency and packaging #464

Local dependency and packaging #464

nhuray commented May 18, 2021

pawamoy commented May 18, 2021 •

edited

nhuray commented May 18, 2021 •

edited

pawamoy commented May 18, 2021

nhuray commented May 18, 2021

frostming commented May 19, 2021 •

edited

AlexAegis commented Sep 23, 2021

matejetz commented Apr 14, 2022

frostming commented Apr 15, 2022 via email

erikreppel commented Sep 1, 2022

jdoiro3 commented Oct 6, 2022

frostming commented Oct 7, 2022 •

edited

adriangb commented Oct 7, 2022

frostming commented Oct 7, 2022

adriangb commented Oct 7, 2022

jdoiro3 commented Oct 7, 2022

JohnHardy commented Oct 10, 2022 •

edited

frostming commented Oct 11, 2022

adriangb commented Oct 11, 2022 •

edited

pawamoy commented Oct 11, 2022 •

edited

adriangb commented Oct 11, 2022

pawamoy commented Oct 11, 2022

adriangb commented Oct 11, 2022 •

edited

0xallie commented Oct 20, 2022

txchen commented Apr 14, 2023

adriangb commented Apr 14, 2023 •

edited

frostming commented Apr 14, 2023 •

edited

adriangb commented Apr 14, 2023

adriangb commented Apr 14, 2023

frostming commented Apr 14, 2023 •

edited

adriangb commented Apr 14, 2023 •

edited

frostming commented Apr 14, 2023 •

edited

oleksii-symon-corva-ai commented Oct 9, 2023

Local dependency and packaging #464

Local dependency and packaging #464

Comments

nhuray commented May 18, 2021

Is your feature request related to a problem? Please describe.

Add B as a local dependency into A project

Install A

Describe the solution you'd like

pawamoy commented May 18, 2021 • edited

nhuray commented May 18, 2021 • edited

pawamoy commented May 18, 2021

nhuray commented May 18, 2021

frostming commented May 19, 2021 • edited

AlexAegis commented Sep 23, 2021

matejetz commented Apr 14, 2022

frostming commented Apr 15, 2022 via email

erikreppel commented Sep 1, 2022

jdoiro3 commented Oct 6, 2022

frostming commented Oct 7, 2022 • edited

adriangb commented Oct 7, 2022

frostming commented Oct 7, 2022

adriangb commented Oct 7, 2022

jdoiro3 commented Oct 7, 2022

JohnHardy commented Oct 10, 2022 • edited

frostming commented Oct 11, 2022

adriangb commented Oct 11, 2022 • edited

pawamoy commented Oct 11, 2022 • edited

adriangb commented Oct 11, 2022

pawamoy commented Oct 11, 2022

adriangb commented Oct 11, 2022 • edited

0xallie commented Oct 20, 2022

txchen commented Apr 14, 2023

adriangb commented Apr 14, 2023 • edited

frostming commented Apr 14, 2023 • edited

adriangb commented Apr 14, 2023

adriangb commented Apr 14, 2023

frostming commented Apr 14, 2023 • edited

Footnotes

adriangb commented Apr 14, 2023 • edited

frostming commented Apr 14, 2023 • edited

oleksii-symon-corva-ai commented Oct 9, 2023

pawamoy commented May 18, 2021 •

edited

nhuray commented May 18, 2021 •

edited

frostming commented May 19, 2021 •

edited

frostming commented Oct 7, 2022 •

edited

JohnHardy commented Oct 10, 2022 •

edited

adriangb commented Oct 11, 2022 •

edited

pawamoy commented Oct 11, 2022 •

edited

adriangb commented Oct 11, 2022 •

edited

adriangb commented Apr 14, 2023 •

edited

frostming commented Apr 14, 2023 •

edited

frostming commented Apr 14, 2023 •

edited

adriangb commented Apr 14, 2023 •

edited

frostming commented Apr 14, 2023 •

edited