Is there a canonical way to organize a single repo with multiple subprojects?

[rohit507]

Since creating #1059 I've been experimenting with multiple ways to host multiple subprojects within a single git repo. In particular, I've wanted to have the following:

• Multiple subprojects with dependencies on each other.
• The ability to use those subprojects in editable mode.
• Internal references (esp. in pyproject.toml) that work out of the box after a clone.
• A way for external projects, outside of the repo, to depend on individual subprojects.
  • The assumption is that external projects will have access to this repo via a git submodule.

My current setup has four subprojects A,B,C, and D. B, C, and D all depend on A. Additionally D depends on C. I want external projects to be able to import A, C, and D individually or in combination. Finally all of the project have CLIs that need to be accessible.

So far I've tried a few possible solutions:

Subprojects in a shared parent directory with relative dependencies.

Setup:

<repo-root>
  |-- A/
  |   `-- ...
  |
  |-- B/
  |   `-- ...
  |
  |-- C/
  |   `-- ...
  |
  `-- D/
      `-- ...

With a dependencies like "-e file:///${PROJECT_ROOT}/../A" to the pyproject.toml of B-D.

Issues:

• Functions like pdm add don't support this form and will resolve to an absolute path if you try.
• Installing a project with a path from the parent of PROJECT_ROOT fails with KeyError. #1059 makes it clear that this isn't supported.

Using subdirectories and symlinks to connect subprojects.

Setup:

<repo-root>
  |-- A/
  |   `-- ...
  |
  |-- B/
  |   |-- deps/
  |   |   `-- A/ --> ../../A/ 
  |   `-- ...
  |
  |-- C/
  |   |-- deps/
  |   |   `-- A/ --> ../../A/ 
  |   `-- ...
  |
  `-- D/
      |-- deps/
      |   |-- A/ --> ../../A/ 
      |   `-- C/ --> ../../C/ 
      `-- ...

With pdm add -e ./deps/A to add "-e file:///${PROJECT_ROOT}/deps/A" to pyproject.toml.

Issues:

• Is very fragile to path naming. If C/deps/ is renamed C/dependencies/ then cd D; pdm install -d
  complains about there being no D/dependencies/A/ directory.
• This confusion about ${PROJECT_ROOT} also appears when an external project depends on one of
  the subprojects.

For now, I've just thrown up my hands and piled everything into a single project even if it means a bunch of mostly orthogonal code lives together and forces the installation of a number of unnecessary dependencies.

Is there an intended way for PDM to support projects like this? Ideally one that lets me keep things mostly modular without having to use 15 separate git submodules and manage updating them in lockstep.

[frostming]

You may want to try monas, which I created as a monorepo solution. It supports not only pdm, but also some common backends setuptools, flit and hatchling

[rohit507]

Good to know, I'll give it a try.

Also sorry about creating this as an issue, I forgot discussions existed.

[Hnasar]

I'd be curious to hear what kind of solution you find!