Support multiple source roots

[vlaci]

In a monorepo, top-level packages are defined like this:

.
├── package_1_root
│   ├── package_1  # python source lives here
│   ... # possibly additional data directories
│   └── tests
├── package_2_root
│   ├── package_2 
│   └── tests
...
└── package_n_root
    ├── package_n 
    └── tests

The source is split into subdirectories, but the whole repository consists of a single python package. This layout can be achieved with e.g. poetry:

[tool.poetry]
name = "monorepo"
packages = [
    { include = "package_1", from = "package_1_root" },
    { include = "package_2", from = "package_2_root" },
#   ...
    { include = "package_n", from = "package_n_root" },
]

Then these packages can be used like this from python code:

>>> import package_1.bar
>>> package_1.bar
<module 'package_1.bar' from '/.../package_1_root/package_1/bar/__init__.py'>

I need to set source_root to e.g. package_1, because otherwise tach thinks that the package_<n>_root is part of the module path.

I'd like to enforce dependencies in and between all of these python_<n> packages, so it would be great to be able to tell tach that individual packages can reside under different prefixes.

[dhananjaypai08]

Wouldn't we much rather prefer package_1_root, package_2_root, .... package_n_root to be n different modules inside one single package folder that can act as a source root. Having multiple <source_root> will only become complex for the end developers using tach integrations as it would have n different tach.yml but on a broader scope this can include defining <source_root> in multiple packages but only checking/syncing those source roots that the user wants

[emdoyle]

In a monorepo, top-level packages are defined like this:

.
├── package_1_root
│   ├── package_1  # python source lives here
│   ... # possibly additional data directories
│   └── tests
├── package_2_root
│   ├── package_2 
│   └── tests
...
└── package_n_root
    ├── package_n 
    └── tests

The source is split into subdirectories, but the whole repository consists of a single python package. This layout can be achieved with e.g. poetry:

[tool.poetry]
name = "monorepo"
packages = [
    { include = "package_1", from = "package_1_root" },
    { include = "package_2", from = "package_2_root" },
#   ...
    { include = "package_n", from = "package_n_root" },
]

Then these packages can be used like this from python code:

>>> import package_1.bar
>>> package_1.bar
<module 'package_1.bar' from '/.../package_1_root/package_1/bar/__init__.py'>

I need to set source_root to e.g. package_1, because otherwise tach thinks that the package_<n>_root is part of the module path.

I'd like to enforce dependencies in and between all of these python_<n> packages, so it would be great to be able to tell tach that individual packages can reside under different prefixes.

I think this use case makes sense. Do you essentially use poetry to build and run everything as if it were a single package? And do all of your other tools work with this structure?

Supporting multiple source roots might require significant refactoring, but I think it's probably common so I can sketch something out.

@dhananjaypai08 I think since in this case it is a single logical Python package, Tach should be able to support a single 'check/sync' across all the source roots. It seems like it is just a matter of understanding the import paths correctly since it doesn't quite match the filesystem.

[vlaci]

I think this use case makes sense. Do you essentially use poetry to build and run everything as if it were a single package? And do all of your other tools work with this structure?

Yes, this is a single package with a single dependency specification.
Occasionally some extra configuration is needed. E.g. I need to configure Pyright's extraPath configuration option¹.

Footnotes

1. https://github.com/microsoft/pyright/blob/main/docs/configuration.md ↩

[gwdekker]

I just discovered tach and love it! It would be great if it would support monorepo setups.

I want to provide a concrete example of a monorepo setup to help scoping this feature request.

We are using a polylith based setup. An example repo is posted here: https://github.com/DavidVujic/python-polylith-example-rye

tach is already very useful: when I point the source root at components it works as expected for everything in components. It would be awesome if we could define multiple roots, in this case probably components and bases.

Congratulations on building such a cool tool!

[emdoyle]

This is now supported natively in 0.9.0!
https://docs.gauge.sh/usage/configuration#source-roots

Marking each source root with tach mod should automatically give the expected behavior. Let us know if you hit any issues!

cc: @vlaci @gwdekker

[vlaci]

That's awesome!

One thing I don't understand currently, how it deals with common module names that are present in multiple source roots, e.g. if there is an utils under two roots will its usage be merged? If so it looks like an error.

[vlaci]

Sorry, I was holding it wrong: they'll be prefixed by the root module name (obviously 🤦 )

It works great!

[gwdekker]

Tried it out for our use case, and it works perfectly. Thanks a lot for the quick turnaround!

[caelean]

@vlaci @gwdekker would love to learn more about your usecases if you're open to chatting 😄 are either of you in our Discord?