[BUG] package-dir with subpackages (namespaces) does not work with editable install

[DavidRV00]

setuptools version

setuptools==69.1.0

Python version

Python 3.12.2

OS

Artix Linux

Additional environment information

I'm running this in a fresh conda environment.

Description

I have the following slightly odd directory structure:

pyproject.toml
src
- my_package
- - my_module.py
src2
- my_package2
- - my_module2.py

And this clause in my pyproject.toml:

[tool.setuptools.package-dir]
"different_name" = "src/my_package"
"different_name.subpkg" = "src2/my_package2"

After running pip install . (ie non-editable), I'm able to import modules like this:

from different_name import my_module
from different_name.subpkg import my_module2

... but with pip install -e . (editable), the second import fails with:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'different_name.subpkg'

Expected behavior

I expect the import behavior to be the same whether the package is installed editable or not.

How to Reproduce

1. git clone https://github.com/DavidRV00/package-dir-editable and cd inside.
2. Create a virtual environment (I use conda create -n package-dir-editable python=3) and activate it.
3. pip install -e . inside the project dir.
4. python -c "from different_name.subpkg import my_module2"

Output

❯ pip install -e .

Obtaining file:///home/david/projects/package-dir-editable
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: package-dir-editable
  Building editable for package-dir-editable (pyproject.toml) ... done
  Created wheel for package-dir-editable: filename=package_dir_editable-0.0.1-0.editable-py3-none-any.whl size=3548 sha256=7754770e4c45aeed608f47517905691f06c62b31638ed8614815d60564a29f72
  Stored in directory: /tmp/pip-ephem-wheel-cache-6sz16frl/wheels/9f/30/48/30cb3e3e3b5268cb75fe782edf9c59670e56859d2cc847d60f
Successfully built package-dir-editable
Installing collected packages: package-dir-editable
Successfully installed package-dir-editable-0.0.1

❯ python -c "from different_name import my_module" # works fine

❯ python -c "from different_name.subpkg import my_module2"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'different_name.subpkg'

[abravalheri]

Hi @DavidRV00 , what is the configuration in your packages field?

Sorry, I just saw that you provided a github repository example.

I believe that to make it work you need to set the tool.setuptools.packages field with a list of all packages and subpackages. Because your package structure deviates from the 2 conventional styles we support natively (vanilla src- and flat-layout, without directory name remapping), the auto discovery is limited.

Can you please remove all the cache (by rm-ing the build, dist and any .egg-info folders), try doing the change I pointed above in pyproject.toml, and let me know if it works?

[DavidRV00]

Hi @abravalheri ,

I want to avoid explicitly listing the packages, because auto-discovery seems to work great, at least for the non-editable install; seems like it's really close to being able to handle it. But if the issue is just that what I'm doing isn't fully supported, then I think we can close this as not-a-bug. Maybe I can make it work with packages.find.

[abravalheri]

Hi @DavidRV00 , with a customisation like that, auto-discovery is not supposed to work.

It is probably just working by accident...

[DavidRV00]

Alright, thanks for letting me know. I'll see if I can find some other way to make what I'm trying to do work.

[abravalheri]

Hi @DavidRV00, thank you very much for the feedback.

I will close this issue for now, because the customisation of package-dir does require explicitly listing packages for everything to work fine. (The only situations where packages can be safely auto-discovered are vanilla src-layout and vanilla flat-layout, i.e. no directory re-mapping).

[DavidRV00]

Hello again @abravalheri ,

commenting to re-open, because now I've tried out using an explicit list of packages, but I get the same result as before with editable installs.

This is what I've added to pyproject.toml:

[tool.setuptools]
packages = [
	"different_name",
	"different_name.subpkg",
]

Again pip install . works fine, but trying to import the subpkg after pip install -e . fails.

Here, again, is the log output from pip install -e .:

(package-dir-editable) package-dir-editable ❯ pip install -e .
Obtaining file:///home/david/projects/package-dir-editable
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: package-dir-editable
  Building editable for package-dir-editable (pyproject.toml) ... done
  Created wheel for package-dir-editable: filename=package_dir_editable-0.0.1-0.editable-py3-none-any.whl size=3548 sha256=6c6c2bbedd1e41a8e0eb18574447f13af5fcb81b75673d07fe3eff32d8cbfd42
  Stored in directory: /tmp/pip-ephem-wheel-cache-9mokx92b/wheels/9f/30/48/30cb3e3e3b5268cb75fe782edf9c59670e56859d2cc847d60f
Successfully built package-dir-editable
Installing collected packages: package-dir-editable
Successfully installed package-dir-editable-0.0.1

Reproducible example

1. Either pull and switch branches on the github repo above to branch explicit-packages, or just run git clone -b explicit-packages https://github.com/DavidRV00/package-dir-editable to get it directly.
2. cd inside, then pip install -e .
3. python -c "from different_name.subpkg import my_module2" (you should see this fail)

Hopefully this is still just a case of me doing something unsupported, but I'm not sure what it could be.

Thanks again for any help!

[abravalheri]

Hi @DavidRV00, thank you for the reproducer.

This seems to be a tricky one.

If I do:

>>> import __editable___package_dir_editable_0_0_1_finder as ed
>>> ed._EditableNamespaceFinder.find_spec("different_name.subpkg")
ModuleSpec(name='different_name.subpkg', loader=None, submodule_search_locations=['/tmp/package-dir-editable/src2/my_package2'])
>>> ed._EditableFinder.find_spec("different_name.subpkg.my_module2")
ModuleSpec(name='different_name.subpkg.my_module2', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fcd914fd490>, origin='/tmp/package-dir-editable/src2/my_package2/my_module2.py')

###

>>> import sys
>>> sys.meta_path
[<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>, <class '__editable___package_dir_editable_0_0_1_finder._EditableFinder'>]
>>> sys.path_hooks
[<class 'zipimport.zipimporter'>, <function FileFinder.path_hook.<locals>.path_hook_for_FileFinder at 0x7fcd917245e0>, <bound method _EditableNamespaceFinder._path_hook of <class '__editable___package_dir_editable_0_0_1_finder._EditableNamespaceFinder'>>]
>>> sys.path
['', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/tmp/package-dir-editable/.venv/lib/python3.12/site-packages', '__editable__.package_dir_editable-0.0.1.finder.__path_hook__']
>>> from different_name.subpkg import my_module2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'different_name.subpkg'

I can see that the custom finders can successfully locate the packages themselves... But somehow they don't seem to work effectively togheter with the import machinery.

This might be a very tricky problem to solve (and take time research, debugging).
Related to python/cpython#92054.

If anyone in the community would like to provide a PR, that would be much appreciated.

[DavidRV00]

Thanks for the quick reply, and for looking into it so quick!

FWIW, for a little context, the reason I want this kind of directory structure is that I'm trying to put together a system for writing hybrid Python/C packages using just setuptools, CMake, and SWIG. CMake tends to generate code in parallel directory trees that mirror the src tree, so I end up with directories that look like src/a/b/c... , build/src/a/b/c.... The challenge is to find a nice way to import the SWIG-generated python interfaces under the build dir, homogenously with the regular python packages under the src dir, hence the remapping.

[abravalheri]

Maybe as a workaround while this issue is not solved, try adding __init__.py to the directories?

Namespaces are notably more complex to handle in custom finders (and that is related to the advice I asked in python/cpython#92054).

[DavidRV00]

Great, using __init__.py's does make it work. That'll be a fine workaround for now, thanks.

[DavidRV00]

Another note if it's useful: Using pip install -e . --config-settings editable_mode=strict also makes it work.