Users can no longer rely on sys.path side-effects with editable installs

[wesk]

setuptools version

67.6.1

Python version

3.8

OS

Ubuntu and Mac

Additional environment information

No response

Description

Context: I'm one of the maintainers of Syne Tune: https://github.com/awslabs/syne-tune/

Our repository has a top-level examples/ folder that has example scripts that demonstrate our library's functionality. We do not include the examples/ folder in our pip-installable bundle. Today the way we recommend that people run the example scripts is to install Syne Tune in editable mode using python -m pip install -e .. Then just run python examples/<EXAMPLE_SCRIPT.py> and it works fine.

The problem is that with the release of pip 23.1 two days ago, this install pattern stopped working. This broke my team's CI, see for example:

1. https://github.com/awslabs/syne-tune/actions/runs/4718647382?pr=625
2. https://github.com/awslabs/syne-tune/actions/runs/4716666731/jobs/8369933296?pr=624

We got import errors like the following:

Traceback (most recent call last):
  File "/Users/jkkndr/code/repositories/syne-tune/examples/launch_tensorboard_example.py", line 35, in <module>
    from examples.training_scripts.height_example.train_height import (
ModuleNotFoundError: No module named 'examples'

I opened an issue in the pip repository: pypa/pip#11972, and as a temporary mitigation they recommended installing the wheel package, which succeeds in mitigating the issue, see mitigation PR that we merged: syne-tune/syne-tune#626

However, as a better root-cause fix, I'm hoping you can comment on the following pypa/pip#11972 (comment):

[...] This is effectively a setuptools change in behaviour, so you would be best explaining your use case to them and asking for advice. I don't know if they provide any advice or documentation on how to transition from "traditional" editable installs to PEP 660-based ones. I imagine they would say that you should either include the examples in your package, or provide the examples as a separate project that the users can install normally. Or maybe just don't use editable installs and manually modify sys.path to put the examples on the import path.

What is the most correct / long-term way to fix this issue?

Expected behavior

I expect to be able to:

1. Install our library in editable mode
2. Run scripts that import modules from top-level folders in our repository that are not present in the bundle that we publish to PyPi
3. Not need to install any additional dependencies in order to do this (e.g. wheel)

How to Reproduce

git clone git@github.com:awslabs/syne-tune.git
cd syne-tune
python3 -m venv st_venv
. st_venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[extra]'
python examples/launch_tensorboard_example.py 

Note that the behavior only starts with pip 23.1, released two days ago. The entry in their CHANGELOG / release notes that describes this behavior change is here:

When the wheel package is not installed, pip now uses the default build backend instead of setup.py install for project without pyproject.toml. (pypa/pip#8559)

Which is in turn picking up the updated setuptools behavior.

Output

% python examples/launch_tensorboard_example.py 

Traceback (most recent call last):
  File "/Users/jkkndr/code/repositories/syne-tune/examples/launch_tensorboard_example.py", line 35, in <module>
    from examples.training_scripts.height_example.train_height import (
ModuleNotFoundError: No module named 'examples'

[pradyunsg]

(FYI: This issue/concern was identified/raised because the latest pip uses the build-backend hooks rather than setup.py based editable installs more often)

[abravalheri]

Hi @wesk, thank you very much for the ticket.

If I understood correctly there are 2 aspects in play here:

1. When someone runs a script with python path/to/script.py, Python will automatically add the directory that contains the script to sys.path, but not CWD:

If the script name refers directly to a Python file, the directory containing that file is added to the start of sys.path, and the file is executed as the __main__ module.

2. You also said that your package configuration does not include include the examples/ folder in our pip-installable bundle, this means that the examples folder is not installed as a package in a regular installation.

In recent versions of setuptools we improved the editable installation mode to behave more closely to the way a regular installation would happen. If a folder is not included in the regular installation we do our best to not make it available in the editable installation too.

In the end of the day, there is no clear indication here why the examples folder should be importable, and the error that you are seeing seems to be compatible with that.

My conclusion is that there seems to be no action point on the setuptools side, it is only doing its best to make the editable installation as close as possible to the regular installation (while still allowing the Python files to be modified incrementally).

If you would like to retain the "add everything to sys.path" behaviour, please consider using --config-settings editable_mode=compat with pip. You can see more details on the docs: https://setuptools.pypa.io/en/latest/userguide/development_mode.html.

Otherwise, you can use other sys.path manipulation or import techniques to make sure that the examples package is importable at runtime in your scripts.

[abravalheri]

If manipulating sys.path directly or using --config-settings editable_mode=compat in the frontend/installer is too inconvenient you can also change they way you use imports in your examples.

If you use relative imports in the scripts inside the examples folder and advise the users to run

python -m examples.EXAMPLE_SCRIPT
# instead of python examples/EXAMPLE_SCRIPT.py

You probably can take advantage of Python automatically inserting the working directory into sys.path.

This documentation has more details on how the different ways of running the Python command will interfere with sys.path.

[abravalheri]

I will go ahead and close this issue for now since behaving more like the regular installation and get rid of accidental behaviour was a goal when implementing PEP 660 in setuptools.

[brupelo]

@abravalheri Firstly, I appreciate the tip regarding --config-settings editable_mode=compat. It's been immensely helpful, especially considering the recent breaking change in pip. Previously, I relied on utilizing sys.path within my custom plugins to facilitate IDE autocompletion. However, with this new pip functionality, I'm grappling with how to extract import paths from file paths within the virtualenv context. Any insights or suggestions on adapting to this change would be greatly appreciated. Thanks again for sharing your knowledge.

[abravalheri]

Hi @brupelo, I am not 100% about which change in pip you are referring to.

If the concern is maximum compatibility with IDE autocompletion, I imagine that the best is to design you project directory structure to be a vanilla src-layout, so that setuptools is compelled to use a text-only .pth file.