Editable installs are broken after __editable__ and __path_hook__ changes

[abhinavsingh]

Description

I am starting to observe broken editable installs since past week.

Specifically, editable install now creates a __editable__.<pkg>.pth and __editable___<pkg>_finder.py files. These paths are returned as part of namespace.__path__, which has broken a lot of tooling.

1. pdoc3 went broke. Skipping __editable__ package seems to fix (hack) it.
   • See Skip __editable__ paths during iter_modules pdoc3/pdoc#408
2. We also noticed, namespace packages that installs a CLI are now also broken, because while the binary is under .venv/bin, it is unable to import the modules since it was installed using -e.
   • Modifying PYTHONPATH is the only option.
3. We also noticed, due to these changes even VSCode/Pylance is now broken, as it is unable to resolve paths to editable installed packages.
   • Adding "python.analysis.extraPaths" to .vscode/settings.json is the way out

I was unable to find any search result on Google for __editable__ and __path_hook__. So, I am unsure what I am up against here.

Expected behavior

1. Installed CLI from namespace package must work fine without modification of PYTHONPATH
2. namespace.__path__ must return actual path instead of __editable__.* which seems to break a lot of existing tooling

pip version

22.2.2

Python version

3.10.5

OS

MacOS 12.5

How to Reproduce

1. Create a namespace package that installs CLI (entry point)
2. Install the package with -e
3. Installed CLI is broken due to missing PYTHONPATH

Our projects are using pyproject.toml and setup.cfg.

Output

`ImportError: cannot import name 'entry_point' from 'namespace.cli' (unknown location)`

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

This is likely because setuptools released a new mechanism for implementing editable installs, as part of their PEP600 support. The pdoc3 project is probably not yet aware of the new mechanism, and will need updating.

There isn’t really anything for pip to do here, so I’m closing this issue. If it turns out that there is a pip problem here, feel free to reopen it.

[abhinavsingh]

@pfmoore This is likely going to create a lot of havoc. Editable install is how people operate in Python community. And if all of a sudden, entire tooling around editable install is going to break, then likely we must be planning better around it.

My 2 cents. May be, instead of enforcing the new mechanism as default, this could have been an opt in to start with. Instead of fighting this out, we decided to simply pin setuptools "setuptools <= 62.6.0". While this has put us out of misery, we are now stuck in the past. FWIW, new mechanism has decided to return a Path which returns in __path_hook__ as part of namespace. While that file actually doesn't exist on the disk.

[pradyunsg]

Please reach out to the setuptools maintainers. While various Python packaging tools do interoperate, the behaviour changes you’re seeing come from setuptools.

The maintainers of pip don’t control / maintain setuptools, which is where I recommend you reach out.

[pradyunsg]

I can actually transfer the issue, so… here you go. :)

[potiuk]

It looks like we started to have the same problem in Airflow.

• pip install -e . after checking out basically any version (including main) and running any Airlfow cli command results in

Traceback (most recent call last):
  File "/home/jarek/.pyenv/versions/airflow-test-cli3/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/jarek/code/airflow/airflow/__main__.py", line 38, in main
    args.func(args)
  File "/home/jarek/code/airflow/airflow/cli/cli_parser.py", line 50, in command
    func = import_string(import_path)
  File "/home/jarek/code/airflow/airflow/utils/module_loading.py", line 32, in import_string
    module = import_module(module_path)
  File "/home/jarek/.pyenv/versions/3.9.13/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/jarek/code/airflow/airflow/cli/commands/standalone_command.py", line 35, in <module>
    from airflow.www.app import cached_app
  File "/home/jarek/code/airflow/airflow/www/app.py", line 43, in <module>
    from airflow.www.extensions.init_views import (
  File "/home/jarek/code/airflow/airflow/www/extensions/init_views.py", line 28, in <module>
    from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
  File "/home/jarek/code/airflow/airflow/www/views.py", line 685, in <module>
    class AirflowBaseView(BaseView):
  File "/home/jarek/code/airflow/airflow/www/views.py", line 699, in AirflowBaseView
    extra_args['sqlite_warning'] = settings.engine.dialect.name == 'sqlite'
AttributeError: module 'airflow.settings' has no attribute 'engine'

This started to happen few days ago and adding Airlfow's source to PYTHONPATH solves the problem.

I did some bisecting and pin-pointed it to version 64.0.0 of setuptools. Version 63.4.3 worked perfectly fine, 64.0.0 breaks it. The reason seems to be as described in the issue. I've added those lines to our pyproject.toml and they solved the problem:

[build-system]
requires = ['setuptools==63.4.3']
build-backend = "setuptools.build_meta"

See: apache/airflow#25848

It is super-easy to reproduce now:

1. checkout main from https://github.com/apache/airflow
2. create and activate a new virtualenv
3. run pip install -e .
4. run airflow standalone

This works perfectly well (because of the 63.4.3 limit)

5. change the line in pyproject.toml to:

requires = ['setuptools==64.0.0']

6. create and activate a new virtualenv
7. run pip install -e .
8. run airflow standalone

You see the stack-trace above

[potiuk]

(The error is also present in all other 64.* and 65* versions BTW).

[abravalheri]

Hi @abhinavsingh,

Could you provide a minimal reproducer explaining more in details the issues you are having? (Maybe it is worthy to split multiple issues in multiple reproducers).

I tried to investigate what you are describing with the following reproducer, but I was not able to find the problematic behaviour you are referring to... (Maybe I understood something wrong? Or maybe the reproducer is missing something? Or -- best case scenario -- the problem might have already gotten solved in the latest release?).

# docker run --rm -it python:3.10.6 bash

cd /tmp && rm -rf /tmp/proj_root
mkdir -p /tmp/proj_root
mkdir -p /tmp/proj_root/namespace/cli
echo "def run(): print('hello world')" > /tmp/proj_root/namespace/cli/__init__.py
cat <<EOF > /tmp/proj_root/pyproject.toml
[build-system]
requires = ["setuptools>=65.1.0"]
build-backend = "setuptools.build_meta"

[project]
name = "namespace.cli"
version = "42"

[project.scripts]
namespace-cli = "namespace.cli:run"
EOF
cd /tmp/proj_root
python3.10 -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e .
cd /var
/tmp/proj_root/.venv/bin/python -c 'import namespace; print(f"{namespace.__path__=}")'
# ==> namespace.__path__=_NamespacePath(['/tmp/proj_root/namespace'])
/tmp/proj_root/.venv/bin/namespace-cli
# ==> hello world

As you can see above, the reproducer show that whenever possible setuptools will add the corresponding path entry to __path__.
Cases when this does not occur are likely associated with configurations by which users explicitly opt out from certain packages or there is simply no directory in the file system for the package to be associated with...

---

There are a few other remarks I would like to make:

• In Python, path entries do not need to be existing locations in the file system, this is mentioned in Python docs:

  Path entries need not be limited to file system locations. They can refer to URLs, database queries, or any other location that can be specified as a string.
  ...
  Most path entries name locations in the file system, but they need not be limited to this.
  ...
  Entries in sys.path can name directories on the file system, zip files, and potentially other “locations” (see the site module) that should be searched for modules, such as URLs, or database queries.

  We can also demonstrate this with the following snippet:

  # docker run --rm -it python:3.10.6 bash
  python3.10 -c 'import sys; print(sys.path)'
  # ==> ['', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages']
  ls -la /usr/local/lib/python310.zip
  # ==> ls: cannot access '/usr/local/lib/python310.zip': No such file or directory

• Most static analysis tools (including the ones used in code editors) have limitations in terms of aspects of the Python language they cannot handle. Import hooks (which have been part of the Python language specification for years, and are listed in PEP 660 as a perfectly viable implementation for editable installs), are notably one of these limitations. This is reported in Due to limitation, static analysis don't support import hook used in editable install v64+ #3518.
  Setuptools understand this limitation, and I am personally happy to contribute to an effort of improving this. However we do need someone to champion a design/architecture/standard that is well received in both packaging and static analysis community. I recommend anyone interested in pushing this forward to engage in the mailing list discussion, or on the thread on discourse.

• I would also recommend setuptools' docs on editable installs. There might be some useful information there.

[abravalheri]

@potiuk I suspect airflow's use case is different.

By having a look on the setup.py, it seems that airflow implements a custom develop command.

Setuptools's implementation of PEP 660 does not use the develop command¹. Instead, setuptools now rely in a complete different command. Please have a look on the docs for a few possible options on how to customise the editable installation.

---

By running:

# docker run --rm -it python:3.10.6 bash

apt update && apt install -y git build-essential
cd /tmp
git clone https://github.com/apache/airflow
cd /tmp/airflow
sed -i 's/==63.4.3/>=65.1.0/g' pyproject.toml
python -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e .
cd /var
/tmp/airflow/.venv/bin/python -c 'import airflow.settings; print(dir(airflow.settings))'
# ==> ['AIRFLOW_HOME', 'AIRFLOW_MOVED_TABLE_PREFIX', 'ALLOW_FUTURE_EXEC_DATES', 'CAN_FORK', 'CHECK_SLAS', 'COMPRESS_SERIALIZED_DAGS', 'Callable', 'DAEMON_UMASK', 'DAGS_FOLDER', 'DASHBOARD_UIALERTS', 'DEFAULT_ENGINE_ARGS', 'DONOT_MODIFY_HANDLERS', 'EXECUTE_TASKS_NEW_PYTHON_INTERPRETER', 'Engine', 'GUNICORN_WORKER_READY_PREFIX', 'HEADER', 'HIDE_SENSITIVE_VAR_CONN_FIELDS', 'IS_K8S_OR_K8SCELERY_EXECUTOR', 'KILOBYTE', 'LAZY_LOAD_PLUGINS', 'LAZY_LOAD_PROVIDERS', 'LOGGING_CLASS_PATH', 'LOGGING_LEVEL', 'LOG_FORMAT', 'List', 'MASK_SECRETS_IN_LOGS', 'MEGABYTE', 'MIN_SERIALIZED_DAG_FETCH_INTERVAL', 'MIN_SERIALIZED_DAG_UPDATE_INTERVAL', 'NullPool', 'Optional', 'PLUGINS_FOLDER', 'SASession', 'SIMPLE_LOG_FORMAT', 'SQL_ALCHEMY_CONN', 'STATE_COLORS', 'TIMEZONE', 'TYPE_CHECKING', 'USE_JOB_SCHEDULE', 'Union', 'WEBSERVER_CONFIG', 'WEB_COLORS', '__annotations__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_get_rich_console', 'atexit', 'conf', 'configure_action_logging', 'configure_adapters', 'configure_logging', 'configure_orm', 'configure_vars', 'create_engine', 'custom_show_warning', 'dag_policy', 'dispose_orm', 'exc', 'executor_constants', 'functools', 'get_airflow_context_vars', 'get_dagbag_import_timeout', 'get_session_lifetime_config', 'import_local_settings', 'initialize', 'json', 'log', 'logging', 'original_show_warning', 'os', 'pendulum', 'pod_mutation_hook', 'prepare_engine_args', 'prepare_syspath', 'reconfigure_orm', 'replace_showwarning', 'scoped_session', 'sessionmaker', 'setup_event_handlers', 'sqlalchemy', 'sys', 'task_instance_mutation_hook', 'task_policy', 'tz', 'validate_session', 'warnings']

I can see that the package is indeed installed in the editable mode, however the engine attribute does not seem to be initialised. Maybe this is related to the customisations you guys are doing in the develop command, but that are no longer invoked when you run pip install -e .?

---

Footnotes

1. The PEP 660 standard is incompatible with the develop command. In the later, setuptools writes files directly to the installation directories, but the PEP 660 standard requires setuptools to handle a .whl file to pip, so pip can take care of creating files. ↩

[potiuk]

By having a look on the setup.py, it seems that airflow implements a custom develop command.

Thanks for the pointer @abravalheri . I will take a closer look and see if I can work it out. I was kinda suspecting that we could have been doing something that PEP 660 does not really like :). I will keep you posted.

[potiuk]

Time to do some more deep PEP reading

[ofek]

@potiuk Hey! Airflow's setup.py looks quite complex which intrigues me. Would you be interested in me opening a PR for a PoC switch to Hatchling (which will fix your issue)?

[abravalheri]

(Copying some opinion from the PyPA discord for eventual readers of this thread).

setup.py implementations tend to grow complex with time. Any backend that supports extensions can be used to refactor (which includes setuptools). Documentation on custom build steps can be found at https://setuptools.pypa.io/en/latest/userguide/extension.html.

[potiuk]

@potiuk Hey! Airflow's setup.py looks quite complex which intrigues me. Would you be interested in me opening a PR for a PoC switch to Hatchling (which will fix your issue)?

@ofek Airlfow is an ASF project and community makes decisions not me personally. If you want to make a POC and have some good arguments, you will have to start dicsussion at the airflow devlist https://airflow.apache.org/community/ . But you have to - in general - have a good reasoning and justification for such a change in build system. just fixing an error that we can fix without switching to another build mechanism is not good enough reason.

[abravalheri]

Hi @lonetwin, pkgutil.iter_modules does not cover the same spectrum as the importlib machinery: it is much more limited.

Based on this comment I would say that (so far) the stdlib developers have no intention the increase the scope or add features to pkgutil. The phrase used was: pkgutil is on its way out as soon as people have the time to deprecate it.

If you absolutely need to use pkgutil you can try to opt into a different editable installation mode (e.g. pip install -e . --config-settings editable_mode=strict) -- I haven't tested it myself, so I am not sure it will work.

[pradyunsg]

In the spirit of over-communicating: I'm unsubscribing from this since I don't think there's anything for me to do here. To the maintainers, please feel free to @-mention me in case there's something here that's on pip's end.

[abhinavsingh]

@abravalheri Apologies, haven't checked back on GitHub in weeks. Missed your message altogether. A point to note in our case is that, we have all our packages under a namespace. As others have figured, v64 is an issue. At our end, I had to pin setup tools to v62 to let our company workflows pass. In our setup, we simply go by paths returned by namespace.__path__. But ever since v64, we realise, __path__ contains entries which are actually not real file paths. They needs to be resolved using latest mechanism adopted by setup tools. The day I ran into it, I wasn't even able to find any formal documentation around it.

I am not on my work system today. But, I'll give you a reproducible scenario as soon as I get to my system. We surely don't want to pin ourself to v62 forever.

[remram44]

I'm running into a related issue, where after installing -e ./foo (containing ./foo/setup.py, ./foo/foo.py), somehow import foo gives me a namespace package for ./foo rather than the module ./foo/foo.py. This is a change in behavior that breaks various CI and seems extremely weird; why isn't an installed package preferred over an implicit namespace package?

I can pin setuptools 65 but this seems like a bug on your side.

[ofek]

Have you tried Hatchling?

[abhinavsingh]

@abravalheri I apologise for missing out on it. I checked our codebase today and we are still using "setuptools <= 62.6.0", which is not good in the long run (setuptools is now on v67). Likely I simply missed the notifications but today's activity has brought me back here.

I really want to get past this pinned setuptools issue in our codebase. Will investigate back into it over the coming weekend and come up with a minimal example to reproduce the issue at your end.

[pankajkoti]

hi team, just checking my luck, by any chance, do we have an update on this issue? I am still facing issues with editable installs in a local virtualenv for Apache Airflow.

[abravalheri]

Hi @pankajkoti to keep investigating this issue we need a reproducer of the original problem.

I reported some thoughts in #3548 (comment) (please note the remarks about entries in sys.path).

[abravalheri]

note that after the first execution of our script we create a folder with the same name as our project (so the first execution works). However, the second execution of the script fails, since the importer "finds" the package in the current working directory i.e. the newly created folder.

Hi @kptkin, in our docs we list the limitation of coinciding names of file entries in the working directory. If you need to support that use case you can try other installation modes, e.g. pip install -e . --config-settings editable_mode=strict.

[kptkin]

note that after the first execution of our script we create a folder with the same name as our project (so the first execution works). However, the second execution of the script fails, since the importer "finds" the package in the current working directory i.e. the newly created folder.

Hi @kptkin, in our docs we list the limitation of coinciding names of file entries in the working directory. If you need to support that use case you can try other installation modes, e.g. pip install -e . --config-settings editable_mode=strict.

Ah thanks for the reply! Yeah, I found this exact flag in the docs, good to have a verification that this is the recommended approach and it won't be something that is/should be fixed.

[marcospgp]

Just ran into this issue after updating to python 3.12, should I just revert to 3.11 and wait or is there a different approach I should take?

[HexDecimal]

If you're affected by this issue to the point where even editable_mode=strict won't work then the Legacy Behavior can still be enabled as a temporary workaround:

pip install -e . --config-settings editable_mode=compat

This is not a long term solution. If any of your tools requires editable_mode= to work correctly then it should probably be reported as an issue to that tools repo.

[marcospgp]

@HexDecimal What if it's our tool having the issue? How should it be updated to conform to editable installs in python 3.12?

[HexDecimal]

@HexDecimal What if it's our tool having the issue? How should it be updated to conform to editable installs in python 3.12?

This issue is not caused by your Python version. It is caused by installing setuptools packages in editable mode using the latest version of setuptools.

I've only experienced this issue from the perspective of a tool user (VSCode/Pylance) so I'm not sure what to do on the development side.