Editable install stops working with setuptools 64 and above

[ianjosephwilson]

Took me a while to figure out what happened because it seemed I didn't change anything. I have a simple pyproject.toml file:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

I also have a setup.py file:

from distutils.core import setup
from setuptools import find_packages
import sys

if sys.version_info < (3,):
    print("Please use python3.")
    sys.exit(1)


requires = [
    "pyjwt",
    "requests",
    "zope.interface",
]

sqlalchemy_deps = ["sqlalchemy"]

pyramid_deps = ["pyramid"]


setup(
    name="shopauth",
    version="0.2a",
    description="Unofficial auth library for shopify apps.",
    author="Ian Wilson",
    author_email="ian@laspilitas.com",
    url="https://www.github.com/ianjosephwilson/shopauth",
    install_requires=requires,
    packages=find_packages(),
    #package_dir={"shopauth": "shopauth"},
    include_package_data=True,
    zip_safe=False,
    extras_require={
        "sqlalchemy": sqlalchemy_deps,
        "pyramid": pyramid_deps,
        "dev": ["flake8", "black"],
    },
)

I use pip -e shopauth to install an editable package I wrote alongside my main apppackage during development (also an editable install but no pyproject.toml file). In setuptools 64 the new pep660 implementation I guess went into affect and broke this setup. The helper package imports but doesn't have anything in it. If you dir it there is just the boilerplate attributes like __name__, __path__, etc.

I'm reluctant to report this as a bug because I have no idea how this configuration should work. I realize now I should have setuptools pinned to a version to prevent this sort of suprise breakage in the future but want to know how to shoehorn my workflow to be compatible with the future of python packages.

Looking for some sort of direction. Thanks.

[abravalheri]

Hello @ianjosephwilson, could you describe a little bit more in detail how you are reproducing this error?

I tried the following, but everything runs smoothly, I don't seem to find any error:

cd /tmp
git clone https://www.github.com/ianjosephwilson/shopauth
cd shopauth
virtualenv -p python3.10 .venv
.venv/bin/python -m pip install -U 'pip==22.1.2'  # latest version
.venv/bin/python -m pip install -e .
mkdir _avoid_implicit_pythonpath  # can be avoided if you are using src-layout
cd _avoid_implicit_pythonpath
../.venv/bin/python -c 'import shopauth; print(dir(shopauth))'

As a response, I am getting the following:

['AssociatedUser', 'DAY_IN_SECONDS', 'DEFAULT_SESSION_TYPE_LOOKUP', 'IAppInstalledHandler', 'INSTALL_SESSION_TYPE', 'ISessionSerializer', 'IStorageShim', 'IWebShim', 'InstallSession', 'MINUTE_IN_SECONDS', 'OAUTH_SESSION_TYPE', 'OAuthSession', 'OFFLINE_ACCESS_MODE', 'ONLINE_ACCESS_MODE', 'OfflineSession', 'OnlineAccessInfo', 'OnlineSession', 'ShopAuthConfig', 'ShopAuthService', 'ShopSessionSerializer', 'ShopifyJWTPayload', 'ZERO_SECONDS', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'asdict', 'b64decode', 'build_shop_host', 'dataclass', 'datetime', 'extract_shop_name', 'field', 'hashlib', 'hmac', 'interfaces', 'json', 'jwt', 'logger', 'logging', 'random', 're', 'requests', 'scopes', 'scopes_have_changed', 'string', 'time', 'timedelta', 'timezone', 'urlencode', 'util', 'uuid', 'zope']

[ianjosephwilson]

It is a bit more complicated but I wasn't sure if it was necessary to explain everything.

I have a Dockerfile that that builds an image of my application and installs a wheel of shopauth, that is the typical production setup.

During development I set an environmental variable that is checked in my entrypoint.py, the Docker CMD, and I also mount the shopauth code as a volume inside the container, ie. mapping my local code directory into the docker container. If that variable is set then I install the shopauth code as editable, overwriting the prior install, during the execution of entrypoint.py before my actual service starts. This allows reloading to work because it can track the volume from in the container (using gunicorn) and restart when files change and I can change the files in my editor, etc.

In the examples below the shopauth dir on my host system contains setup.py (and friends) and a subdirectory called shopauth that actually contains the source code.

Snippet from the entrypoint.py that performs the editable install:

if env.EDITABLE_SHOPAUTH:
    # But also try installing it.
    print("Linking shopauth with pip -e.")
    check_call(["pip", "install", "-e", "/app/shopauth"])

Snippet from the docker-compose file that maps the host dir to the docker container dir:

#....
environment:
      #...
      - EDITABLE_SHOPAUTH=1
volumes:
      # ....
      - ../shopauth:/app/shopauth

Using PYTHON_PATH seems to work as well as pinning the setuptools version to <64. I have tried removing the production workflow that pre-installs the wheel and that didn't seem to have any affect. Could setuptools or pip be having a problem accessing a docker volume ? How could I check that ?

Some versions:
Docker image python:3.10.5-buster
Docker version 20.10.17, build 100c701
docker-compose version 1.28.2, build 67630359
pip 22.1.2

The setuptools version is whatever is pulled in based on if I pinned or didn't pin it in the pyproject.yaml.

Thanks.

[ianjosephwilson]

I think I figured it out actually. It seems that the directory with setup.py is being picked up as an __init__.py-less package. That is why it has no attributes because it is empty. I could probably do import shopauth.setup and it would import in the setup file. If I break it up then it works fine. I'm not sure what combination of factors causes this issues.

My failing source mapping looks like this:

/myapp/setup.py --> /app/setup.py
/mapp/myapp/__init__.py --> /app/myapp/__init__.py

/shopauth/setup.py --> /app/shopauth/setup.py
/shopauth/shopauth/__init__.py --> /app/shopauth/shopauth/__init__.py

Keeping the source isolated like this works, ie. don't make shopauth a subdirectory of main package:

/myapp/setup.py --> /app/setup.py
/mapp/myapp/__init__.py --> /app/myapp/__init__.py

/shopauth/setup.py --> /shopauth/setup.py
/shopauth/shopauth/__init__.py --> /shopauth/shopauth/__init__.py

[ianjosephwilson]

Maybe this is because /app is being installed as editable and that is getting put in the PYTHON_PATH?

[abravalheri]

I see, Python automatically adds the CWD to sys.path (first entry! this takes priority).

Moreover, Python considers any directory as a package (it does not need to have a __init__.py).

Other things you can consider are:

• change the name of the folder when mapping the directory inside the container to something different than your project folder
• use a different working dir

[abravalheri]

Maybe this is because /app is being installed as editable and that is getting put in the PYTHON_PATH?

I believe that previously the setup was working because unknowingly you were activating a undocumented/undefined/unreliable behaviour of namespaces packages:

• Accidentally your container was creating a namespace package via the implicit CWD entry in sys.path,
• But the namespace package was "overridden" by a non-namespace package (your editable installation).

I believe that this "overriding" behaviour is undefined and accidental.

Since version 64, setuptools changed its editable implementation to rely on import hooks instead of static .pth files, which are not subject to the accidental/undefined "namespace overriding" behaviour.

(I might be wrong, and the "overriding" is well-defined. If that is the case, could anyone please point me the description of this behaviour in the Python docs or any PEP?)

[ianjosephwilson]

Whew, thought I was losing my mind but this makes more sense now. Thanks for your help.

I think for now I should probably just map the volume in somewhere else. Maybe even create a parent parent directory like, /app/libs/shopauth and /app/myapp.

[abravalheri]

Yeah, the implicit CWD entry on sys.path is the source of a lot of packaging problems in Python.

I thought that using a src-layout was enough to prevent them, but it seems that some causes are surfacing due to coincidental name of the parent directory...

The good news are that in Python 3.11, you have a new command line argument -P that can help in these situations: https://docs.python.org/3.11/whatsnew/3.11.html#summary-release-highlights