Path dependencies with setup.py cannot be installed if setup.py imports numpy (or more general, imports any site-packages)

[hyliu1989]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10
• Poetry version: 1.1.4

Issue

I have searched in here and found #2638 is similar but not the same problem as I am facing. I have also tried a possible solution of [build-system].requires and consulted its documentation (as specified in #744) but no success. Here is a snapshot of the error message:

C:\Users\XXX\Desktop\test-module>poetry update -vvv
Updating dependencies
Resolving dependencies...

  PackageInfoError

  Unable to determine package info for path: C:\Users\XXX\Desktop\test-module\lib\my-dev-module

  Fallback egg_info generation failed.

  Command C:\Users\XXX\AppData\Local\Temp\tmpg5w9jw0j\.venv\Scripts\python.exe setup.py egg_info errored with the following return code 1, and output:
  Traceback (most recent call last):
    File "setup.py", line 11, in <module>
      import numpy
  ModuleNotFoundError: No module named 'numpy'

  at ~\.poetry\lib\poetry\inspection\info.py:503 in _pep517_metadata
      499│                     venv.run("python", "setup.py", "egg_info")
      500│                     return cls.from_metadata(path)
      501│                 except EnvCommandError as fbe:
      502│                     raise PackageInfoError(
    → 503│                         path, "Fallback egg_info generation failed.", fbe
      504│                     )
      505│                 finally:
      506│                     os.chdir(cwd.as_posix())
      507│

I will reuse its structure and show how the problem is reproduced.

The project structure is something like

test-module/
  + pyproject.toml
  + testmodule/
      + __init__.py
  + lib/
      + my-dev-module/
          + setup.py
          + mydevmodule/
              + __init__.py

The toml file is

[tool.poetry]
name = "testmodule"
version = "0.1.0"

[tool.poetry.dependencies]
python = "^3.8"
numpy = "^1.19.1"
mydevmodule = {path = "./lib/my-dev-module", develop = true}

[build-system]
requires = ["numpy>=1.19.1,<1.20.0","poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

And ./lib/my-dev-module/setup.py is like

from distutils.core import setup
import numpy

if __name__ == "__main__":
    setup(
        name="mydevmodule",
        version="0.2",
        description="",
        author="me",
    )

The setup() in setup.py is a placeholder and the script does not get executed to setup(). It erred out at line 2 when importing numpy. In my actual application, module numpy is used to provide its C library for non-Python extension.

My question is: why does it fail at importing numpy? Shouldn't specifying [build-system].requires fix this? If not, I got that to fix this problem requires a [build-system].requires and a pyproject.toml in mydevmodule but that is not an ideal solution. Is there any other ways to specify the installation time requirements of a path-dependency module? Note that I have put numpy in both the dependencies and build-system requirement of the top-level project testmodule.

[sinoroc]

@hyliu1989

Looks to me like you might need a ./lib/my-dev-module/pyproject.toml file such as:

[build-system]
build-backend = 'setuptools.build_meta'
requires = [
    'setuptools >= 43.0.0',
    'numpy',
]

Although it might get complicated for the editable installation...

[hyliu1989]

Yes that could help, but it is less ideal.

This means that all the previously written setup.py containing importing site-packages would not work. I am not sure if many people encounter this problem but we can wait and see if this issue is severe.

[sinoroc]

setup.py containing site-packages

@hyliu1989 I do not understand what you mean.

---

Edit

Oh, you mean setup.py files that try to import modules or packages from the current environment? No, since there is build isolation, things like that do not work by default any more. But adding a pyproject.toml file as I have shown is easy and straightforward. When it can not be added, then most likely build isolation can be deactivated for that particular case.

[hyliu1989]

Are there a discussion and documentations about build isolation? I would like to know how to deactivate it.

[sinoroc]

Are there a discussion and documentations about build isolation? I would like to know how to deactivate it.

@hyliu1989 I do not have a self contained link to suggest, nothing that comes to mind. You might have to read a bit everywhere.

• maybe PEP 517 is interesting for some related background
• with pip one can use pip install --no-build-isolation, to deactivate it
• I do not know how to deactivate build isolation for poetry

[finswimmer]

@hyliu1989 your case is exactly why PEP 517 was invented: The need to know which packages are needed to build the package before starting the build process. So the way that @sinoroc shows above is the correct solution.

[hyliu1989]

@finswimmer I got that to fix this problem requires a [build-system].requires and a pyproject.toml in mydevmodule but that is not an ideal solution. It is forcing the developers of mydevmodule to adopt pyproject.toml while it has a complicated but working setup.py and a `requirements.txt.

[sinoroc]

@hyliu1989 You can add the minimal pyproject.toml without changing anything else. No modification to setup.py. Developers of the project can still use python setup.py develop or whatever other setuptools commands they need. There should be no drawbacks when done correctly. But the big advantage of making sure that numpy is available when pip (or poetry or any other build frontend) need to build that project.

[hyliu1989]

@sinoroc Thanks! Now I remember seeing that poetry says setup.py exists and it is going to use that setup.py. Having a pyproject.toml with minimal specification will be a solution to my question.

A follow-up question is: will poetry support running an existing setup.py in the foreseeable future? Is this behavior something poetry will get rid of?

[sinoroc]

That is the beauty of it. poetry nor pip (the build front ends) do not need to support setuptools (or any other build back ends) directly. As long as they all agree to the PEP 517 standard (which is declared in the [build-system] section of pyproject.toml), then it will all work just fine.

In those cases, no need for poetry (or pip) to execute the setup.py directly. Instead they call the function declared in the build-backend field which in the case of setuptools is setuptools.build_meta. And that function most likely reads and/or executes the setup.py (but this is an implementation detail of the backend, that the frontends don't need to know about).

I hope it is clear, let me know if some things need more details.

[sinoroc]

@hyliu1989

Now that I re-read your question:

will poetry support running an existing setup.py in the foreseeable future? Is this behavior something poetry will get rid of?

I think I missed the point of it. So allow me to rectify...

To be honest, I do not know if poetry will forever be able to install projects that have a setup.py and are not PEP517 conform (i.e. no pyproject.toml). I am not a maintainer, so it is not my place to talk on this topic.

My personal opinion is: as much as I would like to get rid of those, there is such a huge amount of older pre-PEP517 projects (i.e. setup.py only) still in use, that it would make sense to keep this compatibility for as long as possible.

[finswimmer]

To be honest, I do not know if poetry will forever be able to install projects that have a setup.py and are not PEP517 conform (i.e. no pyproject.toml). I am not a maintainer, so it is not my place to talk on this topic.

There are currently no plans to drop the support for non-pyproject.toml projects. Furthermore PEP517 says "If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py (either directly, or by implicitly invoking the setuptools.build_meta:legacy backend)."

[zoj613]

is there a way to install a local project that is only contained in a __init__.py file? I keep getting:

$ poetry install
Installing dependencies from lock file

No dependencies to install or update

  ValueError

  pyhtnorm is not a package.

  at ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/masonry/utils/package_include.py:73 in check_elements
      69│                 self._package = root.name
      70│                 self._elements = sorted(list(root.glob("**/*")))
      71│ 
      72│                 if not self.is_stub_only() and not self.has_modules():
    → 73│                     raise ValueError("{} is not a package.".format(root.name))
      74│ 
      75│                 self._is_package = True
      76│             else:

Anytime I use poetry install. It will only go away if there is a <project name>.py file in the source tree. I dont like the idea of having an empty .py file just to get poetry to work.

[sinoroc]

@zoj613 This is not related to the rest of the current discussion, is it? I think your query is similar to this one, please read that discussion. And if it does not correspond to what you want then I would recommend you to open your own new ticket (or come to poetry's Discord chat channel).

[roshambo919]

I'm trying to install a local package that has a submodule. The submodule's build needs torch to compile some extensions. I'm having trouble making sure torch is available during the build. Specifically, this is causing difficulties creating a poetry.lock file when I try to run poetry lock. I have tried making a pyproject.toml file within the submodule and adding torch as a build-system requirement, but it doesn't seem to be doing anything. Any suggestions on this?

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.