build uninstalls setuptools even if needed by a dependency

[astrojuanlu]

This is a continuation of #109, although slightly different. Not really, setuptools is still installed in my venv after this happens. So, build isolation is working.

I have a project that uses flit as build backend, however some of its dependencies use setuptools as build backend. When I call python -m build ., what I observe is:

1. setuptools gets uninstalled
2. Build dependencies are installed
3. Runtime dependencies start being downloaded
4. When it reaches a dependency that needs setuptools, it fails with No module named 'setuptools'

Example:

$ python -m build .                                                                                                                                                                                                                                                    
Found existing installation: setuptools 49.2.1                                                                                                                                                                                                                                                                               
Uninstalling setuptools-49.2.1:                                                                                                                                                                                                                                                                                              
  Successfully uninstalled setuptools-49.2.1                                                                                                                                                                                                                                                                                 
Collecting oldest-supported-numpy                                                                                                                                                                                                                                                                                            
  Using cached oldest_supported_numpy-0.8-py3-none-any.whl (3.7 kB)                                                                                                                                                                                                                                                          
Collecting flit_core<3,>=2.0                                                                                                                                                                                                                                                                                                 
  Using cached flit_core-2.3.0-py2.py3-none-any.whl (40 kB)                                                                                                                                                                                                                                                                  
Collecting wheel                                                                                                                                                                                                                                                                                                             
  Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB) 
Collecting numpy==1.19.3; python_version == "3.9" and platform_python_implementation != "PyPy"                                                                                                                                                                                                                       [79/560]
  Using cached numpy-1.19.3-cp39-cp39-manylinux2010_x86_64.whl (14.9 MB)                                                                                                                                                                                                                                                     
Collecting pytoml                                                                                                                                                                                                                                                                                                            
  Using cached pytoml-0.1.21-py2.py3-none-any.whl (8.5 kB)                                                                                                                                                                                                                                                                   
Installing collected packages: numpy, oldest-supported-numpy, pytoml, flit-core, wheel                                                                                                                                                                                                                                       
Successfully installed flit-core-2.3.0 numpy-1.19.3 oldest-supported-numpy-0.8 pytoml-0.1.21 wheel-0.36.2                                                                                                                                                                                                                    
WARNING: You are using pip version 20.2.3; however, version 21.0.1 is available.                                                                                                                                                                                                                                             
You should consider upgrading via the '/tmp/build-env-yoyx_wof/bin/python -m pip install --upgrade pip' command.                                                                                                                                                                                                             
Ignoring pycodestyle: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                  
Ignoring hypothesis: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                   
Ignoring tox: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                          
Ignoring sphinx: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                       
Ignoring coverage: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                     
Ignoring myst-parser: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                  
Ignoring sphinx-gallery: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                               
Ignoring notebook: markers 'extra == "jupyter"' don't match your environment                                                                                                                                                                                                                                                 
Ignoring isort: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                        
Ignoring nbconvert: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                    
Ignoring sphinx-notfound-page: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                         
Ignoring pytest-mypy: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                  
Ignoring sphinx-autoapi: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                               
Ignoring sphinx-rtd-theme: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                             
Ignoring jupytext: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                     
Ignoring pytest-remotedata: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                            
Ignoring nbsphinx: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                     
Ignoring ipywidgets: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                   
Ignoring mypy: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                            
Ignoring pytest-cov: markers 'extra == "dev"' don't match your environment                                                                                                                                                                    
Ignoring black: markers 'extra == "dev"' don't match your environment                                                                                                                                                                         
Ignoring czml3: markers 'extra == "cesium"' don't match your environment                                               
Ignoring pytest: markers 'extra == "dev"' don't match your environment                                                                                                                                                                        
Ignoring pytest-mpl: markers 'extra == "dev"' don't match your environment                                             
Ignoring jupyter-client: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                  
Ignoring ipython: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                         
Ignoring pytest-doctestplus: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                              
Ignoring sphinx-hoverxref: markers 'extra == "dev"' don't match your environment                                       
Ignoring cached-property: markers 'python_version < "3.8"' don't match your environment                                                                                                                                                                                                                                      
Ignoring ipykernel: markers 'extra == "dev"' don't match your environment                                              
Collecting matplotlib!=3.0.1,>=2.0                                                                                                                                                                                                                                                                                           
  Downloading matplotlib-3.3.4-cp39-cp39-manylinux1_x86_64.whl (11.5 MB)                                               
     |████████████████████████████████| 11.5 MB 1.8 MB/s                                                                                                                                                                                                                                                                     
Collecting astropy<5,>=3.2                                                                                             
  Downloading astropy-4.2-cp39-cp39-manylinux1_x86_64.whl (9.6 MB)                                                     
     |████████████████████████████████| 9.6 MB 1.5 MB/s                                                                                                                                                                                                                         
Collecting numba>=0.46                                                                                                                                                                                                                        
  Using cached numba-0.51.2.tar.gz (2.1 MB)                                                                                                                                                                                                                                                                                  
    ERROR: Command errored out with exit status 1:                                                                                                                                                                                                                                                                           
     command: /tmp/build-env-yoyx_wof/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vpxvqzau/numba/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vpxvqzau/numba/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');
f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-5n_1l6e9                                                                                                                                   
         cwd: /tmp/pip-install-vpxvqzau/numba/                                                                                                                
    Complete output (3 lines):                                                                                         
    Traceback (most recent call last):                 
      File "<string>", line 1, in <module>                                                                             
    ModuleNotFoundError: No module named 'setuptools'                                                                                   
    ----------------------------------------                                                                                            
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.                                                                                                                                                                                                              
WARNING: You are using pip version 20.2.3; however, version 21.0.1 is available.                                       
You should consider upgrading via the '/tmp/build-env-yoyx_wof/bin/python -m pip install --upgrade pip' command.                                                                                                                                                                
Traceback (most recent call last):                                                                                                      
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main                                                 
    return _run_code(code, main_globals, None,                                                                         
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code                                                            
    exec(code, run_globals)                                                                                            
  File "/home/juanlu/Personal/poliastro/poliastro/.venv39/lib/python3.9/site-packages/build/__main__.py", line 214, in <module>                                                                                                               
    main(sys.argv[1:], 'python -m build')                                                                              
  File "/home/juanlu/Personal/poliastro/poliastro/.venv39/lib/python3.9/site-packages/build/__main__.py", line 206, in main                                                                                                                   
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)  
  File "/home/juanlu/Personal/poliastro/poliastro/.venv39/lib/python3.9/site-packages/build/__main__.py", line 94, in build_package                                                                                                           
    _build_in_isolated_env(builder, outdir, distributions, config_settings)                   
  File "/home/juanlu/Personal/poliastro/poliastro/.venv39/lib/python3.9/site-packages/build/__main__.py", line 58, in _build_in_isolated_env                                                                                                  
    env.install(builder.get_dependencies(distribution))                                                                
  File "/home/juanlu/Personal/poliastro/poliastro/.venv39/lib/python3.9/site-packages/build/env.py", line 143, in install                                                                                                                     
    subprocess.check_call(cmd)                                                                                         
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call                                                     
    raise CalledProcessError(retcode, cmd)                                                                             
subprocess.CalledProcessError: Command '['/tmp/build-env-yoyx_wof/bin/python', '-Im', 'pip', 'install', '--no-warn-script-location', '-r', '/tmp/build-reqs-s3r1fubv.txt']' returned non-zero exit status 1.

[astrojuanlu]

(Edit: Misleading reference to old issue, or so I think)

[FFY00]

Aghh, seems like pip is not building the dependencies in isolation. The dependencies should not be build in the build environment. They should be built separately and then installed onto the build environment. I think this is probably due to an old pip version.

@gaborbernat do you know when did pip started building in isolation?
We should bump pip to that version.

[uranusjr]

I believe build isolation was introduced to pip with the PEP 517 implementation.

[gaborbernat]

Which project are you trying to build? Do you have a link?

[astrojuanlu]

Yeap sorry, it's https://github.com/poliastro/poliastro/blob/master/pyproject.toml

[astrojuanlu]

About the old pip version: the logs say WARNING: You are using pip version 20.2.3; however, version 21.0.1 is available., so I tried upgrading pip and still failed with the same error.

We should bump pip to that version.

Sorry, you're referring to pypa/build itself, now I get it

[FFY00]

Looks like that pip should be recent enough 🤔

[layday]

pip does not use isolation for pyproject.toml-less packages. What I'm really curious about is why we're installing astroquery which is a runtime dependency - not a build dependency!

[layday]

-> env.install(builder.get_dependencies(distribution))
(Pdb) builder.build_dependencies
{'flit_core >=2.0,<3', 'oldest-supported-numpy', 'wheel'}
(Pdb) builder.get_dependencies('sdist')
{'jupyter-client ; extra == "dev"', 'astroquery >=0.3.9', 'pycodestyle ; extra == "dev"', 'scipy >=1.4.0', 'sphinx-hoverxref==0.5b1 ; extra == "dev"', 'notebook ; extra == "jupyter"', 'pyerfa', 'jplephem', 'jupytext ; extra == "dev"', 'nbconvert>=5.5 ; extra == "dev"', 'coverage ; extra == "dev"', 'sphinx ; extra == "dev"', 'czml3 ~=0.5.3 ; extra == "cesium"', 'sphinx-notfound-page ; extra == "dev"', 'ipykernel ; extra == "dev"', 'ipywidgets>=7.0 ; extra == "dev"', 'isort ; extra == "dev"', 'pytest-remotedata ; extra == "dev"', 'nbsphinx>=0.5.0 ; extra == "dev"', 'ipython>=5.0 ; extra == "dev"', 'plotly ~=4.0', 'pytest-cov<2.6.0 ; extra == "dev"', 'black==20.8b1 ; extra == "dev"', 'matplotlib >=2.0,!=3.0.1', 'sphinx_rtd_theme==0.5.0 ; extra == "dev"', 'pandas', "cached_property ; python_version<'3.8'", 'astropy >=3.2,<5', 'pytest-doctestplus>=0.8 ; extra == "dev"', 'numba >=0.46', 'myst-parser>=0.13.1 ; extra == "dev"', 'tox ; extra == "dev"', 'pytest-mypy ; extra == "dev"', 'pytest>=3.2 ; extra == "dev"', 'mypy>=0.740 ; extra == "dev"', 'sphinx-autoapi ; extra == "dev"', 'sphinx-gallery ; extra == "dev"', 'numpy', 'pytest-mpl ; extra == "dev"', 'hypothesis ; extra == "dev"'}

I, uh... what?

---

I've traced that back to flit 2.0 - I suggest requiring flit 3.0 at a minimum if you don't want to have all of your runtime dependencies installed at build time.

As for the build issue, I assume it's only surfaced now because we're no longer inheriting pip from the environment build is installed in. One option would be to enforce build isolation in pip with --use-pep517.

[FFY00]

One option would be to enforce build isolation in pip with --use-pep517.

+1 on this, should be backwards compatible.