Console scripts not available during build

[kumaraditya303]

When building, console scripts paths are not injected into the path hence console scripts are not available during build.

Example:
When cmake is installed, cmake.exe is not on path, leading to error during when when cmake is invoked with subprocess.

[henryiii]

For cmake, you can import cmake and it has a utility for calling CMake there. (not a general solution)

[FFY00]

PATH should be preserved when building in isolation. Can you provide the steps to reproduce?

[henryiii]

I literally ran into this immediately after posting! :)

I'm hitting it in scikit-hep/awkward#683, but you can just use the cmake_example from pybind11:

docker run -it python:3.9 bash
git clone https://github.com/pybind/cmake_example
cd cmake_example
pip install build
python -m build

Will produce:

error: [Errno 2] No such file or directory: 'cmake'

The workaround I mentioned does work:

try:
    import cmake
    CMAKE = os.path.join(cmake.CMAKE_BIN_DIR, "cmake")
except ImportError:
    CMAKE = "cmake"

.... # Replace "cmake" with CMAKE in calls below

But it should be in the path.

[kumaraditya303]

I gave CMake just for an example it to be easily reproducible, as far as I think during building the console scripts should be on the PATH @FFY00

[gaborbernat]

You're right, per PEP-517

All command-line scripts provided by the build-required packages must be present in the build environment's PATH. 

The question is why is not in your case.

[kumaraditya303]

build/src/build/env.py

Line 131 in d8a30a3

'--no-warn-script-location',

is supressing the warning that the console scripts are not on path. @gaborbernat @FFY00

[gaborbernat]

So indeed we fail on the requirement.

All command-line scripts provided by the build-required packages must be present in the build environment's PATH.

Why?

• the default subprocess runner we use https://github.com/pypa/pep517/blob/master/pep517/wrappers.py#L69-L75 does not add the scripts folder to the PATH - see https://github.com/pypa/pep517/blob/master/pep517/wrappers.py#L254-L264

What can we do to fix this?

• https://github.com/pypa/build/blob/master/src/build/env.py#L30 needs to provide a scripts_dir to the console scripts folder of the environment (note this is not os.path.dirname(env_executable) in the general case - scripts_dir and bin_dir are separate configs); we likely want to use sysconfig module to load the scripts_dir
• We can override the default runner here https://github.com/pypa/build/blob/master/src/build/__init__.py#L162, to prepend scripts_dir to the PATH env-var.

@FFY00 @layday do you agree with the proposed solution? Can put in a PR if so.

[layday]

That sounds like the right approach. For Python 3, I thought you might be able to pull the scripts folder from the EnvBuilder but EnvBuilder.create returns None; you'd have to call EnvBuilder.ensure_directories again which performs I/O.

[gaborbernat]

You can avoid the I/O by using sysconfig instead 👍🏻 so that's a better path.

[gaborbernat]

@FFY00 can you confirm please before I start writing code 😄

[FFY00]

That seems fine 👍

[gaborbernat]

@FFY00 and @layday so turn out our approach to borrow pip from the global namespace will not work. When pip generates a console script it always uses the current interpreter as the she-bang line, and this is non configurable. This ends up generating scripts with a bad shebang line (on UNIX a script within /var/folders/jh/kx1127wj32l8c3s97131rhmr0000gp/T/build-env-cv4mltrb/bin/demo-pkg-inline would look like):

#!/Users/bgabor8/git/github/python-build/.tox/4/dev/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from demo_pkg_inline import do
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(do())

Which then fails at startup because obviously, the pip host package does not have the required dependencies. The problem:

• the pip code part that does this https://github.com/pypa/pip/blob/master/src/pip/_internal/operations/install/wheel.py#L708-L722
• a distlib issue detailing how one can set a custom interpreter - https://bitbucket.org/pypa/distlib/issues/134/allow-specifying-the-version-information

However, because pip does not expose this interface, we now must invoke the pip install with pip being installed in the same virtual environment like the one we're targeting. This sadly will make isolated builds a lot slower, but needed for correct functionality. 🤷🏻 any better ideas? (making a PR to expose this configuration in pip seems out of the question because we could only use it in new pip versions)

[henryiii]

I feel like this is related to the problem with cross-compiling, where these get the wrong interpreter burned in.