Skip to content

Conversation

@seiko2plus
Copy link
Member

@seiko2plus seiko2plus commented Nov 12, 2021

this patch is really important since most of the users aren't able to determine
the build error when the toolchain or the built environment missing
executable files of compiler, linker, assembler, etc.

related to #20335

@rgommers
Copy link
Member

Doing some testing:

$ touch numpy/core/src/npysort/radixsort.cpp
$ export CXX=THIS_IS_NOT_A_CXX_COMPILER
$ python setup.py build_ext -i

gives:

compile options: '-DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DHAVE_CBLAS -Inumpy/core/src/common -Inumpy/core/src/umath -Inumpy/core/include -Inumpy/core/include/numpy -Ibuild/src.macosx-11.0-arm64-3.9/numpy/distutils/include -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/Users/rgommers/mambaforge/envs/numpy-dev/include/python3.9 -Inumpy/core/src/common -Inumpy/core/src/npymath -c'
extra options: '-std=c++11 -D__STDC_VERSION__=0 -fno-exceptions -fno-rtti'
THIS_IS_NOT_A_CXX_COMPILER: numpy/core/src/npysort/radixsort.cpp
[Errno 2] No such file or directory: 'THIS_IS_NOT_A_CXX_COMPILER'
error: Command "THIS_IS_NOT_A_CXX_COMPILER -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/rgommers/mambaforge/envs/numpy-dev/include -arch arm64 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/rgommers/mambaforge/envs/numpy-dev/include -arch arm64 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/rgommers/mambaforge/envs/numpy-dev/include -D_FORTIFY_SOURCE=2 -isystem /Users/rgommers/mambaforge/envs/numpy-dev/include -ftrapping-math -DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DHAVE_CBLAS -Inumpy/core/src/common -Inumpy/core/src/umath -Inumpy/core/include -Inumpy/core/include/numpy -Ibuild/src.macosx-11.0-arm64-3.9/numpy/distutils/include -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/Users/rgommers/mambaforge/envs/numpy-dev/include/python3.9 -Inumpy/core/src/common -Inumpy/core/src/npymath -c numpy/core/src/npysort/radixsort.cpp -o build/temp.macosx-11.0-arm64-3.9/numpy/core/src/npysort/radixsort.o -MMD -MF build/temp.macosx-11.0-arm64-3.9/numpy/core/src/npysort/radixsort.o.d -std=c++11 -D__STDC_VERSION__=0 -fno-exceptions -fno-rtti" failed with exit status 2

########### EXT COMPILER OPTIMIZATION ###########
...
<many lines of output left - need to fix later>

So it changes exit status from 127 to 2. We expected 127:

% $CXX compiler; echo $?
zsh: command not found: THIS_IS_NOT_A_CXX_COMPILER
127

Maybe 2 is the result here because we're running this from Python rather than in a terminal directly? It does have a more informative (probably message):

[Errno 2] No such file or directory: 'THIS_IS_NOT_A_CXX_COMPILER'

That message is just hard to miss in the output, because you have to scroll up quite a bit and then there are no blank lines around the output. How about we add a few blank lines above and below raise DistutilsExecError @seiko2plus?

@seiko2plus
Copy link
Member Author

@rgommers,

So it changes exit status from 127 to 2. We expected 127:

shell environment returns 127 instead of returning the last system call error number, since it is reserved for the subprocess.

So I thought it would be better to not follow this way in case of the build log is redirected to file instead of stdout to make it easier to trace but after reconsidering I think it's a bad idea since it may make the user mixup between parent and subprocess error numbers.

How about we add a few blank lines above and below raise DistutilsExecError @seiko2plus?

yes, it would be nice

  this patch is really important since most of the users aren't able to determine
  the build error when the toolchain or the built environment missing
  executable files of compiler, linker, assembler, etc.
@seiko2plus
Copy link
Member Author

seiko2plus commented Nov 13, 2021

errno reverted to 127 and blank lines added above and after the OS error.

export CC=NO_SI_SI
python setup.py build
....
build_src
building py_modules sources
building library "npymath" sources


[Errno 2] No such file or directory: 'NO_SI_SI'


Traceback (most recent call last):
  File "/home/seiko/work/repos/numpy/setup.py", line 450, in <module>
    setup_package()
  File "/home/seiko/work/repos/numpy/setup.py", line 442, in setup_package
    setup(**metadata)
  File "/home/seiko/work/repos/numpy/numpy/distutils

Copy link
Member

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @seiko2plus!

@rgommers rgommers merged commit 4978118 into numpy:main Nov 14, 2021
@rgommers rgommers added this to the 1.22.0 release milestone Nov 14, 2021
@charris charris removed the 09 - Backport-Candidate PRs tagged should be backported label Nov 25, 2021
@charris charris removed this from the 1.21.5 release milestone Nov 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants