New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot make install opencv from origin/master on ubuntu 17.10 #10771

Open
Guilmort opened this Issue Feb 3, 2018 · 29 comments

Comments

@Guilmort

Guilmort commented Feb 3, 2018

System information (version)
  • OpenCV => cloned from github origin/master
  • Cuda 9.1/ CuDNN 7.0
  • FFMPEG compiled from sources (cloned from github origin/master)
  • OpenCV contrib : cloned from github origin/master
  • Operating System / Platform => Ubuntu artful 17.10
  • Compiler => gcc6
Detailed description

Error during last "sudo make install" although no problem during other phases.

[ 83%] Building CXX object modules/python3/CMakeFiles/opencv_python3.dir/__/src2/cv2.cpp.o
/home/guilmort/INSTALLATION/opencv-3.4.0/modules/python/src2/cv2.cpp:18:40: fatal error: pyopencv_generated_include.h: No such file or directory
 #include "pyopencv_generated_include.h"
                                        ^
compilation terminated.
Steps to reproduce
cmake command:
cmake -D CMAKE_BUILD_TYPE=RELEASE     -D CMAKE_INSTALL_PREFIX=/opt/opencv     
-D INSTALL_PYTHON_EXAMPLES=ON     -D INSTALL_C_EXAMPLES=OFF     
-D OPENCV_EXTRA_MODULES_PATH=~/INSTALLATION/opencv-contrib/opencv_contrib/modules     
-D PYTHON_EXECUTABLE=~/.virtualenvs/cv/bin/python     -D BUILD_EXAMPLES=ON 
-D WITH_CUDA=ON -D ENABLE_FAST_MATH=1 -D CUDA_FAST_MATH=1 -D WITH_CUBLAS=1 
-D CUDA_NVCC_FLAGS="--std=c++11 --expt-relaxed-constexpr" -DCMAKE_C_COMPILER=gcc-6 
-DCMAKE_CXX_COMPILER=g++-6 ..

make -j4

sudo make install

@StatML

This comment has been minimized.

StatML commented Feb 8, 2018

I had the same problem on Ubuntu 16.04.3 (G++5.4)

@whizzzkid

This comment has been minimized.

Contributor

whizzzkid commented Feb 28, 2018

@Guilmort @StatML Can you try this script to install opencv?

@StatML

This comment has been minimized.

StatML commented Mar 10, 2018

hi @whizzzkid I tried that script, it does not work. :(

@lesreaper

This comment has been minimized.

lesreaper commented Mar 11, 2018

Same exact error, 16.04.3.

Near fresh install of Ubuntu, but with CUDA 9.1 installed.

@kinchungwong

This comment has been minimized.

Contributor

kinchungwong commented Mar 11, 2018

If possible, please post a nearly complete copy of the output and error logs: CMakeFiles/CMakeOutput.log and CMakeFiles/CMakeError.log.

You may edit these files to remove personally identifiable information from them.

However, do not try to extract just a small portion of those files, because the true issue may have happened in a build step that is much earlier than what you have found. Also, these files contain a verbatim copy of the configuration settings of your OpenCV CMake project, which is necessary for others trying to replicate your issue.

You may post the entire file anywhere online that is publicly accessible (for example, as a Github Gist), and then just post the link as a reply to this issue.

Just looking at the limited error excerpt from your post, it is some C++ code that is generated as part of the build. In other words, in an earlier build step, some python programs would be executed; these programs will write into a C++ file. This happens before the C++ project is compiled. Apparently a failure happens somewhere in between.

@whizzzkid

This comment has been minimized.

Contributor

whizzzkid commented Mar 11, 2018

Try building with make -j $(($(nproc)+1)) VERBOSE=1 and paste the entire build log here.

@StatML

This comment has been minimized.

StatML commented Mar 12, 2018

Hi @kinchungwong and @whizzzkid Thanks a lot! My error log can be found from the following link. https://drive.google.com/file/d/1-Yo5piK4rqCOgnAtwUqonwLerjTUIuO3/view?usp=sharing

@alalek

This comment has been minimized.

Contributor

alalek commented Mar 12, 2018

Perhaps this workaround may help (this target generates "missing" file):

make gen_opencv_python_source VERBOSE=1
make <...>
@StatML

This comment has been minimized.

StatML commented Mar 12, 2018

Hi @alalek Thanks a lot! The command make gen_opencv_python_source VERBOSE=1 cannot generate pyopencv_generated_include.h, so the error is still the same...

modules/python/src2/cv2.cpp:22:40: fatal error: pyopencv_generated_include.h: No such file or directory compilation terminated.

@alalek

This comment has been minimized.

Contributor

alalek commented Mar 12, 2018

cannot generate

Logs?

@StatML

This comment has been minimized.

StatML commented Mar 12, 2018

Hi @alalek, the log is as follows. Thanks a lot!

/home/xxx/Downloads/software-devel/cmake-3.10.2-Linux-x86_64/bin/cmake -H/home/xxx/DevLib/opencv-git -B/home/xxx/DevLib/opencv-git/build --check-build-system CMakeFiles/Makefile.cmake 0
make -f CMakeFiles/Makefile2 gen_opencv_python_source
make[1]: Entering directory '/home/xxx/DevLib/opencv-git/build'
/home/xxx/Downloads/software-devel/cmake-3.10.2-Linux-x86_64/bin/cmake -H/home/xxx/DevLib/opencv-git -B/home/xxx/DevLib/opencv-git/build --check-build-system CMakeFiles/Makefile.cmake 0
/home/xxx/Downloads/software-devel/cmake-3.10.2-Linux-x86_64/bin/cmake -E cmake_progress_start /home/xxx/DevLib/opencv-git/build/CMakeFiles 0
make -f CMakeFiles/Makefile2 modules/python_bindings_generator/CMakeFiles/gen_opencv_python_source.dir/all
make[2]: Entering directory '/home/xxx/DevLib/opencv-git/build'
make -f modules/python_bindings_generator/CMakeFiles/gen_opencv_python_source.dir/build.make modules/python_bindings_generator/CMakeFiles/gen_opencv_python_source.dir/depend
make[3]: Entering directory '/home/xxx/DevLib/opencv-git/build'
cd /home/xxx/DevLib/opencv-git/build && /home/xxx/Downloads/software-devel/cmake-3.10.2-Linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/xxx/DevLib/opencv-git /home/xxx/DevLib/opencv-git/modules/python/bindings /home/xxx/DevLib/opencv-git/build /home/xxx/DevLib/opencv-git/build/modules/python_bindings_generator /home/xxx/DevLib/opencv-git/build/modules/python_bindings_generator/CMakeFiles/gen_opencv_python_source.dir/DependInfo.cmake --color=
make[3]: Leaving directory '/home/xxx/DevLib/opencv-git/build'
make -f modules/python_bindings_generator/CMakeFiles/gen_opencv_python_source.dir/build.make modules/python_bindings_generator/CMakeFiles/gen_opencv_python_source.dir/build
make[3]: Entering directory '/home/xxx/DevLib/opencv-git/build'
Generate files for Python bindings and documentation
make[3]: Leaving directory '/home/xxx/DevLib/opencv-git/build'
Built target gen_opencv_python_source
make[2]: Leaving directory '/home/xxx/DevLib/opencv-git/build'
/home/xxx/Downloads/software-devel/cmake-3.10.2-Linux-x86_64/bin/cmake -E cmake_progress_start /home/xxx/DevLib/opencv-git/build/CMakeFiles 0
make[1]: Leaving directory '/home/xxx/DevLib/opencv-git/build'

@alalek

This comment has been minimized.

Contributor

alalek commented Mar 12, 2018

This target should run "gen2.py" script. Probably there is some mess with file timestamps.
Try to remove "modules/python_bindings_generator/" directory, run from build directory "cmake ." and make again.

Expected output:

Scanning dependencies of target gen_opencv_python_source
make[3]: Leaving directory '/home/alalek/projects/opencv/build/opencv'
make -f modules/python_bindings_generator/CMakeFiles/gen_opencv_python_source.dir/build.make modules/python_bindings_generator/CMakeFiles/gen_opencv_python_source.dir/build
make[3]: Entering directory '/home/alalek/projects/opencv/build/opencv'
Generate files for Python bindings and documentation
cd /home/alalek/projects/opencv/build/opencv/modules/python_bindings_generator && /usr/bin/python2.7 /home/alalek/projects/opencv/dev/modules/python/bindings/..//src2/gen2.py /home/alalek/projects/opencv/build/opencv/modules/python_bindings_generator /home/alalek/projects/opencv/build/opencv/modules/python_bindings_generator/headers.txt

@StatML

This comment has been minimized.

StatML commented Mar 12, 2018

Hi @alalek I had tried the steps you suggested, it did not work... :(

@zapcity

This comment has been minimized.

zapcity commented Mar 14, 2018

I am getting exactly the same error. However my environment is:

When I compile my Release build with BUILD_opencv_python3 and INSTALL_PYTHON_EXAMPLES checked, I get the error:

Error: C:\opencv\opencv-3.4.1\modules\python\src2\cv2.cpp(22): fatal error C1083: Cannot open include file: 'pyopencv_generated_include.h': No such file or directory

I know the OP issue is from a ubuntu install perspective. Just wanted to show that its carried across to ms windows.

@kinchungwong

This comment has been minimized.

Contributor

kinchungwong commented Mar 15, 2018

@Guilmort @StatML @zapcity

May I ask, with humility, please try make without -j ?
One thing I realized, is that throughout the years, I have never once run make with -j on OpenCV. I don't even know if it is meant to be safe to do so. Other projects I've done parallel, but not on OpenCV.

The purpose is purely to see if that makes a different outcome. Please try your best to clean up the folders before this experiment. Or if you can afford, try it in a virtual machine or container. Hopefully we'll be learning a few things from it.

@StatML

This comment has been minimized.

StatML commented Mar 15, 2018

Hi @kinchungwong Thanks a lot for your warm reply. I tried make without -j. The results were still the same.

@dbanda

This comment has been minimized.

dbanda commented Mar 28, 2018

@zapcity @Guilmort you may have to manually execute the generator script (gen2.py)

Assuming you called cmake .. from the build directory you could do:

python ./modules/python/src2/gen2.py ./build/modules/python_bindings_generator ./build/modules/python_bindings_generator/headers.txt

I think the bug arises because cmake doesn't find the right python executable to run the script

@shaurabhbharti

This comment has been minimized.

shaurabhbharti commented Apr 2, 2018

@dbanda thanks. this solution worked for me.

@alalek alalek added bug and removed question (invalid) labels Apr 4, 2018

@Guilmort

This comment has been minimized.

Guilmort commented Apr 8, 2018

Same bug on Bionic Beaver (18.04) + opencv/contrib 3.4.1:

[ 84%] Building CXX object modules/python3/CMakeFiles/opencv_python3.dir/__/src2/cv2.cpp.o
/home/guilmort/INSTALLATION/opencv-3.4.1/modules/python/src2/cv2.cpp:22:40: fatal error: pyopencv_generated_include.h: No such file or directory
 #include "pyopencv_generated_include.h"
                                        ^
compilation terminated.

@dbanda : no ./modules/python/src2/gen2.py found in this environment

@peterbyzhang

This comment has been minimized.

peterbyzhang commented Apr 10, 2018

@dbanda Worked for me as well, except I ran your commands one level up from build/, at opencv-3.4.1/. Thank you so much!

@jinchenglee

This comment has been minimized.

jinchenglee commented Apr 11, 2018

I met this problem that I installed virtual environment with Python3 only that the headers generation relies on python2.7 only, :( It took me several days to finally figure this out.

I then switch out to non virtual env to manually generated headers using python2.7:

python ./modules/python/src2/gen2.py ./build/modules/python_bindings_generator ./build/modules/python_bindings_generator/headers.txt

Then coming back to virtual env with python3 only, to continue installation opencv3.4.1, it works. :)

@skumarlabs

This comment has been minimized.

skumarlabs commented Apr 15, 2018

@Guilmort Comment by @peterbyzhang solved the same issue for me. Run your commands one level up from build i.e. at opencv-3.4.1/

@tibaes

This comment has been minimized.

tibaes commented Apr 27, 2018

Just got a similar issue when trying to compile OpenCV 3.4.1 with only Python3 support.
Fixed passing the path to the python2 for CMake: -DPYTHON2_EXECUTABLE=/usr/bin/python2

@mlrus

This comment has been minimized.

mlrus commented Jun 22, 2018

@dband Thank you, it also works with alternate path to gen2.py, i.e., cmake ../opencv-3.4.1

user@host:~/vis/opencv_build$ python ../opencv-3.4.1/modules/python/src2/gen2.py ./modules/python_bindings_generator ./modules/python_bindings_generator/headers.txt
user@host:~/vis/opencv_build$ make
@mhsmith

This comment has been minimized.

mhsmith commented Jul 27, 2018

I think the comments in #8425 explain more about what's happening here:

for some reason PYTHON2INTERP_FOUND was True, so PYTHON_DEFAULT_EXECUTABLE was set to PYTHON2_EXECUTABLE, which was nothing

I don't know what the cause of this is, but I've traced through its effects. add_custom_command in modules/python/bindings/CmakeLists.txt uses PYTHON_DEFAULT_EXECUTABLE to run gen2.py. When PYTHON_DEFAULT_EXECUTABLE is empty, this somehow causes the command to be completely omitted from the generated makefile.

Specifically, here's how the generated rule in modules/python_bindings_generator/CMakeFiles/gen_opencv_python_source.dir/build.make is supposed to look:

@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold --progress-dir=<redacted>/_skbuild/linux-x86_64-3.6/cmake-build/CMakeFiles --progress-num=$(CMAKE_PROGRESS_1) "Generate files for Python bindings and documentation"
cd <redacted>/_skbuild/linux-x86_64-3.6/cmake-build/modules/python_bindings_generator && /usr/bin/python2 <redacted>/opencv/modules/python/bindings/..//src2/gen2.py <redacted>/_skbuild/linux-x86_64-3.6/cmake-build/modules/python_bindings_generator <redacted>/_skbuild/linux-x86_64-3.6/cmake-build/modules/python_bindings_generator/headers.txt

When I don't include -DPYTHON2_EXECUTABLE, the second line disappears. So it still prints that it's running the command, but doesn't actually run it.

@alalek

This comment has been minimized.

Contributor

alalek commented Jul 27, 2018

@mhsmith Thank you for investigation!

You could trace CMake scripts execution with --trace / --trace-expand CMake options.

Could you try to add ocv_assert(PYTHON_DEFAULT_EXECUTABLE) right before add_custom_command() statement? Does it trigger some error in your environment?

Did you clean build directory before checks? (to get reproducible results)

@mhsmith

This comment has been minimized.

mhsmith commented Jul 27, 2018

For me, this problem only occurs when re-running cmake in an unclean environment.

I'm building for Python 3, so I'm only including -DPYTHON3_EXECUTABLE. I delete the build directory and rerun cmake. CMakeCache.txt now contains:

PYTHON2_EXECUTABLE:FILEPATH=
PYTHON2INTERP_FOUND:INTERNAL=

... and gen_opencv_python_source.dir/build.make contains the gen2 command.

I run cmake again with the same parameters. CMakeCache.txt now contains:

PYTHON2_EXECUTABLE:FILEPATH=
PYTHON2INTERP_FOUND:INTERNAL=TRUE

... and gen_opencv_python_source.dir/build.make does not contain the gen2 command.

I added the assertion as you suggested, but it caused no errors.

I should mention that I am cross-compiling.

@KMouratidis

This comment has been minimized.

KMouratidis commented Oct 19, 2018

Switching out of the virtual env and manually compiling the erroneous file did it for me (18.04).

I did notice though that the system version of Python was changed to 2.7 instead of 3.6, which seems strange. Potential bug?

@Nijisakai

This comment has been minimized.

Nijisakai commented Nov 2, 2018

@zapcity @Guilmort you may have to manually execute the generator script (gen2.py)

Assuming you called cmake .. from the build directory you could do:

python ./modules/python/src2/gen2.py ./build/modules/python_bindings_generator ./build/modules/python_bindings_generator/headers.txt

I think the bug arises because cmake doesn't find the right python executable to run the script

Thank you very much ! This works !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment