Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ManyLinux builds with FFmpeg shared libraries. #50

Closed
techdragon opened this issue Oct 4, 2017 · 26 comments
Closed

ManyLinux builds with FFmpeg shared libraries. #50

techdragon opened this issue Oct 4, 2017 · 26 comments
Labels
enhancement

Comments

@techdragon
Copy link
Contributor

It should be possible, I'll take a look after I get #49 finished.
@techdragon techdragon mentioned this issue Oct 11, 2017
Closed
@abitrolly
Copy link

These are current settings. Nothing is compiled in statically. Installing FFmpeg doesn't help, so I guess dynamic linking won't work too.

>>> import cv2
>>> print(cv2.getBuildInformation())

General configuration for OpenCV 3.3.0 =====================================
  Version control:               3.3.0

  Platform:
    Timestamp:                   2017-09-03T09:41:40Z
    Host:                        Linux 4.4.0-83-generic x86_64
    CMake:                       3.9.0
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/gmake
    Configuration:               Release

  CPU/HW features:
    Baseline:                    SSE SSE2 SSE3
      requested:                 SSE3
    Dispatched code generation:  SSE4_1 SSE4_2 FP16 AVX AVX2
      requested:                 SSE4_1 SSE4_2 AVX FP16 AVX2
      SSE4_1 (2 files):          + SSSE3 SSE4_1
      SSE4_2 (1 files):          + SSSE3 SSE4_1 POPCNT SSE4_2
      FP16 (1 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
      AVX (5 files):             + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
      AVX2 (7 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2

  C/C++:
    Built as dynamic libs?:      NO
    C++ Compiler:                /opt/rh/devtoolset-2/root/usr/bin/c++  (ver 4.8.2)
    C++ flags (Release):         -fPIC   -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections  -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fPIC   -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections  -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /opt/rh/devtoolset-2/root/usr/bin/cc
    C flags (Release):           -fPIC   -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections  -msse -msse2 -msse3 -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fPIC   -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections  -msse -msse2 -msse3 -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):
    Linker flags (Debug):
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:          /opt/Qt4.8.7/lib/libQtGui.so /opt/Qt4.8.7/lib/libQtTest.so /opt/Qt4.8.7/lib/libQtCore.so /lib64/libz.so stdc++ dl m pthread rt
    3rdparty dependencies:       ittnotify libprotobuf libjpeg libwebp libpng libtiff libjasper IlmImf

  OpenCV modules:
    To be built:                 core flann imgproc ml objdetect photo video dnn imgcodecs shape videoio highgui superres features2d calib3d stitching videostab python2
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev java python3 ts viz

  GUI: 
    QT 4.x:                      YES (ver 4.8.7 EDITION = OpenSource)
    QT OpenGL support:           NO
    OpenGL support:              NO
    VTK support:                 NO

  Media I/O: 
    ZLib:                        /lib64/libz.so (ver 1.2.3)
    JPEG:                        libjpeg (ver 90)
    WEBP:                        build (ver encoder: 0x020e)
    PNG:                         build (ver 1.6.24)
    TIFF:                        build (ver 42 - 4.0.2)
    JPEG 2000:                   build (ver 1.900.1)
    OpenEXR:                     build (ver 1.7.1)
    GDAL:                        NO
    GDCM:                        NO

  Video I/O:
    DC1394 1.x:                  NO
    DC1394 2.x:                  NO
    FFMPEG:                      NO
      avcodec:                   NO
      avformat:                  NO
      avutil:                    NO
      swscale:                   NO
      avresample:                NO
    GStreamer:                   NO
    OpenNI:                      NO
    OpenNI PrimeSensor Modules:  NO
    OpenNI2:                     NO
    PvAPI:                       NO
    GigEVisionSDK:               NO
    Aravis SDK:                  NO
    UniCap:                      NO
    UniCap ucil:                 NO
    V4L/V4L2:                    NO/NO
    XIMEA:                       NO
    Xine:                        NO
    Intel Media SDK:             NO
    gPhoto2:                     NO

  Parallel framework:            pthreads

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Use Intel IPP:               NO
    Use Intel IPP IW:            NO
    Use Intel IPP Async:         NO
    Use VA:                      NO
    Use Intel VA-API/OpenCL:     NO
    Use Lapack:                  NO
    Use Eigen:                   NO
    Use Cuda:                    NO
    Use OpenCL:                  YES
    Use OpenVX:                  NO
    Use custom HAL:              NO

  OpenCL:                        <Dynamic loading of OpenCL library>
    Include path:                /io/opencv/3rdparty/include/opencl/1.2
    Use AMDFFT:                  NO
    Use AMDBLAS:                 NO

  Python 2:
    Interpreter:                 python (ver 2.7.13)
    Libraries:                   
    numpy:                       /opt/python/cp27-cp27mu/lib/python2.7/site-packages/numpy/core/include (ver 1.11.1)
    packages path:               /opt/python/cp27-cp27mu/lib/python2.7/site-packages

  Python 3:
    Interpreter:                 NO

  Python (for build):            python

  Java:
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Matlab:                        Matlab not found or implicitly disabled

  Tests and samples:
    Tests:                       NO
    Performance tests:           NO
    C/C++ Examples:              NO

  Install path:                  /usr/local

  cvconfig.h is in:              /io/opencv/build
-----------------------------------------------------------------

@gitcoinbot
Copy link

This issue now has a funding of 0.045 ETH (21.0 USDT) attached to it.

  • If you would like to work on this issue you can claim it here.
  • If you've completed this issue and want to claim the bounty you can do so here

@abitrolly
Copy link

I've added some ether as a bonus to make it happen. )

@skvark
Copy link
Member

skvark commented Nov 26, 2017

None of the current builds are linked/built with FFmpeg. I will build new layers with FFmpeg and libjpeg-turbo (#52) to the Docker images as soon as I find some more free time. Thanks for the bonus 👍

@owocki
Copy link

owocki commented Nov 27, 2017

@abitrolly thanks :)

@owocki
Copy link

owocki commented Dec 4, 2017 •
edited

hi from gitcoin.co -- doing my best to source someone to turn this around. will keep yall posted

@skvark
Copy link
Member

skvark commented Dec 14, 2017

Just a quick status update. I have now built new layers which include FFmpeg to the manylinux containers. The OpenCV build is not working yet (linker does not find FFmpeg libs) but I hope I can get that fixed in the next few days.

@abitrolly
Copy link

The OpenCV build is not working yet (linker does not find FFmpeg libs)

Is there more information about that? Upstream bug maybe?

@skvark
Copy link
Member

skvark commented Dec 15, 2017

Current ouput from Travis:

--     FFMPEG:                      YES
--       avcodec:                   YES (ver 58.6.103)
--       avformat:                  YES (ver 58.3.100)
--       avutil:                    YES (ver 56.5.100)
--       swscale:                   YES (ver 5.0.101)
--       avresample:                NO

Error:

[100%] Linking CXX shared module ../../lib/python3/cv2.cpython-35m-x86_64-linux-gnu.so
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-CentOS-linux/4.8.2/ld: cannot find -lavcodec
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-CentOS-linux/4.8.2/ld: cannot find -lavformat
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-CentOS-linux/4.8.2/ld: cannot find -lavutil
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-CentOS-linux/4.8.2/ld: cannot find -lswscale
collect2: error: ld returned 1 exit status
make[3]: *** [lib/python3/cv2.cpython-35m-x86_64-linux-gnu.so] Error 1
make[2]: *** [modules/python3/CMakeFiles/opencv_python3.dir/all] Error 2
make[1]: *** [modules/python3/CMakeFiles/opencv_python3.dir/rule] Error 2
make: *** [opencv_python3] Error 2

It might also be related to custom install location. Full build logs are at Travis: https://travis-ci.org/skvark/opencv-python/builds/316114040

@skvark
Copy link
Member

skvark commented Dec 15, 2017

Builds are working now. I added FFmpeg lib path to LDFLAGS.

@skvark skvark mentioned this issue Dec 15, 2017
Closed
@gitcoinbot
Copy link

The funding of 0.045 ETH (30.83 USD) attached has been claimed by @@skvark.

@@skvark, please leave a comment to let the funder (@abitrolly) and the other parties involved your implementation plan. If you don't leave a comment, the funder may expire your claim at their discretion.

@owocki
Copy link

owocki commented Dec 18, 2017

👋 @skvark

@skvark
Copy link
Member

skvark commented Dec 18, 2017

@abitrolly @owocki I'll merge these changes during this week after I have done some local tests with the packages.

@abitrolly
Copy link

@skvark great. Looking forward to see this committed. What could help to fix this sooner? Would there be less hassle if CMake could provide a better error message, like checking for libraries in advance?

@skvark
Copy link
Member

skvark commented Dec 18, 2017

Probably would get fixed sooner if I had more time to focus on this :) But I'll try to get this done today or tomorrow (and couple of other issues too). And yes, it would be a lot faster to test the build if the compilation failed instantly vs in the very end.

@owocki
Copy link

owocki commented Dec 21, 2017

great to see the progress here.. keep it up @skvark !

@skvark
Copy link
Member

skvark commented Dec 21, 2017 •
edited

The changes are already in master but I'll close #53 and #48 before release.

@hroemer
Copy link

hroemer commented Dec 22, 2017

If you prefer building the current OpenCV version for Python 3 including video support on your own locally, I provided a simple CLI tool on PyPI named cvbuilder that downloads the official sources, configures and compiles them in a virtual environment taking installed libraries into account.

@skvark
Copy link
Member

skvark commented Dec 22, 2017

Thanks. The local build support in this repository will be done via setup.py and scikit-build (a future proof tool targeting for upcoming PEP 518, see https://www.python.org/dev/peps/pep-0518/ and https://github.com/scikit-build/scikit-build).

The local build support is not yet there but first transition step will be after #60 has been merged. After that will follow also sdist so users will be able to build this package locally with pip install opencv-python if their system is missing prebuilt wheels or force an sdist build with pip install opencv-python --no-binary.

@skvark skvark closed this as completed Dec 23, 2017
@skvark
Copy link
Member

skvark commented Dec 23, 2017

@abitrolly @owocki New releases with FFmpeg should be soon at PyPI.

@owocki
Copy link

owocki commented Dec 27, 2017

@abitrolly let us know if its time to payout the bounty

@abitrolly
Copy link

abitrolly commented Dec 28, 2017 •
edited

Sorry for the delay. Christmas and all the stuff. =) Checking that it works..

$ virtualenv xxx
New python executable in /home/anatoli/a/xxx/bin/python2
Also creating executable in /home/anatoli/a/xxx/bin/python
Installing setuptools, pip, wheel...done.
$ cd xxx
$ ./bin/pip install opencv-python
Collecting opencv-python
  Downloading opencv_python-3.3.1.11-cp27-cp27mu-manylinux1_x86_64.whl (24.7MB)
    100% |████████████████████████████████| 24.7MB 32kB/s 
Collecting numpy>=1.11.1 (from opencv-python)
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Installing collected packages: numpy, opencv-python
Successfully installed numpy-1.13.3 opencv-python-3.3.1.11

Installation is ok. Now I need to find some example to run..

@abitrolly
Copy link

I can't find anything to run right now and there is little time on my back. Info is reporting FFMPEG is compiled in, so I see no reason to delay the award anymore. )

>>> import cv2
>>> print(cv2.getBuildInformation())
General configuration for OpenCV 3.3.1 =====================================
  Version control:               3.3.1-dirty
...
  Video I/O:
    DC1394 1.x:                  NO
    DC1394 2.x:                  NO
    FFMPEG:                      YES
      avcodec:                   YES (ver 58.6.103)
      avformat:                  YES (ver 58.3.100)
      avutil:                    YES (ver 56.5.100)
      swscale:                   YES (ver 5.0.101)
      avresample:                NO
...
...

@gitcoinbot
Copy link

The funding of 0.045 ETH (30.61 USD) attached to this issue has been approved & issued to @skvark.

@skvark
Copy link
Member

skvark commented Dec 28, 2017

Thanks!

@cachitas
Copy link

I tested yesterday in Ubuntu 17.10 x86_64 with Python 3.6.3.

$ pip freeze
opencv-python==3.3.1.11
...

Before I had to link to an OpenCV installed against my Python version with FFMPEG.

Now it just works out of the box 😃 Thanks everyone!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@owocki @skvark @techdragon @cachitas @abitrolly @hroemer @gitcoinbot