Please allow to build just a python module

[yurivict]

On the FreeBSD we can have several python versions installed simultaneously.
The C++ library is built once, and then the python module is build for each python version.

This was done with version 0.2 with these options:
BUILD_SHARED_LIBS=ON BUILD_PYTHON_MODULE=ON BUILD_TESTS=OFF BUILD_PYTHON_TESTS=OFF BUILD_PYTHON_TUTORIALS=OFF BUILD_TINYFILEDIALOGS=OFF
This produced just one file: open3d.so.

With version 0.3.0 this doesn't work any more.

Please advise what is the right way to build just a python binding, and not the C++ library?

[yxlao]

Do you mean building two versions of python bindings together? If that is the case, we need to specify -DPYTHON_EXECUTABLE to point to the python binary and run make respectively.

The open3d.so you mentioned is the compiled python module for python 2. For python 3, the name is longer, like open3d.cpython-35m-x86_64-linux-gnu.so.

See the following screen cast for building two python versions:

[yurivict]

The problem is that this command installs other files:

$ make makeplist
include/Open3D/3rdparty/tinyfiledialogs/tinyfiledialogs.h
include/Open3D/Core/Camera/PinholeCameraIntrinsic.h
include/Open3D/Core/Camera/PinholeCameraTrajectory.h
include/Open3D/Core/ColorMap/ColorMapOptimization.h
include/Open3D/Core/Core.h
include/Open3D/Core/Geometry/Geometry.h
include/Open3D/Core/Geometry/Geometry2D.h
include/Open3D/Core/Geometry/Geometry3D.h
...
...
lib/CMake/Open3D/Open3DConfig.cmake
lib/CMake/Open3D/Open3DConfigVersion.cmake
lib/libOpen3D.so
lib/libtinyfiledialogs.a
/usr/ports/graphics/py-open3d-python/work-py27/.local/%%PYTHON_SITELIBDIR%%/open3d.so

It installs headers, main libraries, libtinyfiledialogs.a, and puts open3d.so into a wrong prefix.
It should only install open3d.so, and use the pre-installed lib/libOpen3D.so.

[yxlao]

I see, so what you're referring to is the make install step.

... puts open3d.so into a wrong prefix

The current behavior of make install installs the open3d.so the default python's user site-package path from python -m site --user-site as shown here https://github.com/IntelVCL/Open3D/blob/master/src/Python/CMakeLists.txt#L28

When you want to install the open3d.so to the correct location, first make sure that python -m site --user-site returns the desired path.

It should only install open3d.so, and use the pre-installed lib/libOpen3D.so

open3d.so is just the compiled python module, this only works with python. If we need to link C++ executable against open3d, we'll need the headers and etc.

Another question, could you clarify who installed the "pre-installed lib/libOpen3D.so" if make install only installs open3d.so. Was it done in a previous step?

[yurivict]

open3d.so should only contain the binding, not the library, and should be installed alone, so that multiple python versions could have their own bindings.

When you want to install the open3d.so to the correct location, first make sure that python -m site --user-site returns the desired path.

You should use python2.7 -c "import site; print(site.getsitepackages()[0])" instead. This returns the system site-packages location.

could you clarify who installed the "pre-installed lib/libOpen3D.so

The separate port graphics/open3d installed lib/libOpen3D.so and the headers.

[yxlao]

I see two issues here

1. What shall open3d.so contain?

open3d.so should only contain the binding, not the library

1. If you use cmake -DBUILD_SHARED_LIBS=ON, libOpen3D.so will be built. open3d.so does not contain libOpen3D.so, here's the output:

➜  ~/repo/Open3D/build (master) readelf -d lib/Python/open3d.so

Dynamic section at offset 0x1508b8 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libOpen3D.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [/home/yixing/repo/Open3D/build/lib:]
...

2. If you use cmake -DBUILD_SHARED_LIBS=OFF, libOpen3D.a will be built. open3d.so does contain libOpen3D.a. So it has the whole library embedded in.

➜  ~/repo/Open3D/build_static (master) readelf -d lib/Python/open3d.so

Dynamic section at offset 0x2daeb8 contains 35 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libGL.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libGLEW.so.1.13]
 0x0000000000000001 (NEEDED)             Shared library: [libglfw.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [libjpeg.so.8]
 0x0000000000000001 (NEEDED)             Shared library: [libjsoncpp.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpng16.so.16]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgomp.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
...

We prefer approach 2 in general as DBUILD_SHARED_LIBS=OFF is the default option . Now the python module open3d.so is self-contained. We use approach 2 for our python wheel or conda releases. In fact, we go one step further to build all dependencies from source, such that open3d.so does not depend on 3rd party libraries (e.g. libpng16.so.16) so it works out-of-the-box when user do a pip install.

2. When and where shall open3d.so be installed?

I agree with you that this part is confusing, since python -m site --user-site happens under-the-hood.

To make it more complicated, if the python is under a virtualenv, python -m site --user-site could point to location outside of the virtualenv, and import site; print(site.getsitepackages()) might not exist at all:

(py3-new) ➜  ~/repo/Open3D/build (master) which python
/home/yixing/repo/venvs/py3-new/bin/python
(py3-new) ➜  ~/repo/Open3D/build (master) python -m site --user-site
/home/yixing/.local/lib/python3.5/site-packages
(py3-new) ➜  ~/repo/Open3D/build (master) python -c "import site; print(site.getsitepackages())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'site' has no attribute 'getsitepackages'
(py3-new) ➜  ~/repo/Open3D/build (master) python -m site
sys.path = [
    '/home/yixing/repo/Open3D/build',
    '/home/yixing/repo/venvs/py3-new/lib/python35.zip',
    '/home/yixing/repo/venvs/py3-new/lib/python3.5',
    '/home/yixing/repo/venvs/py3-new/lib/python3.5/plat-x86_64-linux-gnu',
    '/home/yixing/repo/venvs/py3-new/lib/python3.5/lib-dynload',
    '/usr/lib/python3.5',
    '/usr/lib/python3.5/plat-x86_64-linux-gnu',
    '/home/yixing/repo/venvs/py3-new/lib/python3.5/site-packages',  # <== what we want!
]
USER_BASE: '/home/yixing/.local' (exists)
USER_SITE: '/home/yixing/.local/lib/python3.5/site-packages' (exists)
ENABLE_USER_SITE: False

One proposal is to separate the python module installation from the make install step. That is, make install do not install the compiled open3d.so python module. The user must do another command e.g. make install-python to install the python module, and the user must specify the explicit path for installation. Do you have some suggestions on this @takanokage @syncle ?

[yurivict]

Just to be 100% clear: I want to have a set of cmake options that would build and install only open3d.so (the python binding), which is linked to the previously installed /usr/local/lib/libOpen3D.so.

This is the outcome that is highly desired when python can be installed in several separate versions. This outcome was easily achieved with the version 0.2, but now isn't working any more. This is a general expectation for all projects that have both C++ and python interfaces, not just Open3D.

So it should be rather like a separate project (py-Open3D), that depends on the main C++ Open3D project. If you could just split it out, and create a separate project with its own setup.py, this would have been a perfect solution.

[yxlao]

I want to have a set of cmake options that would build and install only open3d.so (the python binding), which is linked to the previously installed /usr/local/lib/libOpen3D.so.

Actually we also want to build the python binding without having to install open3d.

So it should be rather like a separate project (py-Open3D)

Splitting it to a separate project will be hard to manage.

This is a general expectation for all projects that have both C++ and python interfaces, not just Open3D.

I dont' see it's a general expectation. E.g. if you build TensorFlow from source, we don't need to install TF C++ library in order to build the Python wrapper. Ref: here. It depends on a bunch of Bazel targets inside the build tree, not the installed libraries.

This is the outcome that is highly desired when python can be installed in several separate versions.

I would say this shall be our main discussion point. Let's try to help you to enable this use case. We also use several different python versions and our current approach is working fine with us. Do you have any issues blocking you from using multiple python versions?

[yurivict]

I dont' see it's a general expectation. E.g. if you build TensorFlow from source, we don't need to install TF C++ library in order to build the Python wrapper.

TensorFlow is a bad example, because bazel doesn't even have the ability to install anything.

If some project defines both C++ and python interfaces, it's only reasonable to have the python interface being in a separate project, or being completely separable with build options.

[yxlao]

Currently "building open3d library" and building "open3d python binding" are already separate. In fact, you can turn BUILD_PYTHON_MODULE=OFF, so we can see it's separated. We also tell from the video, when the python version is switched, the python binding was re-compiled, but the "open3d library" was not re-built.

The core argument is whether the python module shall depend on the installed open3d library or the open3d library in the build directory. Hopefully you can see our use case of building the python binding without installing open3d. So my opinion is to build the python binding without installing open3d.

What do you think about separating the make install step in #566 (comment) @yurivict ? Seems like the installation location of the python module is your main concern?

[yurivict]

Practically speaking, there are two choices: to rebuild the C++ library every time, and use it for binding, or just rebuild the binding and use the pre-installed library. The only problem with the former way is that it is much more resource-intense. Why would we want to rebuild the same code over and over again? This is why I am for compartmentalizing things, and building the binding alone, separately.

[yxlao]

So this is more like a new feature request: "build open3d python binding with pre-installed open3d c++ library".

Agreed with your statement a small twist. As long as we're using a pre-built library (doesn't need to be pre-installed), we can save the compilation. In fact, we're already using the pre-built library.

As we can see from the video above, when the python version is switched, the python binding was re-compiled, but the "open3d library" was not re-built. Please compare the build speed of the first and second build. So there's no waste of rebuilding the same code with our current approach.

[yurivict]

As we can see from the video above, when the python version is switched, the python binding was re-compiled, but the "open3d library" was not re-built.

Now we only need to figure out how to make it to not build libOpen3D.so even the first time. -)

[yurivict]

0.4.0 still suffers from this problem. Python binding can't be built separately.

[yxlao]

@yurivict we should be able to disable python binding by: http://www.open3d.org/docs/compilation.html#disable-python-binding

[yurivict]

I want to build it, separately, not disable.