enable ENABLE_HEADLESS_RENDERING=ON cause undefined reference to `open3d::visualization::gui::Application::GetInstance

[lichangW]

Checklist

• I have searched for similar issues.
• For Python issues, I have tested with the latest development wheel.
• I have checked the release documentation and the latest documentation (for master branch).

Steps to reproduce the issue

I first cloned Open3D by:

I build C++ open3d lib from source , version: 0.16.0;
it worked well when the ENABLE_HEADLESS_RENDERING option is OFF, but when I open the Option like below and build out the lib, the bin directory in the install location is disappear:

when I build the demo (https://github.com/isl-org/Open3D/blob/master/examples/cpp/OffscreenRendering.cpp, the cpp in the example seems build with BUILD_GUI on ?) using the open3d lib(ENABLE_HEADLESS_RENDERING=ON) , it will be failed during linking to Open3D::Open3D with many undefined errors;

thanks for help

I build Open3D (on Ubuntu 18.04) with:

mkdir build
cd build
cmake -DCMAKE_FIND_ROOT_PATH=/usr/lib/aarch64-linux-gnu \
      -DCMAKE_C_COMPILER=/usr/bin/aarch64-linux-gnu-gcc \
      -DCMAKE_CXX_COMPILER=/usr/bin/aarch64-linux-gnu-g++ \
      -DCMAKE_SYSTEM_PROCESSOR=aarch64 \
      -DCMAKE_INSTALL_PREFIX=/opt/aarch64/open3d_0.16 \
      -DBUILD_SHARED_LIBS=ON \
      -DBUILD_PYTHON_MODULE=OFF \
      -DBUILD_EXAMPLES=OFF \
      -DCMAKE_BUILD_TYPE=Release \
      -DGLIBCXX_USE_CXX11_ABI=ON \
      -DENABLE_HEADLESS_RENDERING=ON \
      ..
make -j$(nproc)

when build open3d lib warning occure and the bin directory disappear when install :
the bin directory in the install location is disappear:

when build example , error occure:

Error message

/usr/bin/ld: warning: //usr/lib/aarch64-linux-gnu/libLLVM-10.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0000000
/usr/bin/ld: warning: //usr/lib/aarch64-linux-gnu/libLLVM-10.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0000000
/usr/bin/ld: warning: //usr/lib/aarch64-linux-gnu/libLLVM-10.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0000000
/usr/bin/ld: warning: //usr/lib/aarch64-linux-gnu/libLLVM-10.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0000000
CMakeFiles/easyedge_pointcloud_demo.dir/demo.cpp.o: In function `save_pointcloud2png_without_screen(int, char const**, std::shared_ptr<open3d::geometry::PointCloud>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)':
demo.cpp:(.text+0xaec): undefined reference to `open3d::visualization::gui::Application::GetInstance()'
demo.cpp:(.text+0xb00): undefined reference to `open3d::visualization::gui::Application::Initialize(int, char const**)'
demo.cpp:(.text+0xb04): undefined reference to `open3d::visualization::rendering::EngineInstance::EnableHeadless()'
demo.cpp:(.text+0xb08): undefined reference to `open3d::visualization::rendering::EngineInstance::GetInstance()'
demo.cpp:(.text+0xb10): undefined reference to `open3d::visualization::rendering::EngineInstance::GetResourceManager()'
demo.cpp:(.text+0xb38): undefined reference to `open3d::visualization::rendering::FilamentRenderer::FilamentRenderer(filament::Engine&, int, int, open3d::visualization::rendering::FilamentResourceManager&)'
demo.cpp:(.text+0xb58): undefined reference to `open3d::visualization::rendering::Open3DScene::Open3DScene(open3d::visualization::rendering::Renderer&)'
demo.cpp:(.text+0xba8): undefined reference to `open3d::visualization::rendering::Open3DScene::AddGeometry(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, open3d::geometry::Geometry3D const*, open3d::visualization::rendering::MaterialRecord const&, bool)'
demo.cpp:(.text+0xbc4): undefined reference to `open3d::visualization::rendering::Open3DScene::ShowAxes(bool)'
demo.cpp:(.text+0xbcc): undefined reference to `open3d::visualization::rendering::Open3DScene::GetCamera() const'
demo.cpp:(.text+0xc1c): undefined reference to `open3d::visualization::rendering::Open3DScene::GetCamera() const'
demo.cpp:(.text+0xcc8): undefined reference to `open3d::visualization::rendering::Open3DScene::GetView() const'
demo.cpp:(.text+0xcd4): undefined reference to `open3d::visualization::rendering::Open3DScene::GetScene() const'
demo.cpp:(.text+0xcfc): undefined reference to `open3d::visualization::gui::Application::RenderToImage(open3d::visualization::rendering::Renderer&, open3d::visualization::rendering::View*, open3d::visualization::rendering::Scene*, int, int)'
demo.cpp:(.text+0xd84): undefined reference to `open3d::visualization::rendering::Open3DScene::~Open3DScene()'
demo.cpp:(.text+0xdbc): undefined reference to `open3d::visualization::gui::Application::OnTerminate()'
collect2: error: ld returned 1 exit status

Open3D, Python and System information

- Operating system: Ubuntu 18.04 
- System architecture: x86 / arm64 
- build from source
- Compiler version (if built from source): gcc 7.5 / clang 7.0

Additional information

And we can see there is no EnableHeadless in libOpen3D.so build with ENABLE_HEADLESS_RENDERING=ON

[ssheorey]

Please don't use ENABLE_HEADLESS_RENDERING and BUILD_GUI together - they are not compatible. Use BUILD_GUI and you can do headless rendering as in this example:

https://github.com/isl-org/Open3D/blob/master/examples/python/visualization/headless_rendering.py
The C++ API is analogous. I think the C++ examples need updating.

[lichangW]

really appreciate for reply, I rebuild open3d with ENABLE_HEADLESS_RENDERING=ON and BUILD_GUI=OFF( we can see from above cmake log); do you mean we should build with BUILD_GUI=ON and ENABLE_HEADLESS_RENDERING=OFF?

from the headless_rendering.py example or the doc http://www.open3d.org/docs/release/tutorial/visualization/headless_rendering.html, I conclude that ENABLE_HEADLESS_RENDERING should be ON to use headless and BUILD_GUI should be OFF

[nubertj]

HI @ssheorey,
We have the same problem. As soon as
-DENABLE_HEADLESS_RENDERING=ON
is enabled, the OffscreenRendering.cpp example is failing with undefined reference error.

This is not observable usually, as HEADLESS_RENDERING turns off the BUILD_GUI (which of course makes sense), but this then also disables the OffscreenRendering example from being compiled, as it can be seen here:

Open3D/examples/cpp/CMakeLists.txt

Line 77 in 21e177e

if (BUILD_GUI)

My question is: what is the purpose of the OffscreenRendering when not being used with the headless mode?
@tutunarsl @pascal-roth

[adahbingee]

Also confused.
ENABLE_HEADLESS_RENDERING=ON conflict with BUILD_GUI=ON
but the offscreen example use GUI?

Once I set ENABLE_HEADLESS_RENDERING=ON and BUILD_GUI=OFF
It seems can't use GUI module in the example

Anyone success build the example?