Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNDEBUG flag in cmake doesn't link libtensorflow_cc.so file #58

Closed
achalshah20 opened this issue Feb 28, 2018 · 7 comments
Closed

DNDEBUG flag in cmake doesn't link libtensorflow_cc.so file #58

achalshah20 opened this issue Feb 28, 2018 · 7 comments

Comments

@achalshah20
Copy link
Contributor

achalshah20 commented Feb 28, 2018

Hi,
In your example cmake file I just added this line

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DNDEBUG")

and it breaks the program. I am getting error

"No session factory registered for the given session options: {target: "" config: } Registered factories are {}."

Then I did "ldd example", and here is the result:

linux-vdso.so.1 => (0x00007ffe679b1000) libtensorflow_framework.so => /usr/lib/tensorflow_cc/libtensorflow_framework.so (0x00007f3ccb5d8000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f3ccb256000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3ccb040000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3ccac76000) libcublas.so.9.1 => /usr/local/cuda-9.1/lib64/libcublas.so.9.1 (0x00007f3cc7555000) libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1 (0x00007f3cc69b5000) libcudnn.so.7 => /usr/lib/x86_64-linux-gnu/libcudnn.so.7 (0x00007f3cb5a24000) libcufft.so.9.1 => /usr/local/cuda-9.1/lib64/libcufft.so.9.1 (0x00007f3cae537000) libcurand.so.9.1 => /usr/local/cuda-9.1/lib64/libcurand.so.9.1 (0x00007f3caa5b4000) libcudart.so.9.1 => /usr/local/cuda-9.1/lib64/libcudart.so.9.1 (0x00007f3caa346000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3caa142000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3ca9e39000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3ca9c1c000) /lib64/ld-linux-x86-64.so.2 (0x00007f3ccc511000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f3ca9a14000) libnvidia-fatbinaryloader.so.390.30 => /usr/lib/nvidia-390/libnvidia-fatbinaryloader.so.390.30 (0x00007f3ca97c8000)

libtensorflow_cc.so is missing and if I remove -DNDEBUG flag, it works as expected. It might be issue from tensorflow compilation thing but I am not entirely sure. Can you please help me understand why -DNDEBUG is causing this problem? Thanks!

@FloopCZ
Copy link
Owner

FloopCZ commented Mar 1, 2018

Hi, this behavior is strange, but I don't really know what is wrong. Fortunately, you don't need to use NDEBUG macro, since tensorflow is already configured for optimized build here.

@FloopCZ FloopCZ closed this as completed Mar 1, 2018
@FloopCZ
Copy link
Owner

FloopCZ commented Mar 1, 2018

Sorry, you are talking about the example CMakeLists, not the tensorflow_cc CMakeLists. That may cause some issues. Can you please try to build the example with -DCMAKE_BUILD_TYPE=Release and tell me whether it produces the same error?

@FloopCZ FloopCZ reopened this Mar 1, 2018
@achalshah20
Copy link
Contributor Author

achalshah20 commented Mar 1, 2018

Thanks!

So, I did following steps

cd example  

mkdir build  

cd build  

cmake -DCMAKE_BUILD_TYPE=Release  

make -j  

Then I did "ldd example" and I couldn't find the libtensorflow_cc.so file but it can find libtensorflow_framework.so. Moreover, the same error if I execute ./example.

"No session factory registered for the given session options: {target: "" config: } Registered factories are {}."

I am not able to understand how -DNDEBUG flag affects the library linking part!?

@FloopCZ
Copy link
Owner

FloopCZ commented Mar 2, 2018

Hi, thank you. I was looking into it, but unfortunately, I have not been able to solve it.

Tensorflow uses global register objects. I believe the problem is that without the NDEBUG symbol, the global register objects that appear to be unused get optimized out by the compiler. Unfortunately, I have no idea how to avoid it.

Does anyone have a suggestion?

Relevant issues: tensorflow/tensorflow#3308 tensorflow/tensorflow#6396 tensorflow/tensorflow#4242

@FloopCZ
Copy link
Owner

FloopCZ commented Mar 3, 2018

@achalshah20 It appears that monolithic build solves the issue, can you please try #64, whether it helps you as well?

@achalshah20
Copy link
Contributor Author

Hi @FloopCZ , Yes it works. Thanks for the fix!
I am still wondering what monolithic build has to do with debug flags?

@FloopCZ
Copy link
Owner

FloopCZ commented Mar 7, 2018

Monolithic build merges all the libraries into a single library libtensorflow_cc.so (i.e., no extra libtensorflow_framework.so). This single library has to be linked, so we avoid optimizing out the global registers. It is a little bit hacky, though. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants