Python import error, undefined symbol: _Z14tf_git_versionv

[joshsuihn]

NOTE: Only file GitHub issues for bugs and feature requests. All other topics will be closed.

Environment info

Operating System: Ubuntu16.04

Installed version of CUDA and cuDNN: 8.0 and 5

Python 2.7.12 (default, Jul 1 2016, 15:12:24)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import tensorflow
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/tensorflow/init.py", line 23, in
from tensorflow.python import *
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/init.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 28, in
_pywrap_tensorflow = swig_import_helper()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so: undefined symbol: _Z14tf_git_versionv

I'm getting this error above with r0.10, can you please give me some insights or solutions to this?

Thanks

[drpngx]

Can you attach the copy of:

nm /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so

Did you build this yourself or did you download a binary distribution?

[drpngx]

I am suspecting that this is an older version of the so file, and since we introduced the tf.GIT_VERSION later, you have the new python files loading the old shared object.

[joshsuihn]

I built this myself with this commit:
git clone https://github.com/tensorflow/tensorflow
git checkout a5f8f42

Yes, TF version is r0.10.

This is part of the logs with nm /usr/.../_pywrap_tensorflow.so

0000000006e4b078 b g_initializations
0000000006e4d950 b g_initialized
00000000000000b8 b g_initialized_sigmask
0000000006e4b880 b g_init_mu
0000000006e4bac8 b g_ipv6_loopback_available
0000000000b85d70 t __git_version___swigconstant
00000000065dfb58 a GLOBAL_OFFSET_TABLE
0000000000b0b928 t _GLOBAL__sub_I_adjust_contrast_op.cc
0000000000b157e8 t _GLOBAL__sub_I_aggregate_ops.cc
0000000000b7cfd0 t _GLOBAL__sub_I_algorithm.cc
0000000000b7f9a8 t _GLOBAL__sub_I_allocation_description.pb.cc
0000000000b7eb48 t _GLOBAL__sub_I_allocator.cc
...
0000000000b891b8 t _Z11IsDirectoryRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEP9TF_Status
0000000000b89b08 T _Z12AppendToFileRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN10tensorflow12WritableFileEP9TF_Status
U _Z14tf_git_versionv
0000000000b8a4b0 t _Z16GetMatchingFilesRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEP9TF_Status
0000000000b88078 t _Z16ReadFileToStringRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEP9TF_Status

[drpngx]

Can you check the dates on the so file and the python file? Are you using virtualenv?

[joshsuihn]

No I'm not using virtualenv.

$ ls -alh /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
-rwxr-xr-x 1 root staff 129M Oct 31 04:05 /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so

$ ls -al /usr/bin/python
lrwxrwxrwx 1 root root 9 Dec 10 2015 /usr/bin/python -> python2.7

ls -al /usr/bin/python2.7
-rwxr-xr-x 1 root root 3111304 Jul 2 20:31 /usr/bin/python2.7

Actually, I just reproduced build procedures that I've already been through before.

[drpngx]

OK, it looks recent enough. nm output, please.

[joshsuihn]

The nm output is too big (24 mb txt file) Is there any specific lines you want to check?
I can search a few lines with "_Z14tf_git_versionv"
0000000000b891b8 t _Z11IsDirectoryRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEP9TF_Status
0000000000b89b08 T _Z12AppendToFileRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN10tensorflow12WritableFileEP9TF_Status
U _Z14tf_git_versionv
0000000000b8a4b0 t _Z16GetMatchingFilesRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEP9TF_Status
0000000000b88078 t _Z16ReadFileToStringRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEP9TF_Status

[joshsuihn]

Is this related to gcc version ? or c++ mingling?

[drpngx]

Yes, it shows as U, which means undefined. I'd need to check later, but I'm guessing that the script failed to define the symbol. Could you bazel clean, to be sure? See what happens to the tf_version_info_generate. It's supposed to put something in bazel-genfiles/tensorflow/util/version_info.cc. That should have the git version in there. If it's there, check the lib file and so on.

[joshsuihn]

Ok, Thanks, I'm rebuilding after bazel clean now. Let me check the path and the file then will let you know.

[joshsuihn]

I can't find version_info.cc file, neither util dir. :(
$ ls ./bazel-genfiles/tensorflow/
contrib core models python

[drpngx]

Do you have this?

bazel-bin/tensorflow/core/_objs/version_lib/tensorflow/core/util/version_info.o

What does

nm bazel-bin/tensorflow/libtensorflow.so|grep _git_version

say?

[vrv]

I'd also try to upgrade to 0.11 -- I believe there might have been some git_version issues in 0.10 that have been long since fixed.

[joshsuihn]

I built tensorflow r0.10 on aarch64 with my own patch. In the patch,
I had to uncomment ":version_lib" in tensorflow/core/BUILD due to "duplicate version" error.
With the same version and commit, 2 - 3 weeks later, I tried building again, but I had the issue mentioned above. I don't still know why, but anyway, it turns out ":version_lib" is needed at this moment, again.

Thank you for your help!

[drpngx]

@jart FYI