Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

undefined symbol error for _dataset_ops.so on rasp pi #17619

Closed
matpalm opened this issue Mar 11, 2018 · 12 comments

Comments

Projects
None yet
3 participants
@matpalm
Copy link

commented Mar 11, 2018

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
$ uname -a
Linux pi_c 4.4.13-v7+ #894 SMP Mon Jun 13 13:13:27 BST 2016 armv7l GNU/Linux
$ cat /etc/issue
Raspbian GNU/Linux 8 \n \l
  • TensorFlow installed from (source or binary):

installed from nightly wheel http://ci.tensorflow.org/view/Nightly/job/nightly-pi-python3/

  • TensorFlow version (use command below):
$ python3 -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
v1.6.0-rc1-1295-g851c289 1.6.0
  • Python version:
$ python3 --version
Python 3.4.2
  • CUDA/cuDNN version: N/A running cpu on pi

Have I written custom code NA

Bazel version NA

GPU model and memory NA

  • Exact command to reproduce:
$ python3 -c "import tensorflow.contrib.slim as slim"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/__init__.py", line 33, in <module>
    from tensorflow.contrib import data
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/__init__.py", line 63, in <module>
    from tensorflow.contrib.data.python.ops.error_ops import ignore_errors
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/error_ops.py", line 20, in <module>
    from tensorflow.contrib.data.python.ops import contrib_op_loader  # pylint: disable=unused-import
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/contrib_op_loader.py", line 24, in <module>
    resource_loader.get_path_to_datafile("../../_dataset_ops.so"))
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/util/loader.py", line 56, in load_op_library
    ret = load_library.load_op_library(path)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/load_library.py", line 58, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/../../_dataset_ops.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl15AllocateAlignedEj

Describe the problem

Import of slim fails with .so error (as described above). doubt it's actually slim, it was just the first import to reference _dataset_ops.so

@tensorflowbutler

This comment has been minimized.

Copy link
Member

commented Mar 11, 2018

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
Bazel version
GPU model and memory

@drpngx

This comment has been minimized.

Copy link
Contributor

commented Mar 11, 2018

It looks like you are missing the protobuf library. Can you try to locate it?

@matpalm

This comment has been minimized.

Copy link
Author

commented Mar 11, 2018

the following snippet of code runs in the same environment. i think this shows that, generally, proto buffers are ok ? (this was the simplest tf related proto buff bit of code i could think of)

$ cat test.py
#!/usr/bin/env python3
import tensorflow as tf
c = tf.ConfigProto()
c.intra_op_parallelism_threads = 5
print(c)
 $ ./test.py
intra_op_parallelism_threads: 5

there might be something specific with contrib though; as mentioned here
https://petewarden.com/2017/08/20/cross-compiling-tensorflow-for-the-raspberry-pi/#comment-108790

@drpngx

This comment has been minimized.

Copy link
Contributor

commented Mar 11, 2018

The error you got is this:

tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/../../_dataset_ops.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl15AllocateAlignedEj

which says that it can't load the protobuf runtime.

I just saw what you are trying to do. You can't import directly. You have to use import tensorflow as tf, then use tf.contrib.slim.

@matpalm

This comment has been minimized.

Copy link
Author

commented Mar 11, 2018

that's true. i was just looking for a minimal reproduction case

$ cat test.py
#!/usr/bin/env python3
import tensorflow as tf
import tensorflow.contrib.slim as slim

$ ./test.py
Traceback (most recent call last):
  File "./test.py", line 3, in <module>
    import tensorflow.contrib.slim as slim
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/__init__.py", line 33, in <module>
    from tensorflow.contrib import data
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/__init__.py", line 63, in <module>
    from tensorflow.contrib.data.python.ops.error_ops import ignore_errors
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/error_ops.py", line 20, in <module>
    from tensorflow.contrib.data.python.ops import contrib_op_loader  # pylint: disable=unused-import
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/contrib_op_loader.py", line 24, in <module>
    resource_loader.get_path_to_datafile("../../_dataset_ops.so"))
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/util/loader.py", line 56, in load_op_library
    ret = load_library.load_op_library(path)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/load_library.py", line 58, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/../../_dataset_ops.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl15AllocateAlignedEj

the code i first saw this with was a much bigger piece https://github.com/matpalm/bnn/blob/master/predict.py

(developed on my desktop, but fails on the pi)

$ ./predict.py
$ ./predict.py 
Traceback (most recent call last):
  File "./predict.py", line 9, in <module>
    import model
  File "/home/pi/dev/bnn/model.py", line 5, in <module>
    import tensorflow.contrib.slim as slim
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/__init__.py", line 33, in <module>
    from tensorflow.contrib import data
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/__init__.py", line 63, in <module>
    from tensorflow.contrib.data.python.ops.error_ops import ignore_errors
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/error_ops.py", line 20, in <module>
    from tensorflow.contrib.data.python.ops import contrib_op_loader  # pylint: disable=unused-import
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/contrib_op_loader.py", line 24, in <module>
    resource_loader.get_path_to_datafile("../../_dataset_ops.so"))
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/util/loader.py", line 56, in load_op_library
    ret = load_library.load_op_library(path)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/load_library.py", line 58, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/../../_dataset_ops.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl15AllocateAlignedEj
@matpalm

This comment has been minimized.

Copy link
Author

commented Mar 11, 2018

and, just for my sanity, regarding potential lazy loading / eval of the proto libraries or something...

$ cat test.py
#!/usr/bin/env python3
import tensorflow as tf
c = tf.ConfigProto()
c.intra_op_parallelism_threads = 5
print(c)
import tensorflow.contrib.slim as slim

$ ./test.py
intra_op_parallelism_threads: 5

Traceback (most recent call last):
..... AS BEFORE
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/../../_dataset_ops.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl15AllocateAlignedEj
@drpngx

This comment has been minimized.

Copy link
Contributor

commented Mar 11, 2018

OK. I'm not 100% sure what it is, but you're really not supposed to import directly from tensorflow package. Just don't do that and you should be fine. Also, we are deprecating high level APIs in favor of tf.keras.

@matpalm

This comment has been minimized.

Copy link
Author

commented Mar 12, 2018

Fair enough. This might be the nail in the coffin for me & slim... I'll use this as an excuse to upgrade.

@matpalm matpalm closed this Mar 12, 2018

@matpalm

This comment has been minimized.

Copy link
Author

commented Mar 12, 2018

(oh, i did forget to mention, thanks for the attempted workaround but referencing slim through tf import still didn't work)

$ cat test.py
#!/usr/bin/env python3
import tensorflow as tf
print(dir(tf.contrib.slim.conv2d))

$ ./test.py
Traceback (most recent call last):
  File "./test.py", line 3, in <module>
    print(dir(tf.contrib.slim.conv2d))
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/util/lazy_loader.py", line 53, in __getattr__
    module = self._load()
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/util/lazy_loader.py", line 42, in _load
    module = importlib.import_module(self.__name__)
  File "/usr/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/__init__.py", line 33, in <module>
    from tensorflow.contrib import data
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/__init__.py", line 63, in <module>
    from tensorflow.contrib.data.python.ops.error_ops import ignore_errors
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/error_ops.py", line 20, in <module>
    from tensorflow.contrib.data.python.ops import contrib_op_loader  # pylint: disable=unused-import
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/contrib_op_loader.py", line 24, in <module>
    resource_loader.get_path_to_datafile("../../_dataset_ops.so"))
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/util/loader.py", line 56, in load_op_library
    ret = load_library.load_op_library(path)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/load_library.py", line 58, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.4/dist-packages/tensorflow/contrib/data/python/ops/../../_dataset_ops.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl15AllocateAlignedEj
@drpngx

This comment has been minimized.

Copy link
Contributor

commented Mar 12, 2018

Oh, right I forgot about the lazy loading of contrib. It is basically loaded on demand. There is something broken in your install. It's not to do with slim, but with tf.contrib.data. If you try to use it directly, it will likely break.

@matpalm

This comment has been minimized.

Copy link
Author

commented Mar 12, 2018

agreed it's something broken with that nightly built wheel (from http://ci.tensorflow.org/view/Nightly/job/nightly-pi-python3/) when i built from source (awhile back) it was a pain to get working, but didn't have this problem.

thought it might also be something weird on that pi (since i've built tensorflow on it multiple times) but i tried to install this wheel on a completely fresh install of jessie (on another pi) and got the same problem

@matpalm

This comment has been minimized.

Copy link
Author

commented Mar 12, 2018

and agree it's not slim (that was just the first thing hitting this in my original code (see initial report)) . so i guess moving to whatever other framework for layers isn't going to help. (since they will all reference data i guess)

i'm going to try to use a frozen graph (in the hope the frozen graph doesn't use data) and if that doesn't work i'll give up on the wheel for now & go back to building from source.

mrry added a commit that referenced this issue Mar 19, 2018

Adds missing protobuf dep to tf.contrib.data ops. (#17840)
* Adds missing protobuf dep to tf.contrib.data ops.

I think this will help resolve the following:
tensorflow/serving#421
tensorflow/serving#684
#17619

Or at least I was experiencing a similar issue and this change resolved it for me in my local repo.

* s/third_party//

StanislawAntol pushed a commit to StanislawAntol/tensorflow that referenced this issue Mar 23, 2018

Adds missing protobuf dep to tf.contrib.data ops. (tensorflow#17840)
* Adds missing protobuf dep to tf.contrib.data ops.

I think this will help resolve the following:
tensorflow/serving#421
tensorflow/serving#684
tensorflow#17619

Or at least I was experiencing a similar issue and this change resolved it for me in my local repo.

* s/third_party//
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.