Breaks after upgrading to TensorFlow 1.2.0

[alshedivat]

While building docker images, noticed that sonnet breaks with the new version of TF 1.2.0.

root@a28ad161d26d:/# python
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sonnet as snt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.5/site-packages/sonnet/__init__.py", line 109, in <module>
    from sonnet.python.ops.resampler import resampler
  File "/opt/conda/lib/python3.5/site-packages/sonnet/python/ops/resampler.py", line 33, in <module>
    tf.resource_loader.get_path_to_datafile("_resampler.so"))
  File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/framework/load_library.py", line 64, in load_op_library
    None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: /opt/conda/lib/python3.5/site-packages/sonnet/python/ops/_resampler.so: undefined symbol: _ZN10tensorflow15shape_inference16InferenceContext15WithRankAtLeastENS0_11ShapeHandleEiPS2_

TensorFlow was installed via pip, sonnet was compiled and installed from sources. Perhaps, updating the TF submodule with headers should solve this.

[alshedivat]

Compiling against newer headers solves the problem though the same issue occurs when TF is outdated (i.e., < 1.2.0).

[malcolmreynolds]

@alshedivat we've pushed a new version with the tensorflow submodule updated, can you try again?

[alshedivat]

@malcolmreynolds Thanks, it works now with TF 1.2.0, but seems to be backwards incompatible (as mentioned in the comment above). If I use TF < 1.2.0, I get the following error:

$ python
Python 3.5.2 (default, Jun  6 2017, 00:26:23)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sonnet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/maruan/.pyenv/versions/venv3.5/lib/python3.5/site-packages/sonnet/__init__.py", line 111, in <module>
    from sonnet.python.ops.resampler import resampler
  File "/Users/maruan/.pyenv/versions/venv3.5/lib/python3.5/site-packages/sonnet/python/ops/resampler.py", line 33, in <module>
    tf.resource_loader.get_path_to_datafile("_resampler.so"))
  File "/Users/maruan/.pyenv/versions/venv3.5/lib/python3.5/site-packages/tensorflow/python/framework/load_library.py", line 64, in load_op_library
    None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: dlopen(/Users/maruan/.pyenv/versions/venv3.5/lib/python3.5/site-packages/sonnet/python/ops/_resampler.so, 6): Symbol not found: __ZN10tensorflow15shape_inference16InferenceContext15WithRankAtLeastENS0_11ShapeHandleExPS2_
  Referenced from: /Users/maruan/.pyenv/versions/venv3.5/lib/python3.5/site-packages/sonnet/python/ops/_resampler.so
  Expected in: flat namespace
 in /Users/maruan/.pyenv/versions/venv3.5/lib/python3.5/site-packages/sonnet/python/ops/_resampler.so

I am not sure if there is an easy workaround, but you may want to mention in the readme that the current version supports only the latest TF release and update setup.py. Hope this helps.

[prijatelj]

I get the same error as alshedivat's latest post, but for the latest version of sonnet with latest version of tensorflow installed. I am on Mac OSX 10.12.5. I reinstalled tensorflow from sonnet's recursive git repository of tensorflow (1.2) and it works fine. I then installed sonnet as described in README, but this is what I get. I am outside of install directory. This is with most recent sonnet and sonnet's tensorflow, I did this last night. I'd love to get this working. Should this be a separate issue, or should we try to recreate these errors on more systems for a better consensus?

>>> import sonnet as snt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dsp/virtualenv/tf/lib/python2.7/site-packages/sonnet/__init__.py", line 111, in <module>
    from sonnet.python.ops.resampler import resampler
  File "/Users/dsp/virtualenv/tf/lib/python2.7/site-packages/sonnet/python/ops/resampler.py", line 33, in <module>
    tf.resource_loader.get_path_to_datafile("_resampler.so"))
  File "/Users/dsp/virtualenv/tf/lib/python2.7/site-packages/tensorflow/python/framework/load_library.py", line 64, in load_op_library
    None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: dlopen(/Users/dsp/virtualenv/tf/lib/python2.7/site-packages/sonnet/python/ops/_resampler.so, 6): Symbol not found: __ZN10tensorflow14TensorShapeRep12SlowCopyFromERKS0_
  Referenced from: /Users/dsp/virtualenv/tf/lib/python2.7/site-packages/sonnet/python/ops/_resampler.so
  Expected in: flat namespace
 in /Users/dsp/virtualenv/tf/lib/python2.7/site-packages/sonnet/python/ops/_resampler.so

[Shuolongbj]

FYI.
cd sonnet/tensorflow
git checkout r1.2
./configure
bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pip uninstall tensorflow
pip install /tmp/tensorflow_pkg/tensorflow-1.2.0-cp36-cp36m-macosx_10_7_x86_64.whl
cd ../
bazel build --config=opt //:install
bazel-bin/install /tmp/sonnet_pkg
pip uninstall sonnet
pip install /tmp/sonnet_pkg/sonnet-1.2-py3-none-any.whl

[prijatelj]

Thank you, Shuolongbj. Fortuneately, your post revealed one slight mishap, and I remembered another. My only errors were I did not checkout r1.2 in tensorflow (I instead checked out master), and when I did the configure files for bazel in Tensorflow, I realize I called the opt flags as command line arguments, rather than specify them in the configure file of tensorflow. This potentially desynchronized the Tensorflow and Sonnet configurations and probably resulted in the error. I can confirm that sonnet 1.2 installs successfully on mac with Tensorflow 1.2.0.

I added these flags as they are when the tensorflow configure file asks for optimizatoin flags to optimize my build for my mac (this is for my personal machine, these flags may no apply specifically to your machine. Note the syntax though '-m[flag]'):
-mavx -mavx2 -mfma -msse4.2

[malcolmreynolds]

Closing this issue as everyone in the thread seems to have the situation resolved. Please reopen if necessary.