Newly included absl headers are missing from the include path

[alsrgv]

Newly included absl headers are present in the tf-nightly Python package but do not seem to be part of the include path, and now custom op plugin compilation fails.

This has been broken between tf_nightly-1.11.0.dev20180830 and tf_nightly-1.11.0.dev20180831.

    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DHAVE_CUDA=1 -DHAVE_NCCL=1 -DHOROVOD_GPU_ALLREDUCE='N' -I/usr/local/cuda/include -I/usr/include/python2.7 -c horovod/tensorflow/mpi_ops.cc -o build/temp.linux-x86_64-2.7/horovod/tensorflow/mpi_ops.o -std=c++11 -fPIC -O2 -I/usr/local/include -pthread -Wl,-rpath -Wl,/usr/local/lib -Wl,--enable-new-dtags -L/usr/local/lib -lmpi -I/usr/local/lib/python2.7/dist-packages/tensorflow/include -D_GLIBCXX_USE_CXX11_ABI=0
    cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
    In file included from /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/lib/core/status.h:24:0,
                     from /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/framework/op_def_builder.h:25,
                     from /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/framework/op.h:23,
                     from horovod/tensorflow/mpi_ops.cc:22:
    /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/lib/core/stringpiece.h:34:38: fatal error: absl/strings/string_view.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

cc @yunxing

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce
Mobile device

[mhorikos]

I saw same issue for the horovod installation.
Workaround was to clone abseil-cpp from git. And making soft link of absl directory to your include path.

[skye]

cc @gunan

[willSapgreen]

Hello @mhorikos
does

abseil-cpp from git

mean this one
https://github.com/abseil/abseil-cpp

Thank you.

[gunan]

This is being investigated internally.
WE are currently trying to roll back the change to fix custom ops, and then we will try moving forward with a fix.

[mhorikos]

@willSapgreen

[mhorikos]

Hello @willSapgreen
Yes, it is.

[willSapgreen]

Hello @mhorikos , Thanks!!!

[alsrgv]

@gunan, thanks!

[roshan-dongre]

Trying to build tensorflow as a standalone project for C++ and getting a similar error:

g++ -std=c++11 -Wl,-rpath='$ORIGIN/lib' -Iinclude -Llib main.cpp -ltensorflow_cc -o exec
In file included from include/tensorflow/core/platform/tensor_coding.h:22:0,
from include/tensorflow/core/framework/resource_handle.h:19,
from include/tensorflow/core/framework/allocator.h:24,
from include/tensorflow/core/framework/tensor.h:20,
from include/tensorflow/cc/framework/ops.h:21,
from include/tensorflow/cc/client/client_session.h:24,
from main.cpp:9:
include/tensorflow/core/lib/core/stringpiece.h:34:38: fatal error: absl/strings/string_view.h: No such file or directory
#include "absl/strings/string_view.h"

Any workaround for this? Thanks!

[gunan]

I don't think we have support for that usage. All of our code has a lot of dependencies, and we recommend using bazel to build our code.

For the initial issue reported, I think @angersson submitted a fix. @alsrgv could you try the latest nightly package?

[alsrgv]

@gunan, the issue is fixed, thanks!

I do see a different error in Travis CI, very similar to #19375:

/usr/bin/python: Relink `/usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'

I will look into it separately next week, after I'm back from my vacation.

Update: The integration test issue is resolved.

[dingevin]

hello @mhorikos, can you describe more detailed? I have the same error, thanks.

[mhorikos]

Hello @dingevin

In your tensorflow installed top directory (that has Eigen, external, tensorflow, third_party, unsupported and etc.),

$ git clone https://github.com/abseil/abseil-cpp.git
$ ln -s abseil-cpp/absl ./absl

Then you can install horovod

[dingevin]

Hello @dingevin

In your tensorflow installed top directory (that has Eigen, external, tensorflow, third_party, unsupported and etc.),

$ git clone https://github.com/abseil/abseil-cpp.git
$ ln -s abseil-cpp/absl ./absl

Then you can install horovod

hi ， @mhorikos， thank you very much.

[dingevin]

hello @tensorflowbutler , the same issue was disappear when I compiled older versions(1.8.0). At the beginning i was just executed the command: git clone https://github.com/tensorflow/tensorflow.git, i thought it git the latest version.

[girving]

After updating to MacOS 10.14 and thus Xcode 10, I'm hitting a possibly related linker error when trying to load custom ops:

Traceback (most recent call last):
  File "./safety/debate/test_azero.py", line 4, in <module>
    from safety.debate import azero
  File "/Users/irving/openai/safety/safety/debate/azero.py", line 5, in <module>
    from safety.debate import gomoku
  File "/Users/irving/openai/safety/safety/debate/gomoku.py", line 9, in <module>
    gomoku_value = utils.load_debate_ops().gomoku_value
  File "/Users/irving/openai/safety/safety/core/utils.py", line 212, in load_debate_ops
    return tf.load_op_library(path)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 60, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: dlopen(/Users/irving/openai/safety/debate_ops.cpython-36m-darwin.so, 6): Symbol not found: __ZN10tensorflow11GetNodeAttrERKNS_9AttrSliceENSt3__117basic_string_viewIcNS3_11char_traitsIcEEEEPNS3_12basic_stringIcS6_NS3_9allocatorIcEEEE
  Referenced from: /Users/irving/openai/safety/debate_ops.cpython-36m-darwin.so
  Expected in: flat namespace
 in /Users/irving/openai/safety/debate_ops.cpython-36m-darwin.so

The custom op thinks the symbol is

tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*)

but the actual symbol is

tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::string_view, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*)

I.e., absl::string_view vs. basic_string_view.

[tensorflowbutler]

Nagging Assignee @yunxing: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[liamuk]

@girving I'm facing the same issue, did you find a solution?

[girving]

@liamuk No solution, but I haven't tried in a while. Unfortunately I can't build at all now due to #22902, so I don't know if it's still an issue.

[liamuk]

@girving Ah. If you hit it again, here's an issue I opened for it (with a workaround): #23561

[LJXLJXLJX]

Hello @dingevin

In your tensorflow installed top directory (that has Eigen, external, tensorflow, third_party, unsupported and etc.),

$ git clone https://github.com/abseil/abseil-cpp.git
$ ln -s abseil-cpp/absl ./absl

Then you can install horovod

Hello, I am working on windows10 visual studio 2015, how can I build the links?
I think there must be a better way than just adding the include directory one by one in those vs projects.

[Oceania2018]

@LJXLJXLJX The file struct should looks like below:

[AtekiRyu]

This probably helps 914f68b

[mohantym]

Hi @alsrgv!
We are checking to see if you still need help in this issue , Have you tried latest stable version TF 2.6 yet? Please create a new issue if the issue is replicating in newer versions. Thanks!

[google-ml-butler]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No