tensorflow/core/framework/op_def.pb.h:10:40: fatal error: google/protobuf/port_def.inc: No such file or directory #27697

alsrgv · 2019-04-09T22:10:07Z

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): use plugin
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): tensorflow/tensorflow:latest-gpu-py3
TensorFlow version (use command below): 1.14.1-dev20190409
Python version: 3.5
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

You can collect some of this information using our environment capture script
You can also obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the current behavior
As of 1.14.1-dev20190409, custom plugins fail to build with the following error:

  In file included from /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/framework/op_def_builder.h:24:0,
                   from /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/framework/op.h:23,
                   from horovod/tensorflow/mpi_ops.cc:22:
  /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/framework/op_def.pb.h:10:40: fatal error: google/protobuf/port_def.inc: No such file or directory
   #include <google/protobuf/port_def.inc>
                                          ^
  compilation terminated.

This seems to be related to 6168f47. It appears that not all the files are placed in the proper locations after the installation.

Describe the expected behavior
Plugins should build successfully.

Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.

$ docker run -it --rm tensorflow/tensorflow:latest-gpu-py3
# apt install -y mpich
# HOROVOD_WITH_TENSORFLOW=1 pip install -v horovod

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

cc @martinwicke @gunan

The text was updated successfully, but these errors were encountered:

seanpmorgan · 2019-04-10T11:58:28Z

This is also breaking https://github.com/tensorflow/addons on tf2-nightly.

Logs:
https://source.cloud.google.com/results/invocations/1fcf8227-5161-4cb8-a017-a1a906c9429d/targets/tensorflow_addons%2Fubuntu%2Fgpu%2Fpy3%2Fcontinuous/log

qlzh727 · 2019-04-10T14:36:52Z

Assigning to @yifeif who reviewed cl/242492263, and also cc @gunan.

angerson · 2019-04-10T18:26:28Z

Un-assigning so this'll get into our GitHub triaging system (pinging @rthadur for this).

angerson · 2019-04-10T18:28:52Z

Also: @meteorcloudy

meteorcloudy · 2019-04-10T20:10:35Z

@seanpmorgan @gunan Is tf addons somehow using this BUILD file?

tensorflow/third_party/systemlibs/protobuf.BUILD

Lines 15 to 25 in 3fb94ab

    
           HEADERS = [ 
        
               "google/protobuf/any.pb.h", 
        
               "google/protobuf/any.proto", 
        
               "google/protobuf/arena.h", 
        
               "google/protobuf/compiler/importer.h", 
        
               "google/protobuf/descriptor.h", 
        
               "google/protobuf/descriptor.pb.h", 
        
               "google/protobuf/descriptor.proto", 
        
               "google/protobuf/duration.pb.h", 
        
               "google/protobuf/duration.proto", 
        
               "google/protobuf/dynamic_message.h",

@acozzette Dose it make sense to you to add google/protobuf/port_def.inc here?

seanpmorgan · 2019-04-10T20:34:52Z

We don't use the BUILD file itself. We link to the pre-compiled libtensorflow_framework.

It looks like the op_def.pb header file is referencinng a file which is not where it is expected after the new protobuf install.

tensorflow/core/framework/op_def.pb.h:10:40: fatal error: google/protobuf/port_def.inc: No such file or directory
 #include <google/protobuf/port_def.inc>

alsrgv · 2019-04-10T21:10:40Z

Same here, we're not using Bazel.

meteorcloudy · 2019-04-11T08:26:06Z

Looks like the problem is when creating the pip package, we missed to copy header files that has .inc extension.

tensorflow/tensorflow/tools/pip_package/build_pip_package.sh

Lines 128 to 130 in 1cdae49

    
           for header in $(find protobuf_archive -name \*.h); do 
        
             mkdir -p "${TMPDIR}/google/$(dirname ${header})" 
        
             cp "$header" "${TMPDIR}/google/$(dirname ${header})/"

I'll send a fix for this.

tensorflow-bot · 2019-04-11T17:37:18Z

Are you satisfied with the resolution of your issue?
Yes
No

seanpmorgan · 2019-04-12T13:39:23Z

Still seeing this issue in the 20190412 nightly. Was this merged in for that nightly? If so I'm not sure the correct fix is in place.

https://files.pythonhosted.org/packages/d4/c9/0fb753c0d182cfa10294edaa610d19f236da0396c636c24dccaa669bc81d/tf_nightly_2.0_preview-2.0.0.dev20190412-cp34-cp34m-manylinux1_x86_64.whl (86.5MB)

bazel-out/k8-opt/genfiles/external/local_config_tf/include/tensorflow/core/framework/types.pb.h:10:40: fatal error: google/protobuf/port_def.inc: No such file or directory

https://source.cloud.google.com/results/invocations/fb7847e9-3cd6-4b07-9394-efe9f948f42e/targets/tensorflow_addons%2Fubuntu%2Fgpu%2Fpy3%2Fcontinuous/log

alsrgv · 2019-04-13T05:45:39Z

+1, it's still broken in the current tf-nightly.

meteorcloudy · 2019-04-15T08:22:06Z

Turned out we also need to fix steup.py:

tensorflow/tensorflow/tools/pip_package/setup.py

Line 240 in 1cdae49

list(find_files('*.h', 'google/protobuf_archive/src')) + list(

meteorcloudy · 2019-04-15T08:22:30Z

I verified on my local machine, fixing this file works. Sending a fix from internal.

meteorcloudy · 2019-04-26T07:43:09Z

Can anyone verify find protobuf_archive -name "*.h" -o -name "*.inc" works for their platform?

alsrgv · 2019-04-26T07:51:36Z

It worked on my Mac, Ubuntu 18.04 and your devel container, so seems promising :-)

meteorcloudy · 2019-04-26T08:19:27Z

@alsrgv Thanks! Hopefully this time it will work

Removing -type flag because it is unnecessary and on some platform it doesn't accepted multiple types, which caused: find: Arguments to -type should contain only one letter Fourth try to fix tensorflow#27697#issuecomment-486956391 PiperOrigin-RevId: 245433173

alsrgv · 2019-04-28T03:52:20Z

@meteorcloudy, I can confirm that both Mac and Linux nightly builds are operational. Could you pick this fix to the 1.4 branch?

meteorcloudy · 2019-04-29T07:58:32Z

@gunan @yifeif Can you help pick this fix to 1.4 branch?

martinwicke · 2019-04-29T15:17:23Z

@bananabowl can you cherry-pick this fix?

bananabowl · 2019-04-30T21:41:47Z

Sure - this is part of #28296

andreyhristov · 2019-06-24T13:53:01Z

Hi,
I experience this problem with 1.14.0 . From what I see in the sources, as well as bananabowl's comment / reference to the cherry picks, the needed changes are in 1.14.0. Still getting the problem.
Here is the error:

In file included from ./deps/tensorflow-headers-1.14.0/tensorflow/core/framework/tensor_shape.h:22:0,
                 from ./deps/tensorflow-headers-1.14.0/tensorflow/core/framework/tensor.h:23,
                 from ./deps/tensorflow-headers-1.14.0/tensorflow/cc/framework/ops.h:21,
                 from ./deps/tensorflow-headers-1.14.0/tensorflow/cc/client/client_session.h:24,
                 from ./include/detect_tensorflow.hpp:5,
                 from src/detect_tensorflow.cpp:1:
./deps/tensorflow-headers-1.14.0/tensorflow/core/framework/types.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory
 #include <google/protobuf/port_def.inc>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I am using the headers generated by //tensorflow:install_headers .
Platform : AARCH64
OS : Ubuntu 18.04 LTS
Hardware: Nvidia Xavier AGX
Software: TensorFlow 1.14.0 from a source release. Compiled with Bazel 0.24.1 (from a dist package).

bananabowl · 2019-06-24T20:25:34Z

Sorry to hear that Andrey :( @mihaimaruseac - I noticed you made some changes since the original fix, was this <fd5d844#diff-aa53f753fc372beaeb504d505c278203> or a subsequent change meant to fix this issue?

…

On Mon, Jun 24, 2019 at 7:00 AM Andrey Hristov ***@***.***> wrote: Hi, I experience this problem with 1.14.0 . From what I see in the sources, as well as bananabowl's comment / reference to the cherry picks, the needed changes are in 1.14.0. Still getting the problem. Here is the error: In file included from ./deps/tensorflow-headers-1.14.0/tensorflow/core/framework/tensor_shape.h:22:0, from ./deps/tensorflow-headers-1.14.0/tensorflow/core/framework/tensor.h:23, from ./deps/tensorflow-headers-1.14.0/tensorflow/cc/framework/ops.h:21, from ./deps/tensorflow-headers-1.14.0/tensorflow/cc/client/client_session.h:24, from ./include/detect_tensorflow.hpp:5, from src/detect_tensorflow.cpp:1: ./deps/tensorflow-headers-1.14.0/tensorflow/core/framework/types.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory #include <google/protobuf/port_def.inc> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I am using the headers generated by //tensorflow:install_headers . Platform : AARCH64 OS : Ubuntu 18.04 LTS Hardware: Nvidia Xavier AGX Software: TensorFlow 1.14.0 from a source release. Compiled with Bazel 0.24.1 (from a dist package). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#27697?email_source=notifications&email_token=AKEVL2DVBJ3KOFTZLYRTIZDP4DHRTA5CNFSM4HEW4MV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYM7ZXQ#issuecomment-505019614>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKEVL2HIORNS6JG5AEYT4CTP4DHRTANCNFSM4HEW4MVQ> .

mihaimaruseac · 2019-06-24T20:28:01Z

Hi, the fd5d844 change is for virtual pip, should be on master only, not on a release branch.

There's also some changes related with moving protobuf definitions around, but again, that should only be on master.

mihaimaruseac · 2019-06-24T20:48:18Z

So, the commits that I know can affect this are fd5d844, 338c2f2 and 508f76b

Could try patching each one of these manually and see which works

andreyhristov · 2019-06-25T08:28:15Z

I tried the master branch. Interestingly it seems to be 1.13.1 (because the generated PIP file is tensorflow-1.13.1-cp36-cp36m-linux_aarch64.whl). The problem persist but it could be that master is not the right branch (is not actively used). Is it?

meteorcloudy · 2019-06-25T09:28:00Z

@andreyhristov The master branch is the develop branch.
I tried to build //tensorflow:include_headers on the latest master branch, it does contain the port_def.inc

$ bazel build --config=opt //tensorflow:install_headers
...
Target //tensorflow:install_headers up-to-date:
  bazel-genfiles/tensorflow/include
INFO: Elapsed time: 0.239s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
$ ls ./bazel-genfiles/tensorflow/include/external/com_google_protobuf/src/google/protobuf/port_def.inc
./bazel-genfiles/tensorflow/include/external/com_google_protobuf/src/google/protobuf/port_def.inc

meteorcloudy · 2019-06-25T09:38:19Z

And in 1.14 branch, the ".inc" file headers also exist but under external/protobuf_archive directory.

ls ./bazel-genfiles/tensorflow/include/external/protobuf_archive/src/google/protobuf/port_def.inc
./bazel-genfiles/tensorflow/include/external/protobuf_archive/src/google/protobuf/port_def.inc

andreyhristov · 2019-06-25T11:54:50Z

ok guys, right, I found the file. The problem was that I needed to add the path to protobuf_archive to the include path of the compiler. I compiles now and it works! (I had asserts with 1.12.0 on batch inference which are gone).

Last question, why does master seem to be with version 1.13.1 ? Do you guys change the version after branching for release?

meteorcloudy · 2019-06-25T14:02:41Z

Yes, it's embedded here:

tensorflow/tensorflow/tools/pip_package/setup.py

Line 50 in c19265b

_VERSION = '1.13.1'

@gunan Is this intentionally?

tensorflow-bot · 2019-06-25T14:02:43Z

Are you satisfied with the resolution of your issue?
Yes
No

mihaimaruseac · 2019-06-25T17:07:15Z

Fixed the version number in 4a017ec

andreyhristov · 2019-07-19T08:11:35Z

Hi,
what I did for my program was to pass a -I param to the compiler, so it does know where to look for the files.
I use a Makefile, and also export a TENSORFLOW_PATH variable, which contains the path to the headers. The headers can be extracted with a bazel target. For me the flag to gcc is :
-I$(TENSORFLOW_PATH)/external/protobuf_archive/src

suyashbakshi · 2019-08-01T16:42:34Z

Hello, I'm still seeing this issue with tf1.14.0 while compiling a custom op kernel.
site-packages/tensorflow/include/tensorflow/core/framework/op_def.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory #include <google/protobuf/port_def.inc> compilation terminated.

I'm on windows, installed TF using pip.
The .inc files still reside in "site-packages\tensorflow\include\google\protobuf_archive\src\google\protobuf".

Is there a way to fix this issue without building the sources manually (by using bazel)?

Any help is appreciated. Thanks!

megazone87 · 2019-08-15T07:34:11Z

Hello, I'm still seeing this issue with tf1.14.0 while compiling a custom op kernel.
site-packages/tensorflow/include/tensorflow/core/framework/op_def.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory #include <google/protobuf/port_def.inc> compilation terminated.

I'm on windows, installed TF using pip.
The .inc files still reside in "site-packages\tensorflow\include\google\protobuf_archive\src\google\protobuf".

Is there a way to fix this issue without building the sources manually (by using bazel)?

Any help is appreciated. Thanks!

try include the dir solve this issue? like: -I..\site-packages\tensorflow\include\google\protobuf_archive\src

alsrgv mentioned this issue Apr 9, 2019

Pin tf-nightly due to build break horovod/horovod#1003

Merged

seanpmorgan mentioned this issue Apr 10, 2019

conversion to use tf2 public API - installment 1 tensorflow/addons#145

Merged

qlzh727 assigned yifeif and gunan Apr 10, 2019

angerson unassigned yifeif and gunan Apr 10, 2019

rthadur self-assigned this Apr 10, 2019

rthadur added type:build/install Build and install issues subtype: ubuntu/linux Ubuntu/Linux Build/Installation Issues type:bug Bug labels Apr 10, 2019

armando-fandango mentioned this issue Apr 11, 2019

Build fails on tf-nightly due to new protobuf tensorflow/addons#165

Closed

tensorflow-copybara closed this as completed in 42df8cc Apr 11, 2019

seanpmorgan mentioned this issue Apr 12, 2019

Pin nightly version until fix is in place tensorflow/addons#170

Merged

alsrgv mentioned this issue Apr 13, 2019

Unpin tf-nightly & pin MXNet horovod/horovod#1011

Merged

alsrgv mentioned this issue Apr 13, 2019

pip install horovod won't install with tensorflow master 2afc6f8862ec04708f166627dd7c9a3d045649b4 horovod/horovod#1009

Closed

martinwicke reopened this Apr 13, 2019

seanpmorgan mentioned this issue Apr 26, 2019

Update maintainers tensorflow/addons#213

Merged

andreyhristov mentioned this issue Jun 24, 2019

fatal error: google/protobuf/port_def.inc: No such file or directory compilation terminated. BVLC/caffe#6774

Open

mihaimaruseac reopened this Jun 24, 2019

mihaimaruseac assigned meteorcloudy and unassigned rthadur Jun 24, 2019

meteorcloudy closed this as completed Jun 25, 2019

sharvil mentioned this issue Mar 16, 2020

[Feature] Windows build support lmnt-com/haste#5

Closed

ngam mentioned this issue Feb 20, 2023

WIP: Bump to 2.12 + add aarch64 conda-forge/tensorflow-feedstock#301

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tensorflow/core/framework/op_def.pb.h:10:40: fatal error: google/protobuf/port_def.inc: No such file or directory #27697

tensorflow/core/framework/op_def.pb.h:10:40: fatal error: google/protobuf/port_def.inc: No such file or directory #27697

alsrgv commented Apr 9, 2019 •

edited

seanpmorgan commented Apr 10, 2019

qlzh727 commented Apr 10, 2019

angerson commented Apr 10, 2019 •

edited

angerson commented Apr 10, 2019

meteorcloudy commented Apr 10, 2019

seanpmorgan commented Apr 10, 2019

alsrgv commented Apr 10, 2019

meteorcloudy commented Apr 11, 2019

tensorflow-bot bot commented Apr 11, 2019

seanpmorgan commented Apr 12, 2019 •

edited

alsrgv commented Apr 13, 2019

meteorcloudy commented Apr 15, 2019

meteorcloudy commented Apr 15, 2019

meteorcloudy commented Apr 26, 2019

alsrgv commented Apr 26, 2019

meteorcloudy commented Apr 26, 2019

alsrgv commented Apr 28, 2019

meteorcloudy commented Apr 29, 2019

martinwicke commented Apr 29, 2019

bananabowl commented Apr 30, 2019

andreyhristov commented Jun 24, 2019

bananabowl commented Jun 24, 2019 via email

mihaimaruseac commented Jun 24, 2019

mihaimaruseac commented Jun 24, 2019

andreyhristov commented Jun 25, 2019

meteorcloudy commented Jun 25, 2019

meteorcloudy commented Jun 25, 2019

andreyhristov commented Jun 25, 2019 •

edited

meteorcloudy commented Jun 25, 2019

tensorflow-bot bot commented Jun 25, 2019

mihaimaruseac commented Jun 25, 2019

andreyhristov commented Jul 19, 2019

suyashbakshi commented Aug 1, 2019 •

edited

megazone87 commented Aug 15, 2019

tensorflow/core/framework/op_def.pb.h:10:40: fatal error: google/protobuf/port_def.inc: No such file or directory #27697

tensorflow/core/framework/op_def.pb.h:10:40: fatal error: google/protobuf/port_def.inc: No such file or directory #27697

Comments

alsrgv commented Apr 9, 2019 • edited

seanpmorgan commented Apr 10, 2019

qlzh727 commented Apr 10, 2019

angerson commented Apr 10, 2019 • edited

angerson commented Apr 10, 2019

meteorcloudy commented Apr 10, 2019

seanpmorgan commented Apr 10, 2019

alsrgv commented Apr 10, 2019

meteorcloudy commented Apr 11, 2019

tensorflow-bot bot commented Apr 11, 2019

seanpmorgan commented Apr 12, 2019 • edited

alsrgv commented Apr 13, 2019

meteorcloudy commented Apr 15, 2019

meteorcloudy commented Apr 15, 2019

meteorcloudy commented Apr 26, 2019

alsrgv commented Apr 26, 2019

meteorcloudy commented Apr 26, 2019

alsrgv commented Apr 28, 2019

meteorcloudy commented Apr 29, 2019

martinwicke commented Apr 29, 2019

bananabowl commented Apr 30, 2019

andreyhristov commented Jun 24, 2019

bananabowl commented Jun 24, 2019 via email

mihaimaruseac commented Jun 24, 2019

mihaimaruseac commented Jun 24, 2019

andreyhristov commented Jun 25, 2019

meteorcloudy commented Jun 25, 2019

meteorcloudy commented Jun 25, 2019

andreyhristov commented Jun 25, 2019 • edited

meteorcloudy commented Jun 25, 2019

tensorflow-bot bot commented Jun 25, 2019

mihaimaruseac commented Jun 25, 2019

andreyhristov commented Jul 19, 2019

suyashbakshi commented Aug 1, 2019 • edited

megazone87 commented Aug 15, 2019

alsrgv commented Apr 9, 2019 •

edited

angerson commented Apr 10, 2019 •

edited

seanpmorgan commented Apr 12, 2019 •

edited

andreyhristov commented Jun 25, 2019 •

edited

suyashbakshi commented Aug 1, 2019 •

edited