Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building TensorFlow Serving example code in Docker container fails with compilation error #590

Closed
axbaretto opened this issue Sep 6, 2017 · 7 comments

Comments

@axbaretto
Copy link

commented Sep 6, 2017

Building tensorflow_serving/example/... (from https://www.tensorflow.org/serving/serving_inception, Clone, configure, and build TensorFlow Serving in a container) with bazel fails with the following error:

ERROR: /root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/external/org_tensorflow/tensorflow/core/kernels/BUILD:2537:1: C++ compilation of rule '@org_tensorflow//tensorflow/core/kernels:cwise_op' failed: gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 128 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 4.
gcc: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,

@axbaretto

This comment has been minimized.

Copy link
Author

commented Sep 7, 2017

The compilation error can be addressed by increasing memory available to the docker daemon from 2GB to 6-8GB. This should be documented on https://www.tensorflow.org/serving/serving_inception.

After increasing memory to 8GB, I am now seeing the following errors:
root@32c8076cecd3:/serving# bazel-bin/tensorflow_serving/example/inception_saved_model --checkpoint_dir=inception-v3 --output_dir=inception-export
Traceback (most recent call last):
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/tf_serving/tensorflow_serving/example/inception_saved_model.py", line 27, in
import tensorflow as tf
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/init.py", line 24, in
from tensorflow.python import *
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/python/init.py", line 63, in
from tensorflow.python.framework.framework_lib import *
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/python/framework/framework_lib.py", line 76, in
from tensorflow.python.framework.ops import Graph
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 28, in
from autograd import core as ag_core
ImportError: No module named autograd

@aaxwaz

This comment has been minimized.

Copy link

commented Sep 7, 2017

try 'pip install autograd'

It worked for me!

@axbaretto

This comment has been minimized.

Copy link
Author

commented Sep 7, 2017

Thanks! It works fine.

Let's add that to the doc....

Now it is failing with

bazel build -c opt tensorflow_serving/example/...
ERROR: /serving/tensorflow_serving/example/BUILD:119:1: no such package '@protobuf//': Could not find handler for bind rule //external:protobuf and referenced by '//tensorflow_serving/example:inception_client_cc'.
ERROR: Analysis of target '//tensorflow_serving/example:inception_client_cc' failed; build aborted.

@axbaretto

This comment has been minimized.

Copy link
Author

commented Sep 7, 2017

We really need some complete doc around this. As is, https://www.tensorflow.org/serving/setup does not work.

@kirilg

This comment has been minimized.

Copy link
Member

commented Sep 7, 2017

Sorry for all the issues!

The first problem ("internal compiler error: Killed (program cc1plus)") is a compiler error that's not specific to the inception tutorial or Docker, it's just a resource issue. You can increase the resources given to Docker or constrain the resources used by Bazel to compile (this solution was used in other GitHub issues if you search for "cc1plus").

https://www.tensorflow.org/serving/setup mentions "Consult the TensorFlow install instructions if you encounter any issues with setting up TensorFlow or its dependencies." with a link. TF's dependencies change over time so to avoid duplicating the documentation, we just link to it. Autograd for example was only added as a dep in the last couple of weeks, but you can see all deps here.


Your last problem seems to be due to mismatched versions. I would guess you synced to an older version of TF Serving while still keeping a newer TF submodule (or started with an older TF serving and synced the TF submodule to head, which does not work). If you look at the repo at head, we no longer use "@protobuf//" (TF renamed it) and instead now use "@protobuf_archive" which is what TF defined.

@axbaretto

This comment has been minimized.

Copy link
Author

commented Sep 7, 2017

Thanks. Please still update the doc to call out the same for build instructions as you do for install instructions when issues are encountered.

@kirilg

This comment has been minimized.

Copy link
Member

commented Sep 11, 2017

Sure. Just to follow up on this, I update the documentation and it's in GitHub, should be pushed to the main documentation website soon.

@kirilg kirilg closed this Sep 11, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.