Update Tensorflow to 1.4

[lissyx]

We should merge upstream r1.4 onto mozilla/tensorflow's master branch.

Tracking PR for the WIP is mozilla/tensorflow#43

[lissyx]

So far, we need a few hacks (not considering the usual updates needed, like bazel/cuda/cudnn):

• nsync failure on OSX (suggested fix from nsync upstream: CPU Build Fails on OSX Sierra  tensorflow/tensorflow#13220 (comment))
• nsync failure for RPi3 (Build problem :Configurable attribute "copts" doesn't match this configuration tensorflow/tensorflow#13115)
• LLVM visibility issues for tfcompile (with pending fix documented Target '@llvm//:support' is not visible from target '@org_tensorflow//tensorflow/compiler/xla/client:compile_only_client'. tensorflow/tensorflow#14445 (comment))
• tfcompile_flags not happy with our select() for RPi3 handling
• tfcompile choking on device name XLA_CPU_JIT

[lissyx]

I have a fix for nsync failures on RPi3. As of now, we can get all tensorflow builds to be green. We need then to see if DeepSpeech is green as well :)

[lissyx]

After testing with upstream's code, tfcompile choking on XLA_CPU_JIT seems to be fixed thanks to tensorflow/tensorflow@27412f3

However, shape dimensions [1, 64, 494] are still triggering very huge memory use.

[lissyx]

shape=[1, 16, 494] goes up to ~16GB memory use, and produces AOT model \o/

Also, cherry-picked proper upstream fix for llvm.BUILD changes.

[spakhomov]

Has the issue mentioned by lissyx with

--- tfcompile_flags not happy with our select() for RPi3 handling

been resolved?

I pulled the latest commit but still getting this error that seems to be related to the issue above. If not, any ideas why native_client is not compiling with TensorFlow 1.4.0 and throwing this error? It did compile with an earlier version of Tensorflow (1.0).

user@gpunode:~/tensorflow$ bazel build -c opt --copt=-O3 //tensorflow:libtensorflow_cc.so //native_client:deepspeech //native_client:deepspeech_utils //native_client:ctc_decoder_with_kenlm //native_client:generate_trie
.........
WARNING: The lower priority option '-c opt' does not override the previous value '-c opt'
ERROR: /home/user/tensorflow/native_client/BUILD:18:1: Traceback (most recent call last):
File "/home/user/tensorflow/native_client/BUILD", line 18
tf_library(name = "deepspeech_model", cpp_cla...", <5 more arguments>)
File "/home/user/tensorflow/tensorflow/compiler/aot/tfcompile.bzl", line 136, in tf_library
" ".join([(("'" + arg.replace("'", "'\''... [])])
File "/home/user/tensorflow/tensorflow/compiler/aot/tfcompile.bzl", line 136, in " ".join
[(("'" + arg.replace("'", "'\''")) + "'") for arg in (tfcompile_flags or [])]
type 'select' is not iterable
ERROR: error loading package 'native_client': Package 'native_client' contains errors
INFO: Elapsed time: 0.837s
FAILED: Build did NOT complete successfully (2 packages loaded)

[reuben]

No, the work is still ongoing in mozilla/tensorflow#43

[lissyx]

@spakhomov If you are interested, you should checkout the branch of the PR and also this one, it should work: #971

[spakhomov]

Thanks! In the meantime, I tried @lissyx suggestion and that branch compiled fine - no errors.

[spakhomov]

I spoke too soon... It looks like the libraries built fine, but now I am getting errors with linking to libdeepspeech library when trying to make deepspeech in native_client:

pakh0002@gpunode:~/DeepSpeech/native_client$ make deepspeech
g++ -o deepspeech pkg-config --cflags sox -L/home/pakh0002/tensorflow/bazel-bin/tensorflow -L/home/pakh0002/tensorflow/bazel-bin/native_client client.cc pkg-config --libs sox -ldeepspeech -ldeepspeech_utils -ltensorflow_cc
/home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::Tensor::CheckTypeAndIsAligned(tensorflow::DataType) const' /home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::TensorShapeRep::DestructorOutOfLine()'
/home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::Status::SlowCopyFrom(tensorflow::Status::State const*)' /home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::NewSession(tensorflow::SessionOptions const&, tensorflow::Session**)'
/home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::TensorShape::CheckDimsEqual(int) const' /home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::TensorShapeRep::SlowCopyFrom(tensorflow::TensorShapeRep const&)'
/home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::TensorShapeBase<tensorflow::TensorShape>::dim_size(int) const' /home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::ConfigProto::~ConfigProto()'
/home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::Env::Default()' /home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::Status::ToString() const'
/home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::Tensor::CheckIsAlignedAndSingleElement() const' /home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::SessionOptions::SessionOptions()'
/home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::Tensor::~Tensor()' /home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::Tensor::Tensor(tensorflow::DataType, tensorflow::TensorShape const&)'
/home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::GraphDef::~GraphDef()' /home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::GraphDef::GraphDef()'
/home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::TensorShape::CheckDimsAtLeast(int) const' /home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to tensorflow::ReadBinaryProto(tensorflow::Env*, std::string const&, google::protobuf::MessageLite*)'
/home/pakh0002/tensorflow/bazel-bin/native_client/libdeepspeech.so: undefined reference to `tensorflow::TensorShapeBasetensorflow::TensorShape::TensorShapeBase(tensorflow::gtl::ArraySlice)'
collect2: error: ld returned 1 exit status
make: *** [deepspeech] Error 1

[lissyx]

@spakhomov Please avoid using this space for this kind of discussion, github issues are the where we should do that ; https://discourse.mozilla.org/c/deep-speech is more suited for this :)

Besides, it's likely to be because of no libtensorflow_framework.so being built, which I just updated the doc and build about.

[lissyx]

https://tools.taskcluster.net/groups/YV2ohIi4QAqOiOhTaSyKiQ is the task group for the latest push against current master, once that is green, I will proceed with:
(1) pushing DeepSpeech branch again, with proper home.tar.xz URL (not taskId based)
(2) merge Tensorflow PR

It is expected (1) will fail until (2) produces its output.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.