Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build fails #23

Closed
rasmi opened this issue Sep 5, 2016 · 21 comments
Closed

Build fails #23

rasmi opened this issue Sep 5, 2016 · 21 comments

Comments

@rasmi
Copy link
Member

rasmi commented Sep 5, 2016

My build fails when compiling through Bazel (while building TensorFlow).

ERROR: /hmt/sirius1/skv0/u/4/r/user/.cache/bazel/_bazel_user/d217f35631206796f447d50c6f1d6243/external/highwayhash/BUILD:125:1: C++ compilation of rule '@highwayhash//:sip_hash' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /hmt/sirius1/skv0/u/4/r/user/.cache/bazel/_bazel_user/d217f35631206796f447d50c6f1d6243/execroot/tensorflow && \
  exec env - \
    LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:/usr/local/gurobi/lib:/usr/local/cuda-7.5/extras/CUPTI/lib64 \
    PATH=/vega/astro/users/user/applications/pythonenv/bin:/opt/rh/devtoolset-1.1/root/usr/bin:/usr/local/cuda-7.5/bin:/vega/astro/users/user/applications/bazel/output:/vega/astro/users/user/applications/swig/bin:/vega/astro/users/user/applications/jdk1.8.0_102/bin:/usr/local/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/gurobi/bin:/opt/openlava-3.3/bin \
    TMPDIR=/tmp \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections -g0 '-std=c++11' -MD -MF bazel-out/host/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.d '-frandom-seed=bazel-out/host/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.o' -iquote external/highwayhash -iquote bazel-out/host/genfiles/external/highwayhash -iquote external/bazel_tools -iquote bazel-out/host/genfiles/external/bazel_tools -isystem external/highwayhash -isystem bazel-out/host/genfiles/external/highwayhash -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/highwayhash/highwayhash/sip_hash.cc -o bazel-out/host/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from external/highwayhash/highwayhash/sip_hash.h:24:0,
                 from external/highwayhash/highwayhash/sip_hash.cc:15:
external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)':
external/highwayhash/highwayhash/state_helpers.h:31:13: error: there are no arguments to 'alignas' that depend on a template parameter, so a declaration of 'alignas' must be available [-fpermissive]
external/highwayhash/highwayhash/state_helpers.h:31:13: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
external/highwayhash/highwayhash/state_helpers.h:31:15: error: expected ';' before 'char'
external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope
Target //tensorflow/cc:tutorials_example_trainer failed to build

Adding the -fpermissive does allow it to continue with warnings, at which point I get this error:

ERROR: /hmt/sirius1/skv0/u/4/r/user/.cache/bazel/_bazel_user/d217f35631206796f447d50c6f1d6243/external/highwayhash/BUILD:125:1: C++ compilation of rule '@highwayhash//:sip_hash' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /hmt/sirius1/skv0/u/4/r/user/.cache/bazel/_bazel_user/d217f35631206796f447d50c6f1d6243/execroot/tensorflow && \
  exec env - \
    LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:/usr/local/gurobi/lib:/usr/local/cuda-7.5/extras/CUPTI/lib64 \
    PATH=/vega/astro/users/user/applications/pythonenv/bin:/opt/rh/devtoolset-1.1/root/usr/bin:/usr/local/cuda-7.5/bin:/vega/astro/users/user/applications/bazel/output:/vega/astro/users/user/applications/swig/bin:/vega/astro/users/user/applications/jdk1.8.0_102/bin:/usr/local/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/gurobi/bin:/opt/openlava-3.3/bin \
    TMPDIR=/tmp \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-std=c++11' -MD -MF bazel-out/local_linux-opt/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.pic.d '-frandom-seed=bazel-out/local_linux-opt/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.pic.o' -fPIC -iquote external/highwayhash -iquote bazel-out/local_linux-opt/genfiles/external/highwayhash -iquote external/bazel_tools -iquote bazel-out/local_linux-opt/genfiles/external/bazel_tools -isystem external/highwayhash -isystem bazel-out/local_linux-opt/genfiles/external/highwayhash -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/highwayhash/highwayhash/sip_hash.cc -o bazel-out/local_linux-opt/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from external/highwayhash/highwayhash/sip_hash.h:24:0,
                 from external/highwayhash/highwayhash/sip_hash.cc:15:
external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)':
external/highwayhash/highwayhash/state_helpers.h:31:15: error: expected ';' before 'char'
external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope
Target //tensorflow/cc:tutorials_example_trainer failed to build
@jan-wassenberg
Copy link
Member

Hi, thanks for reporting. We require GCC 4.8 for "alignas". Is it an option for you to compile with a more recent GCC or clang?
Alternatively, you can replace alignas(32) with attribute((aligned(32))).

@rasmi
Copy link
Member Author

rasmi commented Sep 5, 2016

Hi @jan-wassenberg, thanks for the reply. Here's my diff:

-  alignas(32) char final_packet[State::kPacketSize] = {0};
+  attribute((aligned(32))) char final_packet[State::kPacketSize] = {0};

Unfortunately, compiling with gcc 4.9.1 leads with a whole set of other errors (though not with highwayhash), and the next-closest version I have is 4.7.2. I'm using a compute cluster so the gcc versions available to me are limited.
I'm now getting the same error:

In file included from external/highwayhash/highwayhash/sip_hash.h:24:0,
                 from external/highwayhash/highwayhash/sip_hash.cc:15:
external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)':
external/highwayhash/highwayhash/state_helpers.h:31:28: error: expected ';' before 'char'
external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope

Here's the block of code. Could it be a syntax issue?

#ifndef HIGHWAYHASH_HIGHWAYHASH_STATE_H_
#define HIGHWAYHASH_HIGHWAYHASH_STATE_H_

// Helper functions to split inputs into packets and call State::Update on each.

#include <cstddef>
#include <cstring>
#include <memory>

#include "highwayhash/code_annotation.h"
#include "highwayhash/types.h"

namespace highwayhash {

// Copies the remaining bytes to a zero-padded buffer, sets the upper byte to
// size % 256 (always possible because this should only be called if the
// total size is not a multiple of the packet size) and updates hash state.
//
// The padding scheme is essentially from SipHash, but permuted for the
// convenience of AVX-2 masked loads. This function must use the same layout so
// that the vector and scalar HighwayTreeHash have the same result.
//
// "remaining_size" is the number of accessible/remaining bytes
// (size % kPacketSize).
//
// Primary template; the specialization for AVX-2 is faster. Intended as an
// implementation detail, do not call directly.
template <class State>
INLINE void PaddedUpdate(const uint64 size, const char* remaining_bytes,
                         const uint64 remaining_size, State* state) {
  attribute((aligned(32))) char final_packet[State::kPacketSize] = {0};

  // Unusual layout matches the AVX-2 specialization in highway_tree_hash.h.
  const size_t remainder_mod4 = remaining_size & 3;
  uint32 packet4 = static_cast<uint32>(size) << 24;
  const char* final_bytes = remaining_bytes + remaining_size - remainder_mod4;
  for (size_t i = 0; i < remainder_mod4; ++i) {
    const uint32 byte = static_cast<unsigned char>(final_bytes[i]);
    packet4 += byte << (i * 8);
  }

  memcpy(final_packet, remaining_bytes, remaining_size - remainder_mod4);
  memcpy(final_packet + State::kPacketSize - 4, &packet4, sizeof(packet4));

  state->Update(final_packet);
}

// Updates hash state for every whole packet, and once more for the final
// padded packet.
template <class State>
INLINE void UpdateState(const char* bytes, const uint64 size, State* state) {
  // Feed entire packets.
  const int kPacketSize = State::kPacketSize;
  static_assert((kPacketSize & (kPacketSize - 1)) == 0, "Size must be 2^i.");
  const size_t remainder = size & (kPacketSize - 1);
  const size_t truncated_size = size - remainder;
  for (size_t i = 0; i < truncated_size; i += kPacketSize) {
    state->Update(bytes + i);
  }

  PaddedUpdate(size, bytes + truncated_size, remainder, state);
}

// Convenience function for updating with the bytes of a string.
template <class String, class State>
INLINE void UpdateState(const String& s, State* state) {
  const char* bytes = reinterpret_cast<const char*>(s.data());
  const size_t size = s.length() * sizeof(typename String::value_type);
  UpdateState(bytes, size, state);
}

// Computes a hash of a byte array using the given hash State class.
//
// Example: const SipHashState::Key key = { 1, 2 }; char data[4];
// ComputeHash<SipHashState>(key, data, sizeof(data));
//
// This function avoids duplicating Update/Finalize in every call site.
// Callers wanting to combine multiple hashes should repeatedly UpdateState()
// and only call State::Finalize once.
template <class State>
uint64 ComputeHash(const typename State::Key& key, const char* bytes,
                   const uint64 size) {
  State state(key);
  UpdateState(bytes, size, &state);
  return state.Finalize();
}

// Computes a hash of a string's bytes using the given hash State class.
//
// Example: const SipHashState::Key key = { 1, 2 };
// StringHasher<SipHashState>()(key, std::u16string(u"abc"));
//
// A struct with nested function template enables deduction of the String type.
template <class State>
struct StringHasher {
  template <class String>
  uint64 operator()(const typename State::Key& key, const String& s) {
    State state(key);
    UpdateState(s, &state);
    return state.Finalize();
  }
};

}  // namespace highwayhash

#endif  // HIGHWAYHASH_HIGHWAYHASH_STATE_H_

@jan-wassenberg
Copy link
Member

jan-wassenberg commented Sep 5, 2016

Hi, good news, it will compile with just a minor change; GCC wants the type first and attribute second.
attribute((aligned(32))) char -> char attribute((aligned(32))).

@rasmi
Copy link
Member Author

rasmi commented Sep 5, 2016

@jan-wassenberg still getting a similar error. Sorry for the silly fixes:

In file included from external/highwayhash/highwayhash/sip_hash.h:24:0,
                 from external/highwayhash/highwayhash/sip_hash.cc:15:
external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)':
external/highwayhash/highwayhash/state_helpers.h:31:33: error: expected ',' or ';' before 'final_packet'
external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope

@jan-wassenberg
Copy link
Member

jan-wassenberg commented Sep 5, 2016

hehe, I see what happened. The Markdown syntax ate the underscores; it should read char __attribute__((aligned(32))) final_packet[State::kPacketSize] = {0};.

@rasmi
Copy link
Member Author

rasmi commented Sep 5, 2016

@jan-wassenberg Thank you so much! That worked!

@rasmi rasmi closed this as completed Sep 5, 2016
@mnicky
Copy link

mnicky commented Nov 2, 2016

I've run into this issue (while compiling tensorflow) as well.

@jan-wassenberg: if the change you suggested makes the code compatible with more gcc versions, why not to make it permanent?

@jan-wassenberg
Copy link
Member

jan-wassenberg commented Nov 2, 2016

Hi @mnicky, it's a tradeoff - one disadvantage of __attribute__ is that it's not supported by MSVC. I had hoped that C++11 is widely available by now. Can you share which GCC version you have, and why it's infeasible to use a more recent version? Perhaps we might switch to a macro if necessary.

@rasmi
Copy link
Member Author

rasmi commented Nov 2, 2016

@mnicky, if you are in a scientific computing Red Hat / CentOS environment, you might have more recent versions of gcc available. Try module load gcc<tab complete> to check. The default on my system was much older than the other available versions. Disregard if this isn't your situation...

@mnicky
Copy link

mnicky commented Nov 2, 2016

I have gcc 4.7.2 which is included in Debian Wheezy that runs on our cluster. In this case I'm able to modify the source file to get rid of the compilation error. I was just curious why the workaround/fix didn't get it into the master. I think there are still quite a few people running older versions of gcc without possibility of upgrading. And as highwayhash is needed to build tensorflow, I'd assume there are people affected by this.

@jan-wassenberg
Copy link
Member

jan-wassenberg commented Nov 2, 2016

Ah, thanks for mentioning your GCC version. That's still fairly recent; it is unfortunate that alignas didn't make it in until 4.8. We will soon add a HH_ALIGNAS macro that resolves to alignas or falls back to attribute.

@jan-wassenberg jan-wassenberg reopened this Nov 2, 2016
@jan-wassenberg
Copy link
Member

The fix is in - please reopen if there's still an issue :)

@Sinan81
Copy link

Sinan81 commented Feb 3, 2017

I am using GCC5.3.0, and getting a similar error:
C++ compilation of rule '@highwayhash//:sip_hash' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command

My set up is:
Bazel 0.4.3 (compiled from source)
Tensorflow 0.12.1 (from zip file)
GCC 5.3.0
Cuda 8.0
CuDNN 5.1

@jan-wassenberg
Copy link
Member

Can you include more output to show the actual compile error and in which file/header it occurs?
Should be a few lines below the "error executing command" you mentioned.

@Sinan81
Copy link

Sinan81 commented Feb 6, 2017

Sorry for the late reply. I started from scratch (including removing ~/.cache/bazel...), and it seems TF now compiles fine. Should I delete my previous message?

@jan-wassenberg
Copy link
Member

Glad to hear it! No worries, we can leave this issue unchanged/closed.

@diegocrzt
Copy link

diegocrzt commented Mar 15, 2017

Hi guys/girls, I am having the same problem with gcc 4.7.4
ERROR: /home/diegocrzt/.cache/bazel/_bazel_diegocrzt/8df0525fbbd1e101024ee0b817c7eaef/external/highwayhash/BUILD:125:1: C++ compilation of rule '@highwayhash//:sip_hash' failed: gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 32 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. In file included from external/highwayhash/highwayhash/sip_hash.h:24:0, from external/highwayhash/highwayhash/sip_hash.cc:15: external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)': external/highwayhash/highwayhash/state_helpers.h:31:13: error: there are no arguments to 'alignas' that depend on a template parameter, so a declaration of 'alignas' must be available [-fpermissive] external/highwayhash/highwayhash/state_helpers.h:31:13: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated) external/highwayhash/highwayhash/state_helpers.h:31:15: error: expected ';' before 'char' external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope Target //tensorflow/tools/pip_package:build_pip_package failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 239.020s, Critical Path: 225.21s
But after reading this I am not sure what to do. And I can't upgrade my version of GCC

@rasmi
Copy link
Member Author

rasmi commented Mar 15, 2017

@diegocrzt, you need to edit the file external/highwayhash/highwayhash/state_helpers.h to make the changes to line 31 that @jan-wassenberg listed above. It should read
char __attribute__((aligned(32))) final_packet[State::kPacketSize] = {0};
Then try re-compiling.

@diegocrzt
Copy link

@rasmi, thank you for you quick answer and sorry for not understanding at first time, I was lost on the first markdown issues 😆, highwayhash step is done but right now I have other issue, with the core tensorflow, I think this is probably related no more to this thread, but just to be sure, what do you think, it could be related yet ?

ERROR: /home/diegocrzt/codice/dpi/tensorflow/tensorflow-git/tensorflow/contrib/layers/BUILD:19:1: C++ compilation of rule '//tensorflow/contrib/layers:python/ops/_bucketization_op.so' failed: gcc failed: error executing command 
  (cd /home/diegocrzt/.cache/bazel/_bazel_diegocrzt/8df0525fbbd1e101024ee0b817c7eaef/execroot/tensorflow-git && \
  exec env - \
    PATH=/usr/local/bin:/bin:/usr/bin:/opt/texlive/2016/bin/x86_64-linux:/opt/java/jdk/bin \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-march=native' '-std=c++0x' -MD -MF bazel-out/local-py3-opt/bin/tensorflow/contrib/layers/_objs/python/ops/_bucketization_op.so/tensorflow/contrib/layers/ops/bucketization_op.pic.d '-frandom-seed=bazel-out/local-py3-opt/bin/tensorflow/contrib/layers/_objs/python/ops/_bucketization_op.so/tensorflow/contrib/layers/ops/bucketization_op.pic.o' -fPIC -DEIGEN_MPL2_ONLY -iquote . -iquote bazel-out/local-py3-opt/genfiles -iquote external/bazel_tools -iquote bazel-out/local-py3-opt/genfiles/external/bazel_tools -iquote external/eigen_archive -iquote bazel-out/local-py3-opt/genfiles/external/eigen_archive -iquote external/local_config_sycl -iquote bazel-out/local-py3-opt/genfiles/external/local_config_sycl -iquote external/protobuf -iquote bazel-out/local-py3-opt/genfiles/external/protobuf -isystem external/bazel_tools/tools/cpp/gcc3 -isystem external/eigen_archive -isystem bazel-out/local-py3-opt/genfiles/external/eigen_archive -isystem external/protobuf/src -isystem bazel-out/local-py3-opt/genfiles/external/protobuf/src -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare -fno-exceptions -pthread -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c tensorflow/contrib/layers/ops/bucketization_op.cc -o bazel-out/local-py3-opt/bin/tensorflow/contrib/layers/_objs/python/ops/_bucketization_op.so/tensorflow/contrib/layers/ops/bucketization_op.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from ./tensorflow/core/lib/gtl/array_slice.h:101:0,
                 from ./tensorflow/core/lib/strings/str_util.h:23,
                 from ./tensorflow/core/framework/op.h:29,
                 from tensorflow/contrib/layers/ops/bucketization_op.cc:16:
./tensorflow/core/lib/gtl/array_slice_internal.h:232:38: error: 'tensorflow::gtl::array_slice_internal::ArraySliceImplBase<const T>::ArraySliceImplBase' names constructor
./tensorflow/core/lib/gtl/array_slice_internal.h:252:32: error: 'tensorflow::gtl::array_slice_internal::ArraySliceImplBase<T>::ArraySliceImplBase' names constructor
In file included from ./tensorflow/core/lib/gtl/array_slice.h:102:0,
                 from ./tensorflow/core/lib/strings/str_util.h:23,
                 from ./tensorflow/core/framework/op.h:29,
                 from tensorflow/contrib/layers/ops/bucketization_op.cc:16:
./tensorflow/core/lib/gtl/inlined_vector.h: In member function 'void tensorflow::gtl::InlinedVector<T, N>::Destroy(T*, int)':
./tensorflow/core/lib/gtl/inlined_vector.h:396:10: error: 'is_trivially_destructible' is not a member of 'std'
./tensorflow/core/lib/gtl/inlined_vector.h:396:42: error: expected primary-expression before '>' token
./tensorflow/core/lib/gtl/inlined_vector.h:396:43: error: '::value' has not been declared
Target //tensorflow/tools/pip_package:build_pip_package failed to build

@rasmi
Copy link
Member Author

rasmi commented Mar 17, 2017

@diegocrzt no problem. That is an unrelated issue. According to this thread, your version of gcc is too old. If you are using a scientific computing environment, try typing module load gcc and tab-complete that command before running it to see which versions you have available. Try loading the latest version and re-compiling. For example, module load gcc/4.7.1 or module load gcc/4.9.1 depending on what versions you have.

@diegocrzt
Copy link

Thank you @rasmi, is clear for me now. I will see what can I do with my environment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants