Build fails #23

rasmi · 2016-09-05T04:57:28Z

My build fails when compiling through Bazel (while building TensorFlow).

ERROR: /hmt/sirius1/skv0/u/4/r/user/.cache/bazel/_bazel_user/d217f35631206796f447d50c6f1d6243/external/highwayhash/BUILD:125:1: C++ compilation of rule '@highwayhash//:sip_hash' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /hmt/sirius1/skv0/u/4/r/user/.cache/bazel/_bazel_user/d217f35631206796f447d50c6f1d6243/execroot/tensorflow && \
  exec env - \
    LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:/usr/local/gurobi/lib:/usr/local/cuda-7.5/extras/CUPTI/lib64 \
    PATH=/vega/astro/users/user/applications/pythonenv/bin:/opt/rh/devtoolset-1.1/root/usr/bin:/usr/local/cuda-7.5/bin:/vega/astro/users/user/applications/bazel/output:/vega/astro/users/user/applications/swig/bin:/vega/astro/users/user/applications/jdk1.8.0_102/bin:/usr/local/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/gurobi/bin:/opt/openlava-3.3/bin \
    TMPDIR=/tmp \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections -g0 '-std=c++11' -MD -MF bazel-out/host/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.d '-frandom-seed=bazel-out/host/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.o' -iquote external/highwayhash -iquote bazel-out/host/genfiles/external/highwayhash -iquote external/bazel_tools -iquote bazel-out/host/genfiles/external/bazel_tools -isystem external/highwayhash -isystem bazel-out/host/genfiles/external/highwayhash -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/highwayhash/highwayhash/sip_hash.cc -o bazel-out/host/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from external/highwayhash/highwayhash/sip_hash.h:24:0,
                 from external/highwayhash/highwayhash/sip_hash.cc:15:
external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)':
external/highwayhash/highwayhash/state_helpers.h:31:13: error: there are no arguments to 'alignas' that depend on a template parameter, so a declaration of 'alignas' must be available [-fpermissive]
external/highwayhash/highwayhash/state_helpers.h:31:13: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
external/highwayhash/highwayhash/state_helpers.h:31:15: error: expected ';' before 'char'
external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope
Target //tensorflow/cc:tutorials_example_trainer failed to build

Adding the -fpermissive does allow it to continue with warnings, at which point I get this error:

ERROR: /hmt/sirius1/skv0/u/4/r/user/.cache/bazel/_bazel_user/d217f35631206796f447d50c6f1d6243/external/highwayhash/BUILD:125:1: C++ compilation of rule '@highwayhash//:sip_hash' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /hmt/sirius1/skv0/u/4/r/user/.cache/bazel/_bazel_user/d217f35631206796f447d50c6f1d6243/execroot/tensorflow && \
  exec env - \
    LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:/usr/local/gurobi/lib:/usr/local/cuda-7.5/extras/CUPTI/lib64 \
    PATH=/vega/astro/users/user/applications/pythonenv/bin:/opt/rh/devtoolset-1.1/root/usr/bin:/usr/local/cuda-7.5/bin:/vega/astro/users/user/applications/bazel/output:/vega/astro/users/user/applications/swig/bin:/vega/astro/users/user/applications/jdk1.8.0_102/bin:/usr/local/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/gurobi/bin:/opt/openlava-3.3/bin \
    TMPDIR=/tmp \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-std=c++11' -MD -MF bazel-out/local_linux-opt/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.pic.d '-frandom-seed=bazel-out/local_linux-opt/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.pic.o' -fPIC -iquote external/highwayhash -iquote bazel-out/local_linux-opt/genfiles/external/highwayhash -iquote external/bazel_tools -iquote bazel-out/local_linux-opt/genfiles/external/bazel_tools -isystem external/highwayhash -isystem bazel-out/local_linux-opt/genfiles/external/highwayhash -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/highwayhash/highwayhash/sip_hash.cc -o bazel-out/local_linux-opt/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from external/highwayhash/highwayhash/sip_hash.h:24:0,
                 from external/highwayhash/highwayhash/sip_hash.cc:15:
external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)':
external/highwayhash/highwayhash/state_helpers.h:31:15: error: expected ';' before 'char'
external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope
Target //tensorflow/cc:tutorials_example_trainer failed to build

The text was updated successfully, but these errors were encountered:

jan-wassenberg · 2016-09-05T10:02:27Z

Hi, thanks for reporting. We require GCC 4.8 for "alignas". Is it an option for you to compile with a more recent GCC or clang?
Alternatively, you can replace alignas(32) with attribute((aligned(32))).

rasmi · 2016-09-05T14:23:41Z

Hi @jan-wassenberg, thanks for the reply. Here's my diff:

-  alignas(32) char final_packet[State::kPacketSize] = {0};
+  attribute((aligned(32))) char final_packet[State::kPacketSize] = {0};

Unfortunately, compiling with gcc 4.9.1 leads with a whole set of other errors (though not with highwayhash), and the next-closest version I have is 4.7.2. I'm using a compute cluster so the gcc versions available to me are limited.
I'm now getting the same error:

In file included from external/highwayhash/highwayhash/sip_hash.h:24:0,
                 from external/highwayhash/highwayhash/sip_hash.cc:15:
external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)':
external/highwayhash/highwayhash/state_helpers.h:31:28: error: expected ';' before 'char'
external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope

Here's the block of code. Could it be a syntax issue?

#ifndef HIGHWAYHASH_HIGHWAYHASH_STATE_H_
#define HIGHWAYHASH_HIGHWAYHASH_STATE_H_

// Helper functions to split inputs into packets and call State::Update on each.

#include <cstddef>
#include <cstring>
#include <memory>

#include "highwayhash/code_annotation.h"
#include "highwayhash/types.h"

namespace highwayhash {

// Copies the remaining bytes to a zero-padded buffer, sets the upper byte to
// size % 256 (always possible because this should only be called if the
// total size is not a multiple of the packet size) and updates hash state.
//
// The padding scheme is essentially from SipHash, but permuted for the
// convenience of AVX-2 masked loads. This function must use the same layout so
// that the vector and scalar HighwayTreeHash have the same result.
//
// "remaining_size" is the number of accessible/remaining bytes
// (size % kPacketSize).
//
// Primary template; the specialization for AVX-2 is faster. Intended as an
// implementation detail, do not call directly.
template <class State>
INLINE void PaddedUpdate(const uint64 size, const char* remaining_bytes,
                         const uint64 remaining_size, State* state) {
  attribute((aligned(32))) char final_packet[State::kPacketSize] = {0};

  // Unusual layout matches the AVX-2 specialization in highway_tree_hash.h.
  const size_t remainder_mod4 = remaining_size & 3;
  uint32 packet4 = static_cast<uint32>(size) << 24;
  const char* final_bytes = remaining_bytes + remaining_size - remainder_mod4;
  for (size_t i = 0; i < remainder_mod4; ++i) {
    const uint32 byte = static_cast<unsigned char>(final_bytes[i]);
    packet4 += byte << (i * 8);
  }

  memcpy(final_packet, remaining_bytes, remaining_size - remainder_mod4);
  memcpy(final_packet + State::kPacketSize - 4, &packet4, sizeof(packet4));

  state->Update(final_packet);
}

// Updates hash state for every whole packet, and once more for the final
// padded packet.
template <class State>
INLINE void UpdateState(const char* bytes, const uint64 size, State* state) {
  // Feed entire packets.
  const int kPacketSize = State::kPacketSize;
  static_assert((kPacketSize & (kPacketSize - 1)) == 0, "Size must be 2^i.");
  const size_t remainder = size & (kPacketSize - 1);
  const size_t truncated_size = size - remainder;
  for (size_t i = 0; i < truncated_size; i += kPacketSize) {
    state->Update(bytes + i);
  }

  PaddedUpdate(size, bytes + truncated_size, remainder, state);
}

// Convenience function for updating with the bytes of a string.
template <class String, class State>
INLINE void UpdateState(const String& s, State* state) {
  const char* bytes = reinterpret_cast<const char*>(s.data());
  const size_t size = s.length() * sizeof(typename String::value_type);
  UpdateState(bytes, size, state);
}

// Computes a hash of a byte array using the given hash State class.
//
// Example: const SipHashState::Key key = { 1, 2 }; char data[4];
// ComputeHash<SipHashState>(key, data, sizeof(data));
//
// This function avoids duplicating Update/Finalize in every call site.
// Callers wanting to combine multiple hashes should repeatedly UpdateState()
// and only call State::Finalize once.
template <class State>
uint64 ComputeHash(const typename State::Key& key, const char* bytes,
                   const uint64 size) {
  State state(key);
  UpdateState(bytes, size, &state);
  return state.Finalize();
}

// Computes a hash of a string's bytes using the given hash State class.
//
// Example: const SipHashState::Key key = { 1, 2 };
// StringHasher<SipHashState>()(key, std::u16string(u"abc"));
//
// A struct with nested function template enables deduction of the String type.
template <class State>
struct StringHasher {
  template <class String>
  uint64 operator()(const typename State::Key& key, const String& s) {
    State state(key);
    UpdateState(s, &state);
    return state.Finalize();
  }
};

}  // namespace highwayhash

#endif  // HIGHWAYHASH_HIGHWAYHASH_STATE_H_

jan-wassenberg · 2016-09-05T14:33:27Z

Hi, good news, it will compile with just a minor change; GCC wants the type first and attribute second.
attribute((aligned(32))) char -> char attribute((aligned(32))).

rasmi · 2016-09-05T14:41:18Z

@jan-wassenberg still getting a similar error. Sorry for the silly fixes:

In file included from external/highwayhash/highwayhash/sip_hash.h:24:0,
                 from external/highwayhash/highwayhash/sip_hash.cc:15:
external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)':
external/highwayhash/highwayhash/state_helpers.h:31:33: error: expected ',' or ';' before 'final_packet'
external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope

jan-wassenberg · 2016-09-05T15:53:48Z

hehe, I see what happened. The Markdown syntax ate the underscores; it should read char __attribute__((aligned(32))) final_packet[State::kPacketSize] = {0};.

rasmi · 2016-09-05T21:50:52Z

@jan-wassenberg Thank you so much! That worked!

mnicky · 2016-11-02T14:51:46Z

I've run into this issue (while compiling tensorflow) as well.

@jan-wassenberg: if the change you suggested makes the code compatible with more gcc versions, why not to make it permanent?

jan-wassenberg · 2016-11-02T15:24:43Z

Hi @mnicky, it's a tradeoff - one disadvantage of __attribute__ is that it's not supported by MSVC. I had hoped that C++11 is widely available by now. Can you share which GCC version you have, and why it's infeasible to use a more recent version? Perhaps we might switch to a macro if necessary.

rasmi · 2016-11-02T15:28:27Z

@mnicky, if you are in a scientific computing Red Hat / CentOS environment, you might have more recent versions of gcc available. Try module load gcc<tab complete> to check. The default on my system was much older than the other available versions. Disregard if this isn't your situation...

mnicky · 2016-11-02T16:24:38Z

I have gcc 4.7.2 which is included in Debian Wheezy that runs on our cluster. In this case I'm able to modify the source file to get rid of the compilation error. I was just curious why the workaround/fix didn't get it into the master. I think there are still quite a few people running older versions of gcc without possibility of upgrading. And as highwayhash is needed to build tensorflow, I'd assume there are people affected by this.

jan-wassenberg · 2016-11-02T17:47:57Z

Ah, thanks for mentioning your GCC version. That's still fairly recent; it is unfortunate that alignas didn't make it in until 4.8. We will soon add a HH_ALIGNAS macro that resolves to alignas or falls back to attribute.

jan-wassenberg · 2016-11-04T12:15:49Z

The fix is in - please reopen if there's still an issue :)

Sinan81 · 2017-02-03T15:48:34Z

I am using GCC5.3.0, and getting a similar error:
C++ compilation of rule '@highwayhash//:sip_hash' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command

My set up is:
Bazel 0.4.3 (compiled from source)
Tensorflow 0.12.1 (from zip file)
GCC 5.3.0
Cuda 8.0
CuDNN 5.1

jan-wassenberg · 2017-02-03T15:57:40Z

Can you include more output to show the actual compile error and in which file/header it occurs?
Should be a few lines below the "error executing command" you mentioned.

Sinan81 · 2017-02-06T17:35:44Z

Sorry for the late reply. I started from scratch (including removing ~/.cache/bazel...), and it seems TF now compiles fine. Should I delete my previous message?

jan-wassenberg · 2017-02-06T17:49:58Z

Glad to hear it! No worries, we can leave this issue unchanged/closed.

diegocrzt · 2017-03-15T20:43:51Z

Hi guys/girls, I am having the same problem with gcc 4.7.4
ERROR: /home/diegocrzt/.cache/bazel/_bazel_diegocrzt/8df0525fbbd1e101024ee0b817c7eaef/external/highwayhash/BUILD:125:1: C++ compilation of rule '@highwayhash//:sip_hash' failed: gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 32 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. In file included from external/highwayhash/highwayhash/sip_hash.h:24:0, from external/highwayhash/highwayhash/sip_hash.cc:15: external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)': external/highwayhash/highwayhash/state_helpers.h:31:13: error: there are no arguments to 'alignas' that depend on a template parameter, so a declaration of 'alignas' must be available [-fpermissive] external/highwayhash/highwayhash/state_helpers.h:31:13: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated) external/highwayhash/highwayhash/state_helpers.h:31:15: error: expected ';' before 'char' external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope Target //tensorflow/tools/pip_package:build_pip_package failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 239.020s, Critical Path: 225.21s
But after reading this I am not sure what to do. And I can't upgrade my version of GCC

rasmi · 2017-03-15T22:05:52Z

@diegocrzt, you need to edit the file external/highwayhash/highwayhash/state_helpers.h to make the changes to line 31 that @jan-wassenberg listed above. It should read
char __attribute__((aligned(32))) final_packet[State::kPacketSize] = {0};
Then try re-compiling.

diegocrzt · 2017-03-17T03:23:59Z

@rasmi, thank you for you quick answer and sorry for not understanding at first time, I was lost on the first markdown issues 😆, highwayhash step is done but right now I have other issue, with the core tensorflow, I think this is probably related no more to this thread, but just to be sure, what do you think, it could be related yet ?

ERROR: /home/diegocrzt/codice/dpi/tensorflow/tensorflow-git/tensorflow/contrib/layers/BUILD:19:1: C++ compilation of rule '//tensorflow/contrib/layers:python/ops/_bucketization_op.so' failed: gcc failed: error executing command 
  (cd /home/diegocrzt/.cache/bazel/_bazel_diegocrzt/8df0525fbbd1e101024ee0b817c7eaef/execroot/tensorflow-git && \
  exec env - \
    PATH=/usr/local/bin:/bin:/usr/bin:/opt/texlive/2016/bin/x86_64-linux:/opt/java/jdk/bin \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-march=native' '-std=c++0x' -MD -MF bazel-out/local-py3-opt/bin/tensorflow/contrib/layers/_objs/python/ops/_bucketization_op.so/tensorflow/contrib/layers/ops/bucketization_op.pic.d '-frandom-seed=bazel-out/local-py3-opt/bin/tensorflow/contrib/layers/_objs/python/ops/_bucketization_op.so/tensorflow/contrib/layers/ops/bucketization_op.pic.o' -fPIC -DEIGEN_MPL2_ONLY -iquote . -iquote bazel-out/local-py3-opt/genfiles -iquote external/bazel_tools -iquote bazel-out/local-py3-opt/genfiles/external/bazel_tools -iquote external/eigen_archive -iquote bazel-out/local-py3-opt/genfiles/external/eigen_archive -iquote external/local_config_sycl -iquote bazel-out/local-py3-opt/genfiles/external/local_config_sycl -iquote external/protobuf -iquote bazel-out/local-py3-opt/genfiles/external/protobuf -isystem external/bazel_tools/tools/cpp/gcc3 -isystem external/eigen_archive -isystem bazel-out/local-py3-opt/genfiles/external/eigen_archive -isystem external/protobuf/src -isystem bazel-out/local-py3-opt/genfiles/external/protobuf/src -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare -fno-exceptions -pthread -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c tensorflow/contrib/layers/ops/bucketization_op.cc -o bazel-out/local-py3-opt/bin/tensorflow/contrib/layers/_objs/python/ops/_bucketization_op.so/tensorflow/contrib/layers/ops/bucketization_op.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from ./tensorflow/core/lib/gtl/array_slice.h:101:0,
                 from ./tensorflow/core/lib/strings/str_util.h:23,
                 from ./tensorflow/core/framework/op.h:29,
                 from tensorflow/contrib/layers/ops/bucketization_op.cc:16:
./tensorflow/core/lib/gtl/array_slice_internal.h:232:38: error: 'tensorflow::gtl::array_slice_internal::ArraySliceImplBase<const T>::ArraySliceImplBase' names constructor
./tensorflow/core/lib/gtl/array_slice_internal.h:252:32: error: 'tensorflow::gtl::array_slice_internal::ArraySliceImplBase<T>::ArraySliceImplBase' names constructor
In file included from ./tensorflow/core/lib/gtl/array_slice.h:102:0,
                 from ./tensorflow/core/lib/strings/str_util.h:23,
                 from ./tensorflow/core/framework/op.h:29,
                 from tensorflow/contrib/layers/ops/bucketization_op.cc:16:
./tensorflow/core/lib/gtl/inlined_vector.h: In member function 'void tensorflow::gtl::InlinedVector<T, N>::Destroy(T*, int)':
./tensorflow/core/lib/gtl/inlined_vector.h:396:10: error: 'is_trivially_destructible' is not a member of 'std'
./tensorflow/core/lib/gtl/inlined_vector.h:396:42: error: expected primary-expression before '>' token
./tensorflow/core/lib/gtl/inlined_vector.h:396:43: error: '::value' has not been declared
Target //tensorflow/tools/pip_package:build_pip_package failed to build

rasmi · 2017-03-17T03:39:25Z

@diegocrzt no problem. That is an unrelated issue. According to this thread, your version of gcc is too old. If you are using a scientific computing environment, try typing module load gcc and tab-complete that command before running it to see which versions you have available. Try loading the latest version and re-compiling. For example, module load gcc/4.7.1 or module load gcc/4.9.1 depending on what versions you have.

diegocrzt · 2017-03-17T11:32:41Z

Thank you @rasmi, is clear for me now. I will see what can I do with my environment

This was referenced Sep 5, 2016

Using bazel to cross-compile tensorflow for other targets. bazelbuild/bazel#1353

Closed

ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path. tensorflow/tensorflow#4105

Closed

rasmi closed this as completed Sep 5, 2016

jan-wassenberg reopened this Nov 2, 2016

jan-wassenberg closed this as completed Nov 4, 2016

Build fails #23

Build fails #23

rasmi commented Sep 5, 2016 •

edited

jan-wassenberg commented Sep 5, 2016

rasmi commented Sep 5, 2016

jan-wassenberg commented Sep 5, 2016 •

edited

rasmi commented Sep 5, 2016

jan-wassenberg commented Sep 5, 2016 •

edited

rasmi commented Sep 5, 2016

mnicky commented Nov 2, 2016

jan-wassenberg commented Nov 2, 2016 •

edited

rasmi commented Nov 2, 2016

mnicky commented Nov 2, 2016

jan-wassenberg commented Nov 2, 2016 •

edited

jan-wassenberg commented Nov 4, 2016

Sinan81 commented Feb 3, 2017 •

edited

jan-wassenberg commented Feb 3, 2017

Sinan81 commented Feb 6, 2017

jan-wassenberg commented Feb 6, 2017

diegocrzt commented Mar 15, 2017 •

edited

rasmi commented Mar 15, 2017

diegocrzt commented Mar 17, 2017

rasmi commented Mar 17, 2017 •

edited

diegocrzt commented Mar 17, 2017

Build fails #23

Build fails #23

Comments

rasmi commented Sep 5, 2016 • edited

jan-wassenberg commented Sep 5, 2016

rasmi commented Sep 5, 2016

jan-wassenberg commented Sep 5, 2016 • edited

rasmi commented Sep 5, 2016

jan-wassenberg commented Sep 5, 2016 • edited

rasmi commented Sep 5, 2016

mnicky commented Nov 2, 2016

jan-wassenberg commented Nov 2, 2016 • edited

rasmi commented Nov 2, 2016

mnicky commented Nov 2, 2016

jan-wassenberg commented Nov 2, 2016 • edited

jan-wassenberg commented Nov 4, 2016

Sinan81 commented Feb 3, 2017 • edited

jan-wassenberg commented Feb 3, 2017

Sinan81 commented Feb 6, 2017

jan-wassenberg commented Feb 6, 2017

diegocrzt commented Mar 15, 2017 • edited

rasmi commented Mar 15, 2017

diegocrzt commented Mar 17, 2017

rasmi commented Mar 17, 2017 • edited

diegocrzt commented Mar 17, 2017

rasmi commented Sep 5, 2016 •

edited

jan-wassenberg commented Sep 5, 2016 •

edited

jan-wassenberg commented Sep 5, 2016 •

edited

jan-wassenberg commented Nov 2, 2016 •

edited

jan-wassenberg commented Nov 2, 2016 •

edited

Sinan81 commented Feb 3, 2017 •

edited

diegocrzt commented Mar 15, 2017 •

edited

rasmi commented Mar 17, 2017 •

edited