failed to build xsmm_avx (libxsmm_spmdm.c)

[yvdriess]

I cannot seem to build tensorflow-libxsmm, as libxsmm itself fails to build. I have tried multiple options including getting the latest libxsmm-master archive and cloning benoit's branch.

Host: ubuntu 14.04 LTS (trusty), using gcc-6.2.0 (switched using update-alternative)
CPU: Haswell Xeon

bazel build command: bazel build --verbose_failures -c opt --copt=-O3 --copt=-fopenmp-simd --copt=-DLIBXSMM_OPENMP_SIMD --linkopt=-pthread --define tensorflow_xsmm=1 --define eigen_xsmm=1 --define tensorflow_xsmm_backward=1 --copt=-mfma --copt=-mavx2 //tensorflow/tools/pip_package:build_pip_package

ERROR: /home/yvandrie/.cache/bazel/_bazel_yvandrie/6b59d0d00420876982aa8096423cd141/external/libxsmm_archive/BUILD.bazel:46:1: C++ compilation of rule '@libxsmm_archive//:xsmm_avx' failed: gcc failed: error executing command
  (cd /home/yvandrie/.cache/bazel/_bazel_yvandrie/6b59d0d00420876982aa8096423cd141/execroot/tensorflow-xsmm && \
  exec env - \
    LD_LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/6.2.0/lib:/opt/intel/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.4.196/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.4.196/linux/tbb/lib/intel64/gcc4.7:/opt/intel/debugger_2017/iga/lib:/opt/intel/debugger_2017/libipt/intel64/lib:/opt/intel/compilers_and_libraries_2017.4.196/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.4.196/linux/daal/../tbb/lib/intel64_lin/gcc4.4 \
    PATH=/opt/intel/intelpython27/bin:/opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin:/opt/intel/debugger_2017/gdb/intel64_mic/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin \
    PYTHON_BIN_PATH=/opt/intel/intelpython27/bin/python \
    PYTHON_LIB_PATH=/opt/intel/intelpython27/lib/python2.7/site-packages \
    TF_NEED_CUDA=0 \
    TF_NEED_OPENCL=0 \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -O3 -fopenmp-simd -DLIBXSMM_OPENMP_SIMD -mfma -mavx2 -MD -MF bazel-out/local-opt/bin/external/libxsmm_archive/_objs/xsmm_avx/external/libxsmm_archive/src/libxsmm_spmdm.pic.d -fPIC -DLIBXSMM_BUILD '-D__BLAS=0' -iquote external/libxsmm_archive -iquote bazel-out/local-opt/genfiles/external/libxsmm_archive -iquote external/bazel_tools -iquote bazel-out/local-opt/genfiles/external/bazel_tools -isystem external/libxsmm_archive/include -isystem bazel-out/local-opt/genfiles/external/libxsmm_archive/include -isystem external/libxsmm_archive/src -isystem bazel-out/local-opt/genfiles/external/libxsmm_archive/src -isystem external/libxsmm_archive/src/template -isystem bazel-out/local-opt/genfiles/external/libxsmm_archive/src/template -isystem external/bazel_tools/tools/cpp/gcc3 -mavx -Wno-vla -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/libxsmm_archive/src/libxsmm_spmdm.c -o bazel-out/local-opt/bin/external/libxsmm_archive/_objs/xsmm_avx/external/libxsmm_archive/src/libxsmm_spmdm.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
/tmp/ccAcXVWE.s: Assembler messages:
/tmp/ccAcXVWE.s:11850: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:13050: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:14386: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:15730: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:15750: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:15756: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:15759: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:15762: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:15765: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:15771: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:18102: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:18158: Error: operand type mismatch for `vxorps'
/tmp/ccAcXVWE.s:19221: Error: operand size mismatch for `vmovdqu64'
/tmp/ccAcXVWE.s:19228: Error: no such instruction: `vinserti32x8 $0x1,0(%r13,%rcx,2),%zmm0,%zmm0'
/tmp/ccAcXVWE.s:19234: Error: operand type mismatch for `vpunpcklwd'
/tmp/ccAcXVWE.s:19235: Error: operand type mismatch for `vpunpckhwd'
/tmp/ccAcXVWE.s:19239: Error: operand size mismatch for `vmovdqu64'
/tmp/ccAcXVWE.s:19243: Error: no such instruction: `vinserti32x8 $0x1,0(%r13,%rdi,2),%zmm0,%zmm0'
/tmp/ccAcXVWE.s:19249: Error: operand type mismatch for `vpunpcklwd'
/tmp/ccAcXVWE.s:19250: Error: operand type mismatch for `vpunpckhwd'
/tmp/ccAcXVWE.s:19254: Error: operand size mismatch for `vmovdqu64'
/tmp/ccAcXVWE.s:19260: Error: no such instruction: `vinserti32x8 $0x1,0(%r13,%r9,2),%zmm0,%zmm0'
/tmp/ccAcXVWE.s:19265: Error: operand size mismatch for `vmovdqu64'
/tmp/ccAcXVWE.s:19267: Error: operand type mismatch for `vpunpcklwd'
/tmp/ccAcXVWE.s:19269: Error: operand type mismatch for `vpunpckhwd'
/tmp/ccAcXVWE.s:19277: Error: no such instruction: `vinserti32x8 $0x1,0(%r13,%r10,2),%zmm4,%zmm4'

[hfp]

Hello Yves,

you may adjust tensorflow/workspace.bzl (eigen section):

  native.new_http_archive(
      name = "eigen_archive",
      urls = [
          "https://github.com/hfp/eigen/archive/master.zip",
      ],
      sha256 = "9044e112e5d816c69f14c8b7bb3094ab95df6a4cf6941b507583f6a693c55149",
      strip_prefix = "eigen-master",
      build_file = str(Label("//third_party:eigen.BUILD")),
  )

You can also simply clone the "local" branch of the following:

git clone --branch local https://github.com/hfp/tensorflow-xsmm.git

If you do the latter, you need to eventually update the SHA256-sum in tensorflow/workspace.bzl (xsmm section).

[hfp]

Wait a second, I misread your output and thought it is related to AVX-512 compilation. However, it looks like you built GCC 6.x on an outdated Linux distribution (outdated wrt GNU Bintools). Please build GNU Bintools from source, and add/prepend the bin-folder to your PATH environment variable.

[hfp]

Background of the issue that you discovered is that the GNU Bintools assembler does not understand AVX2 instructions (I guess it's from the 2010 era or so).

[yvdriess]

Thanks for the quick follow-up. Outdated bintools might indeed be the problem (ubuntu 14.04 stopped at gcc 4.8, and is still relatively ubiquitous today). I had assumed that the alternatives tool would swap that out as well. Is there a way to compile tensorflow-libxsmm with the Intel compiler?

…

On Mon, May 22, 2017 at 7:05 PM, Hans Pabst ***@***.***> wrote: Background of the issue that you discovered is that the GNU Bintools assembler does not understand AVX2 instructions (I guess it's from the 2010 era or so). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#170 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC1_00HvQpdqxUgFxmgSLf9PRn66Httks5r8cBpgaJpZM4NinkI> .

[hfp]

I think so, but I have not tried it myself. It might be time-consuming, if one piece of the puzzle does not fit (the Binutils problem will be solved for sure, but who knows what else happens). Sorry, I said "Bintools" before, but meant "Binutils".

I really recommend you to use GCC (the one you already prepared), to compile Binutils (./configure --prefix=$HOME/binutils), and do export PATH=$HOME/binutils/bin:$PATH (see script below). I used this on a similar system with the same issue, and it worked. You do not need "to tell" your new GCC about the new Binutils or such, it simply works with a prefixed PATH variable.

After configure and make && make install, you may create a file $HOME/binutils/enable:

#!/bin/bash
BINUTILSROOT=${HOME}/binutils
export LD_LIBRARY_PATH=${BINUTILSROOT}/x86_64-pc-linux-gnu/lib:${BINUTILSROOT}/lib:${LD_LIBRARY_PATH}
export PATH=${BINUTILSROOT}/x86_64-pc-linux-gnu/bin:${BINUTILSROOT}/bin:${PATH}
unset BINUTILSROOT

Then you can source $HOME/binutils/enable, and you're done! With this in the PATH, your new GCC 6.x will just work.

[yvdriess]

quick note: the instructions it chokes on are AVX512 (zmm registers), even though I only passed -mavx2 -mfma

[hfp]

Yes, that's our technology enabling certain code paths using Intrinsics (we have a few intrinsic based codes in addition to JIT kernels e.g., in our SPMDM domain.). We apply the target flags for selected functions to enable compilers such as GCC for Intrinsics beyond the static/main code path (and dispatch according to CPUID). Though, with ICC you can use any kind of Intrinsic independent of the -m or -x code path. However, this is not true for GCC or Clang. It of course chokes if your assembler is outdated (Binutils). I will probably take this issue and incorporate additional check in our build system, and emit a message about outdated Binutils. Have you had any luck with newer Binutils?

[yvdriess]

I am still getting the same assembler errors using bintutils-2.28 as I was with binutils-2.24 before.

[hfp]

You may add --copt=-DLIBXSMM_INTRINSICS_STATIC to your Bazel command line, and thereby avoid the problem. Unfortunately this requires you to rely on the current Master revision of LIBXSMM (you can follow our guide on how to update to the master revision of LIBXSMM as well as updating the corresponding SHA256 sum).

[yvdriess]

I was able to successfully compile libxsmm (tensorflow-xsmm actually) using that new flag, thanks!

[hfp]

Thank you for sharing the final status!