Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Code fails to compile on Mac (issue with AVX intrinsics) #471

Closed
eikehmueller opened this issue Oct 13, 2015 · 6 comments
Closed

Code fails to compile on Mac (issue with AVX intrinsics) #471

eikehmueller opened this issue Oct 13, 2015 · 6 comments
Assignees
@FabioLuporini
Labels
notabug

Comments

@eikehmueller
Copy link

Compilation on my MacBook fails with error messages of the type:

[…]
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccIqsgIb.s:9:no such instruction: `vxorpd %xmm5, %xmm5,%xmm5'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccIqsgIb.s:12:no such instruction: `vmovsd LC1(%rip), %xmm2’
[…]

This can be fixed by adding the compiler flag -Wa,-q, see following patch:

diff --git a/pyop2/compilation.py b/pyop2/compilation.py
index ec7d7af..2daf491 100644
--- a/pyop2/compilation.py
+++ b/pyop2/compilation.py
@@ -210,7 +210,7 @@ class MacCompiler(Compiler):
         if cpp:
             cc = "mpicxx"
             stdargs = []
-        cppargs = stdargs + ['-fPIC', '-Wall', '-framework', 'Accelerate'] + \
+        cppargs = stdargs + ['-Wa,-q','-fPIC', '-Wall', '-framework', 'Accelerate'] + \
             opt_flags + cppargs
         ldargs = ['-dynamiclib'] + ldargs
         super(MacCompiler, self).__init__(cc,

I only tested this on my macbook, the used compiler is gcc:

eikemueller@Eikes-MBP $ mpicxx --version
g++ (Homebrew gcc48 4.8.3) 4.8.3
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
@FabioLuporini
Copy link
Contributor

this honestly doesn't look like stuff that coffee spits out. Can you paste here the whole compilation line? What if you replace that option with just -O3 ?

@eikehmueller
Copy link
Author

986dce2f61c72a82742bf5109164330b_p7138.txt

Compilation line is (so the -O3 flag is already included):

mpicc -std=c99 -fPIC -Wall -framework Accelerate -march=native -O3 -I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/include -I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/include -I/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2 -msse -o /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.so.tmp /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.c -dynamiclib -L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib -L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib -Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib -Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib -lpetsc -l

The full error message is this:

  File "driver.py", line 652, in <module>
    main(parameter_filename)
  File "driver.py", line 569, in main
    mesh_hierarchy = build_mesh_hierarchy(param_grid,param_orography)
  File "driver.py", line 166, in build_mesh_hierarchy
    layer_height= thickness/nlayer)
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/firedrake/firedrake/mg/mesh.py", line 121, in __init__
    for lvl, m in enumerate(mesh_hierarchy)]
  File "<string>", line 2, in __init__
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/profiling.py", line 199, in wrapper
    return f(*args, **kwargs)
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/firedrake/firedrake/mesh.py", line 901, in __init__
    kernel=kernel)
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/firedrake/firedrake/extrusion_utils.py", line 236, in make_extruded_coords
    height(op2.READ))
  File "<string>", line 2, in par_loop
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/versioning.py", line 154, in modifies_arguments
    retval = func(*args, **kwargs)
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/op2.py", line 328, in par_loop
    return backends._BackendSelector._backend.par_loop(kernel, iterset, *args, **kwargs)
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/base.py", line 4588, in par_loop
    return _make_object('ParLoop', kernel, it_space, *args, **kwargs).enqueue()
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/base.py", line 84, in enqueue
    _trace.append(self)
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/base.py", line 101, in append
    computation._run()
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/base.py", line 4045, in _run
    return self.compute()
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/base.py", line 4276, in compute
    fun = self._jitmodule
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/utils.py", line 64, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/sequential.py", line 147, in _jitmodule
    direct=self.is_direct, iterate=self.iteration_region)
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/caching.py", line 203, in __new__
    obj = make_obj()
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/caching.py", line 193, in make_obj
    obj.__init__(*args, **kwargs)
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/host.py", line 687, in __init__
    self.compile()
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/host.py", line 788, in compile
    compiler=compiler.get('name'))
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/compilation.py", line 298, in load
    dll = compiler.get_so(src, extension)
  File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/compilation.py", line 144, in get_so
    Original error: %s""" % (cc, logfile, errfile, e))
pyop2.exceptions.CompilationError: Command "['mpicc', '-std=c99', '-fPIC', '-Wall', '-framework', 'Accelerate', '-march=native', '-O3', '-I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/include', '-I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/include', '-I/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2', '-msse', '-o', '/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.so.tmp', '/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.c', '-dynamiclib', '-L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib', '-L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib', '-Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib', '-Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib', '-lpetsc', '-lm']" returned with error.
Unable to compile code
Compile log in /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.log
Compile errors in /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.err
Original error: status 1 invoking 'mpicc -std=c99 -fPIC -Wall -framework Accelerate -march=native -O3 -I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/include -I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/include -I/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2 -msse -o /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.so.tmp /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.c -dynamiclib -L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib -L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib -Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib -Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib -lpetsc -lm'

And the contents of the file /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.err are:

/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:9:no such instruction: `vxorpd %xmm5, %xmm5,%xmm5'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:12:no such instruction: `vmovsd LC1(%rip), %xmm2'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:15:no such instruction: `vmovsd (%rax), %xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:16:no such instruction: `vmulsd %xmm0, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:17:no such instruction: `vaddsd %xmm5, %xmm0,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:18:no such instruction: `vmovsd 8(%rax), %xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:19:no such instruction: `vmulsd %xmm0, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:20:no such instruction: `vaddsd %xmm0, %xmm1,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:21:no such instruction: `vmovsd 16(%rax), %xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:22:no such instruction: `vmulsd %xmm0, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:23:no such instruction: `vaddsd %xmm0, %xmm1,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:24:no such instruction: `vsqrtsd %xmm0, %xmm1,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:25:no such instruction: `vucomisd %xmm1, %xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:33:no such instruction: `vcvtsi2sd %r10d, %xmm3,%xmm3'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:36:no such instruction: `vmulsd (%rcx), %xmm3,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:37:no such instruction: `vdivsd %xmm1, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:38:no such instruction: `vaddsd %xmm2, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:39:no such instruction: `vmulsd (%rax), %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:40:no such instruction: `vmovsd %xmm0, (%r9)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:41:no such instruction: `vcvtsi2sd %r10d, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:42:no such instruction: `vmulsd (%rcx), %xmm0,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:43:no such instruction: `vdivsd %xmm1, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:44:no such instruction: `vaddsd %xmm2, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:45:no such instruction: `vmulsd (%rax), %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:46:no such instruction: `vmovsd %xmm4, (%r8)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:47:no such instruction: `vmulsd (%rcx), %xmm3,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:48:no such instruction: `vdivsd %xmm1, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:49:no such instruction: `vaddsd %xmm2, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:50:no such instruction: `vmulsd 8(%rax), %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:51:no such instruction: `vmovsd %xmm4, 8(%r9)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:52:no such instruction: `vmulsd (%rcx), %xmm0,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:53:no such instruction: `vdivsd %xmm1, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:54:no such instruction: `vaddsd %xmm2, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:55:no such instruction: `vmulsd 8(%rax), %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:56:no such instruction: `vmovsd %xmm4, 8(%r8)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:57:no such instruction: `vmulsd (%rcx), %xmm3,%xmm3'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:58:no such instruction: `vdivsd %xmm1, %xmm3,%xmm3'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:59:no such instruction: `vaddsd %xmm2, %xmm3,%xmm3'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:60:no such instruction: `vmulsd 16(%rax), %xmm3,%xmm3'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:61:no such instruction: `vmovsd %xmm3, 16(%r9)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:62:no such instruction: `vmulsd (%rcx), %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:63:no such instruction: `vdivsd %xmm1, %xmm0,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:64:no such instruction: `vaddsd %xmm2, %xmm1,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:65:no such instruction: `vmulsd 16(%rax), %xmm1,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:66:no such instruction: `vmovsd %xmm1, 16(%r8)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:75:no such instruction: `vmovsd %xmm2, 40(%rsp)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:76:no such instruction: `vmovsd %xmm5, 32(%rsp)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:83:no such instruction: `vmovapd %xmm0, %xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:84:no such instruction: `vmovsd 40(%rsp), %xmm2'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:85:no such instruction: `vmovsd 32(%rsp), %xmm5'

eikehmueller added a commit that referenced this issue Oct 20, 2015
@eikehmueller
Added -Wa,-q to compiler flags
3377113 
This flag fixes the compile errors reported in issue number 471 (#471).
eikehmueller added a commit that referenced this issue Oct 21, 2015
@eikehmueller
Added -Wa,-q to compiler flags
7ecb244 
This flag fixes the compile errors reported in issue number 471 (#471).
eikehmueller added a commit that referenced this issue Oct 27, 2015
@eikehmueller
Added -Wa,-q to compiler flags
6688e9e 
This flag fixes the compile errors reported in issue number 471 (#471).
eikehmueller added a commit that referenced this issue Oct 27, 2015
@eikehmueller
Added -Wa,-q to compiler flags
b57389e 
This flag fixes the compile errors reported in issue number 471 (#471).
@wence-
Copy link
Member

wence- commented Nov 17, 2015

I think this occurs because the linker you use doesn't understand the avx instructions set. Passing -Wa,-q forces the compiler to always use the system linker which does understand things. Plausibly this is always safe on Mac.

@eikehmueller
Copy link
Author

Thanks! I think there’s something not quite right with my mac, and the compiler toolchain is screwed up (gcc broke when I upgraded to Yosemite a year ago, so this might still be the aftermath...).

The error could be related to the issues I’m seeing with building PETSc (still not resolved - I’m in touch with Satish on petsc@maint). We located the error to this line now:

mpif90 -c -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O   sltimer.f
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//cccVkhkX.s:85:suffix or operands invalid for `movq'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//cccVkhkX.s:96:suffix or operands invalid for `movq’

This looks different since it is a compile error, not a linker error.

I will try to rebuild my entire homebrew tomorrow, maybe this helps.

On 17 Nov 2015, at 19:45, Lawrence Mitchell notifications@github.com wrote:

I think this occurs because the linker you use doesn't understand the avx instructions set. Passing -Wa,-q forces the compiler to always use the system linker which does understand things. Plausibly this is always safe on Mac.

—
Reply to this email directly or view it on GitHub #471 (comment).

@wence-
Copy link
Member

wence- commented Nov 17, 2015

I suspect it very probably has the same root cause.

@eikehmueller
Copy link
Author

It was my compiler toolchain that was broken. I just upgraded my MacBook to OSX El Capitan, updated XCode and reinstalled all homebrew packages. After this I don't need the "-Wa,-q" flag any more (and PETSc compiles without problem as well).

@miklos1 miklos1 closed this as completed Nov 20, 2015
@eikehmueller eikehmueller added notabug and removed bug labels Nov 20, 2015
muellermartin added a commit to muellermartin/homebrew-core that referenced this issue Jul 4, 2016
@muellermartin
Try to fix build error on OS X Mavericks
5f5909a 
The test build on OS X Mavericks failed with errors like "no such
instruction: `vmovss LC1(%rip), %xmm2'" which might relate to AVX
intrinsics - at least that's what I've found in a similar issue:
OP2/PyOP2#471
The suggested fix is to add the compiler flag "-Wa,-q" which this
commit tries to do.
This was referenced Jul 4, 2016
Closed
Closed
muellermartin added a commit to muellermartin/homebrew-core that referenced this issue Mar 26, 2017
@muellermartin
Try to fix build error on OS X Mavericks
f6d10ba 
The test build on OS X Mavericks failed with errors like "no such
instruction: `vmovss LC1(%rip), %xmm2'" which might relate to AVX
intrinsics - at least that's what I've found in a similar issue:
OP2/PyOP2#471
The suggested fix is to add the compiler flag "-Wa,-q" which this
commit tries to do.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@FabioLuporini FabioLuporini

Labels
notabug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@FabioLuporini @wence- @miklos1 @eikehmueller