Problem when compiling with newer versions of Intel compilers (14 and 15) #512

luet · 2015-05-20T18:54:15Z

I am having trouble compiling specfem3d with newer versions of the Intel compilers (14.0.0 and 15.0.2) with different versions of MPI (Open MPI 1.6.3 and Intel MPI 5.0.3).

Note that:

It works with:
- Intel 13.0 with Open MPI 1.6.3.
- the GNU compilers (4.6.8) with Open MPI 1.6.3.
All the steps are successful with specfem3d_globe.

Here is a summary of my tests:

Compiler	MPI	step	Status
Intel 13.0	OpenMPI 1.6.3	configure	OK
		make	OK
		EXAMPLE	OK
Intel 14.0	Open MPI 1.6.3	configure	OK
		make	OK
		EXAMPLE	fail, seg fault
Intel 14.0	Intel MPI 5.0.3	configure	OK
		make	OK
		EXAMPLE	fail, seg fault
Intel 15.0.2	Open MPI 1.6.3	configure	OK
		make	fail, seg fault
Intel 15.0.2	Intel MPI 5.0.3	configure	OK
		make	fail, seg fault
GNU 4.6.8	Open MPI 1.6.3	configure	OK
		make	OK
		EXAMPLE	OK

The details of the tests follows.

Intel 15.0.2 + Intel MPI and Intel 15.0.2 + Open MPI 1.6.3 (same output)

configure:

/configure FC=ifort CC=icc MPIFC=mpif90 MPICC=mpicc --with-mpi --enable-double-precision

compile error

./obj/force_ftz.cc.o: In function `_mm_getcsr':
/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/include/xmmintrin.h:794: undefined reference to `__builtin_ia32_stmxcsr'
./obj/force_ftz.cc.o: In function `_mm_setcsr':
/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/include/xmmintrin.h:826: undefined reference to `__builtin_ia32_ldmxcsr'
make: *** [bin/xspecfem3D] Error 1

The file that it fails to compile is

/home/buildbot/GitRepo/specfem3d/src/shared/force_ftz.cc

Intel 14 + Openmpi 1.6.3

configure:

./configure FC=ifort CC=icc MPIFC=mpif90 MPICC=mpicc --with-mpi --enable-double-precision

compiles properly
Running example homogeneous_halfspace_HEX8_elastic_absorbing_Stacey_5sides fails.
I ran with:

./run_this_example

The error I get:

Databases files in directory: ./OUTPUT_FILES/DATABASES_MPI/
finished successfully


running database generation on  4 processors...


running solver on  4 processors...

--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 17013 on node clover exited on 
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

see results in directory: OUTPUT_FILES/

done
Fri May 15 15:06:30 EDT 2015

Intel 14.0 + Intel MPI

configure

./configure FC=ifort CC=icc MPIFC=mpif90 MPICC=mpicc --with-mpi --enable-double-precision

EXAMPLES/homogeneous_halfspace_HEX8_elastic_absorbing_Stacey_5sides
run_this_example.sh
Error:

```
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
xspecfem3D         00000000006C6C19  Unknown               Unknown  Unknown
xspecfem3D         00000000006C5590  Unknown               Unknown  Unknown
xspecfem3D         0000000000677B92  Unknown               Unknown  Unknown
xspecfem3D         0000000000610243  Unknown               Unknown  Unknown
xspecfem3D         0000000000614B1B  Unknown               Unknown  Unknown
libpthread.so.0    000000330F20F710  Unknown               Unknown  Unknown
libmpi.so.12       00007FA9E9E4591C  Unknown               Unknown  Unknown
libmpi.so.12       00007FA9E9E2E9AB  Unknown               Unknown  Unknown
libmpi.so.12       00007FA9E9C7EA91  Unknown               Unknown  Unknown
libmpi.so.12       00007FA9E9E1E6B3  Unknown               Unknown  Unknown
libmpi.so.12       00007FA9E9DDBE4D  Unknown               Unknown  Unknown
libmpi.so.12       00007FA9E9DCD5D1  Unknown               Unknown  Unknown
libmpifort.so.12   00007FA9EA35A750  Unknown               Unknown  Unknown
xspecfem3D         0000000000605D9C  Unknown               Unknown  Unknown
xspecfem3D         000000000055685A  Unknown               Unknown  Unknown
xspecfem3D         0000000000403E16  Unknown               Unknown  Unknown
libc.so.6          000000330E61ED5D  Unknown               Unknown  Unknown
xspecfem3D         0000000000403D09  Unknown               Unknown  Unknown
```

The text was updated successfully, but these errors were encountered:

QuLogic · 2015-05-20T18:57:21Z

Can you configure with --enable-debug to narrow down the location of the crash?

komatits · 2015-05-20T19:40:14Z

Hi all,

That compiler is known for being unstable when new versions appear; just
wait for the first update / first service pack and the problems should
be solved.

We've noticed that with v6.0, 7.0, 8.0, 9.0, 10.0 and 11.0 (and then I
stopped installing dot zero versions...)

Dimitri.

PS: ./obj/force_ftz.cc.o: In function _mm_getcsr': /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/include/xmmintrin.h:794: undefined reference to__builtin_ia32_stmxcsr' is related to a missing
library (see in src/shared/force_ftz.c)

On 05/20/2015 08:54 PM, David Luet wrote:

I am having trouble compiling specfem3d with newer versions of the Intel
compilers (14.0.0 and 15.0.2) with different versions of MPI (Open MPI
1.6.3 and Intel MPI 5.0.3).

Note that:

It works with:

Intel 13.0 with Open MPI 1.6.3.

the GNU compilers (4.6.8) with Open MPI 1.6.3.

All the steps are successful with specfem3d_globe.

Here is a summary of my tests:

Compiler MPI step Status
Intel 13.0 OpenMPI 1.6.3 configure OK
make OK
EXAMPLE OK
Intel 14.0 Open MPI 1.6.3 configure OK
make OK
EXAMPLE fail, seg fault
Intel 14.0 Intel MPI 5.0.3 configure OK
make OK
EXAMPLE fail, seg fault
Intel 15.0.2 Open MPI 1.6.3 configure OK
make fail, seg fault
Intel 15.0.2 Intel MPI 5.0.3 configure OK
make fail, seg fault
GNU 4.6.8 Open MPI 1.6.3 configure OK
make OK
EXAMPLE OK

The details of the tests follows.

Intel 15.0.2 + Intel MPI and Intel 15.0.2 + Open MPI 1.6.3 (same output)
configure:

|/configure FC=ifort CC=icc MPIFC=mpif90 MPICC=mpicc --with-mpi --enable-double-precision
|
compile error

|./obj/force_ftz.cc.o: In function `_mm_getcsr':
/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/include/xmmintrin.h:794: undefined reference to `__builtin_ia32_stmxcsr'
./obj/force_ftz.cc.o: In function `_mm_setcsr':
/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/include/xmmintrin.h:826: undefined reference to `__builtin_ia32_ldmxcsr'
make: *** [bin/xspecfem3D] Error 1
|

The file that it fails to compile is

|/home/buildbot/GitRepo/specfem3d/src/shared/force_ftz.cc
|
Intel 14 + Openmpi 1.6.3
configure:

|./configure FC=ifort CC=icc MPIFC=mpif90 MPICC=mpicc --with-mpi --enable-double-precision
|
compiles properly
*

Running example
homogeneous_halfspace_HEX8_elastic_absorbing_Stacey_5sides fails.
I ran with:

|./run_this_example
|

The error I get:

|Databases files in directory: ./OUTPUT_FILES/DATABASES_MPI/
finished successfully

running database generation on 4 processors...

running solver on 4 processors...

mpirun noticed that process rank 0 with PID 17013 on node clover exited on
signal 11 (Segmentation fault).

see results in directory: OUTPUT_FILES/

done
Fri May 15 15:06:30 EDT 2015
|

Intel 14.0 + Intel MPI

configure

|./configure FC=ifort CC=icc MPIFC=mpif90 MPICC=mpicc --with-mpi --enable-double-precision
|

EXAMPLES/homogeneous_halfspace_HEX8_elastic_absorbing_Stacey_5sides
o run_this_example.sh
o

  Error:

  |forrtl: severe (174): SIGSEGV, segmentation fault occurred
  Image              PC                Routine            Line        Source
  xspecfem3D         00000000006C6C19  Unknown               Unknown  Unknown
  xspecfem3D         00000000006C5590  Unknown               Unknown  Unknown
  xspecfem3D         0000000000677B92  Unknown               Unknown  Unknown
  xspecfem3D         0000000000610243  Unknown               Unknown  Unknown
  xspecfem3D         0000000000614B1B  Unknown               Unknown  Unknown
  libpthread.so.0    000000330F20F710  Unknown               Unknown  Unknown
  libmpi.so.12       00007FA9E9E4591C  Unknown               Unknown  Unknown
  libmpi.so.12       00007FA9E9E2E9AB  Unknown               Unknown  Unknown
  libmpi.so.12       00007FA9E9C7EA91  Unknown               Unknown  Unknown
  libmpi.so.12       00007FA9E9E1E6B3  Unknown               Unknown  Unknown
  libmpi.so.12       00007FA9E9DDBE4D  Unknown               Unknown  Unknown
  libmpi.so.12       00007FA9E9DCD5D1  Unknown               Unknown  Unknown
  libmpifort.so.12   00007FA9EA35A750  Unknown               Unknown  Unknown
  xspecfem3D         0000000000605D9C  Unknown               Unknown  Unknown
  xspecfem3D         000000000055685A  Unknown               Unknown  Unknown
  xspecfem3D         0000000000403E16  Unknown               Unknown  Unknown
  libc.so.6          000000330E61ED5D  Unknown               Unknown  Unknown
  xspecfem3D         0000000000403D09  Unknown               Unknown  Unknown
  |
—
Reply to this email directly or view it on GitHub
#512.

Dimitri Komatitsch
CNRS Research Director (DR CNRS), Laboratory of Mechanics and Acoustics,
UPR 7051, Marseille, France http://komatitsch.free.fr

luet · 2015-05-20T19:46:08Z

I am running with --enable-debug as @QuLogic suggested. It's taking a while. I'll let you know when it's done.

I have heard that they have changed their release numbering at Intel. They don't do x.1 anymore. Instead 14.0 contained the fixes from 13 and 15.0 contains the fix for 14.0.

komatits · 2015-05-20T19:48:40Z

OK, strange then!

With |--enable-debug| you can kill the run once its completes the fifth
time step and creates the timestamp00005 file, no need to let it run
until the end. (that file means no bug was found)

On 05/20/2015 09:46 PM, David Luet wrote:

I am running with |--enable-debug| as @QuLogic
https://github.com/QuLogic suggested. It's taking a while. I'll let
you know when it's done.

I have heard that they have changed their release numbering at Intel.
They don't do x.1 anymore. Instead 14.0 contained the fixes from 13 and
15.0 contains the fix for 14.0.

—
Reply to this email directly or view it on GitHub
#512 (comment).

Dimitri Komatitsch
CNRS Research Director (DR CNRS), Laboratory of Mechanics and Acoustics,
UPR 7051, Marseille, France http://komatitsch.free.fr

luet · 2015-05-21T13:50:40Z

The EXAMPLE:

homogeneous_halfspace_HEX8_elastic_absorbing_Stacey_5sides

with Intel 14.0 + Open MPI 1.6.3 with --enable-debug ran through with no problem. I am guessing the error might come from some compiler flags. Maybe some optimization options.

luet · 2015-05-21T14:10:36Z

It also work is I remove

-DFORCE_VECTORIZATION

from the Makefile.

I tested with Intel 14 + Open MPI, with configure options:

./configure FC=ifort CC=icc MPIFC=mpif90 MPICC=mpicc --with-mpi --enable-double-precison

luet · 2015-05-21T15:19:24Z

It also works with -O2 instead of -O3 while keeping -DFORCE_VECTORIZATION.

-O2 is the default when compiling with GNU compilers but -O3 is the default for Intel compilers. Is there a reason for that?

komatits · 2015-05-21T15:53:47Z

Hi David,

Thanks!

I'll make O3 the default for gfortran in all three codes (2D, 3D, 3D_GLOBE)

I'll make -DFORCE_VECTORIZATION off by default; that option leads to
very fast code but to do that purposely breaks array range checking (it
sees 3D arrays as 1D vectors); in principle it is safe unless the
compiler makes weird / very weird assumptions. Apparently the new
release of that one does for some reason :-(

Dimitri.

On 05/21/2015 05:19 PM, David Luet wrote:

It also works with |-O2| instead of |-O3| while keeping
|-DFORCE_VECTORIZATION|.

|-O2| is the default when compiling with GNU compilers but |-O3| is the
default for Intel compilers. Is there a reason for that?

—
Reply to this email directly or view it on GitHub
#512 (comment).

Dimitri Komatitsch
CNRS Research Director (DR CNRS), Laboratory of Mechanics and Acoustics,
UPR 7051, Marseille, France http://komatitsch.free.fr

komatits · 2015-05-21T16:03:29Z

Done.

Dimitri.

On 05/21/2015 05:53 PM, Dimitri Komatitsch wrote:

Hi David,

Thanks!

I'll make O3 the default for gfortran in all three codes (2D, 3D, 3D_GLOBE)

I'll make -DFORCE_VECTORIZATION off by default; that option leads to
very fast code but to do that purposely breaks array range checking (it
sees 3D arrays as 1D vectors); in principle it is safe unless the
compiler makes weird / very weird assumptions. Apparently the new
release of that one does for some reason :-(

Dimitri.

On 05/21/2015 05:19 PM, David Luet wrote:

It also works with |-O2| instead of |-O3| while keeping
|-DFORCE_VECTORIZATION|.

|-O2| is the default when compiling with GNU compilers but |-O3| is the
default for Intel compilers. Is there a reason for that?

—
Reply to this email directly or view it on GitHub
#512 (comment).

Dimitri Komatitsch
CNRS Research Director (DR CNRS), Laboratory of Mechanics and Acoustics,
UPR 7051, Marseille, France http://komatitsch.free.fr

komatits closed this as completed May 23, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem when compiling with newer versions of Intel compilers (14 and 15) #512

Problem when compiling with newer versions of Intel compilers (14 and 15) #512

luet commented May 20, 2015

QuLogic commented May 20, 2015

komatits commented May 20, 2015

luet commented May 20, 2015

komatits commented May 20, 2015

luet commented May 21, 2015

luet commented May 21, 2015

luet commented May 21, 2015

komatits commented May 21, 2015

komatits commented May 21, 2015

Problem when compiling with newer versions of Intel compilers (14 and 15) #512

Problem when compiling with newer versions of Intel compilers (14 and 15) #512

Comments

luet commented May 20, 2015

Intel 15.0.2 + Intel MPI and Intel 15.0.2 + Open MPI 1.6.3 (same output)

Intel 14 + Openmpi 1.6.3

Intel 14.0 + Intel MPI

QuLogic commented May 20, 2015

komatits commented May 20, 2015

luet commented May 20, 2015

komatits commented May 20, 2015

luet commented May 21, 2015

luet commented May 21, 2015

luet commented May 21, 2015

komatits commented May 21, 2015

komatits commented May 21, 2015