muelu: KNL build via spack (xsdk) causes "icpc: error #10106" i.e "terminated by kill signal"

[balay]

I'm attemting an xsdk build via spack - which builds trilinos.

It appears muelu is recently enabled in a default spack install of trilinos.

This build breaks within muelu - this compiler errors. This build is on KNL - with intel compilers version 18.0.2 with CFLAGS="-O3 -fp-model fast -xMIC-AVX512"

Attaching the error log.

cc: @jwillenbring @keitat - For now - i'll disable muelu in xsdk@develop build of trilinos

To reproduce:

• use a KNL linux box [centos-7]
• install intel 18.0.2 version of compilers
• get balay/xsdk-develop branch of spack
• setup spack with attached packages.yaml and compilers.yaml
• run spack install xsdk@develop

spack-build.out.txt
packages.yaml.txt
compilers.yaml.txt

[mhoemmen]

icpc: error #10106: Fatal error in /home/intel/18u2/compilers_and_libraries_2018.2.199/linux/bin/intel64/mcpcom, terminated by kill signal

That may mean that your compiler ran out of memory. Try running with less build parallelism (e.g., smaller N in make -j N). I also really wish Spack could set Tpetra_INST_INT_INT=OFF, but there are outstanding issues. See e.g., #3194 .

[balay]

Hm - this machine has 100G+ memory - so out of memory would be wierd. For sure its some Intel compiler issue.

I reran with '-j 64' - this time the build went through without errors [the deafult is probably '-j 272']

For now I'll stick with '-j 64' and see if the error comes up again

[previously I've encountered this error multiple times - before filing this report]

[mhoemmen]

@balay Out of memory would not be weird for -j 272. Each file may take 2 GB on its own. Multiply by 272, and that could easily exceed the 100GB or so memory you have.

[jhux2]

Since using fewer build processes appears to fix the reported problem, I am closing this. @mhoemmen Thank you for the help.