Incorrect arch detection

[quellyn]

Hi guys,

On our local Frankencluster, I've noticed an odd inconsistency with Spack's arch detection. This cluster is composed of many flavors of x86_64, Power 9, and ARM nodes, all running CentOS Linux release 7.7.1908 (Core). My particular issue is with our x86_64 nodes.

Example 1: On node cn123, with a fresh Spack instance:

[quellyn@cn123 cn123]$ git clone https://github.com/spack/spack.git
Cloning into 'spack'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 197245 (delta 6), reused 5 (delta 5), pack-reused 197236
Receiving objects: 100% (197245/197245), 72.96 MiB | 31.81 MiB/s, done.
Resolving deltas: 100% (87768/87768), done.
Checking out files: 100% (5637/5637), done.
[quellyn@cn123 cn123]$ source ./spack/share/spack/setup-env.sh 
[quellyn@cn123 cn123]$ echo $SPACK_ROOT
/home/quellyn/Scratch/cn123/spack
[quellyn@cn123 cn123]$ spack arch
linux-centos7-haswell

The node itself agrees with this assessment:

[quellyn@cn123 cn123]$ cat /sys/devices/cpu/caps/pmu_name 
haswell

[quellyn@cn123 cn123]$ grep Intel /proc/cpuinfo | sort -u
model name	: Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
vendor_id	: GenuineIntel

Example 2: On node cn141, with a fresh Spack instance:

[quellyn@cn141 cn141]$ git clone https://github.com/spack/spack.git
Cloning into 'spack'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 197245 (delta 6), reused 5 (delta 5), pack-reused 197236
Receiving objects: 100% (197245/197245), 72.96 MiB | 35.81 MiB/s, done.
Resolving deltas: 100% (87768/87768), done.
Checking out files: 100% (5637/5637), done.
[quellyn@cn141 cn141]$ source ./spack/share/spack/setup-env.sh 
[quellyn@cn141 cn141]$ echo $SPACK_ROOT
/home/quellyn/Scratch/cn141/spack
[quellyn@cn141 cn141]$ spack arch
linux-centos7-nehalem

But this node disagrees with Spack; it thinks it's a Haswell also:

[quellyn@cn141 cn141]$ cat /sys/devices/cpu/caps/pmu_name
haswell
[quellyn@cn141 cn141]$                 
[quellyn@cn141 cn141]$ grep Intel /proc/cpuinfo | sort -u
model name	: Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
vendor_id	: GenuineIntel

I'm afraid I don't understand the magic of Spack's arch detection well enough to even start looking for a root cause. If you could give me a hint as to where to start that would be great.

Thanks!
Quellyn

P.S. This is my first time opening an issue; please let me know if I've left something out.

[alalazo]

Can you post the "flags" of both nodes in /proc/cpuinfo and see if they differ in any way?

[quellyn]

I lost my allocation on my "correct" node (cn123), but I was able to grab another node (cn126) with the same processor (which Spack identifies correctly as a Haswell).

On cn126 (correct arch detection):

[quellyn@cn126 cn126]$ cat /proc/cpuinfo | grep flags | sort -u
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts

On cn141 (incorrect arch detection):

[quellyn@cn141 ~]$ cat /proc/cpuinfo | grep flags | sort -u
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts

Looks like cn126 has the flag "aes", while cn141 does not.

[alalazo]

Our detection is mainly targeted towards binary compatibility and as such we started from the instruction set mentioned in GCC manual. For haswell there's mention of AES:

haswell
               Intel Haswell CPU with 64-bit extensions, MOVBE, MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT,
               AVX, AVX2, AES, PCLMUL, FSGSBASE, RDRND, FMA, BMI, BMI2 and F16C instruction set support.

as there is for ivybridge, sandy-bridge and westmere. nehalem is the first architecture in the hierarchy that does not have support for that. I would say that detection is working correctly for Spack, as it ensures that the binaries that are generated would run on the node.

Now, it would be interesting to understand why there's no aes on that node... Do they have all the same kernel and OS?

[quellyn]

Same OS ``` [quellyn@cn126 Scratch]$ cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) [quellyn@cn141 ~]$ cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) ``` Same kernel, too: ``` [quellyn@cn126 Scratch]$ uname -r 3.10.0-1062.9.1.el7.x86_64 [quellyn@cn141 ~]$ uname -r 3.10.0-1062.9.1.el7.x86_64 ``` I'll ping the admins and see if there are any differences in BIOS firmware version or settings for cn141. Thanks! Q

…

On 2/21/20 10:53 AM, Massimiliano Culpo wrote: Our detection is mainly targeted towards binary compatibility and as such we started from the instruction set mentioned in GCC manual. For |haswell| there's mention of |AES|: |haswell Intel Haswell CPU with 64-bit extensions, MOVBE, MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, AVX, AVX2, AES, PCLMUL, FSGSBASE, RDRND, FMA, BMI, BMI2 and F16C instruction set support. | as there is for |ivybridge|, |sandy-bridge| and |westmere|. |nehalem| is the first architecture in the hierarchy that does not have support for that. I would say that detection is working correctly for Spack, as it ensures that the binaries that are generated would run on the node. Now, it would be interesting to understand why there's no |aes| on that node... Do they have all the same kernel and OS? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#15151?email_source=notifications&email_token=AH6ATNKDU6P5M4RHQYMJHYTREAIJTA5CNFSM4KZGUNG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTQ7KA#issuecomment-589762472>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH6ATNOEMUA7CS27OJYTWGDREAIJTANCNFSM4KZGUNGQ>.

-- Quellyn L Snead XCP-1, Lagrangian Codes Los Alamos National Laboratory Office: 505-667-4185

[quellyn]

After talking to Massimiliano I'm pretty convinced this is NOT a Spack problem. I've put in a ticket with our cluster admins to see if there are BIOS differences between the Haswell nodes.

Thanks so much for the help in troubleshooting!
Q

[tgamblin]

@boegel: FYI -- another reason not to use pmu_name for binary compatibility!

[boegel]

@tgamblin I just brought up pmu_name because it looked interesting at first sight, but it's clear it not correct enough for our purpose...