cpu_count_physical() always returns None on arm devices

[mtreinish]

Similar to what was reported in #200 about arm systems not having the same /proc/cpuinfo format as x86 systems (and likely other arches too) and cpu_count_physical() for linux systems with arm will always return None.

We should either teach cpu_count_physical() about the different cpuinfo format, or in the alternative use /proc/stat like what is done in cpu_count_logical(). Arm doesn't have logical cores, so the parsing can be the same as cpu_count_logical() for an arm cpu.

[giampaolo]

Can you paste the output of cat /proc/cpuinfo on ARM?

[mtreinish]

Sure, on the aarch64 system I'm running the /proc/cpuinfo contents:

Processor	: AArch64 Processor rev 4 (aarch64)
processor	: 0
model name	: AArch64 Processor rev 4 (aarch64)
BogoMIPS	: 26.00
BogoMIPS	: 26.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 1
model name	: AArch64 Processor rev 4 (aarch64)
BogoMIPS	: 26.00
BogoMIPS	: 26.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 2
model name	: AArch64 Processor rev 4 (aarch64)
BogoMIPS	: 26.00
BogoMIPS	: 26.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 3
model name	: AArch64 Processor rev 4 (aarch64)
BogoMIPS	: 26.00
BogoMIPS	: 26.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

Hardware	: MT6797X

[giampaolo]

...so the right number to return in this case is supposed to be 4?

[mtreinish]

Yes I believe so, there were 4 active physical cores on that device when I read /proc/cpuinfo. (The CPU actually has more cores, but switches between them dynamically based on load and battery)

[giampaolo]

Can you paste the output of python -c "print(repr(open('/proc/cpuinfo').read()))"?

[mtreinish]

Sure, no problem:

'Processor\t: AArch64 Processor rev 4 (aarch64)\nprocessor\t: 4\nmodel name\t: AArch64 Processor rev 4 (aarch64)\nBogoMIPS\t: 26.00\nBogoMIPS\t: 26.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nprocessor\t: 5\nmodel name\t: AArch64 Processor rev 4 (aarch64)\nBogoMIPS\t: 26.00\nBogoMIPS\t: 26.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nHardware\t: MT6797X\n'

[fungi]

I've confirmed this is the case on an AArch64/arm64 Linux 4.15 (not Android) virtual machine to which I have access as well. The contents of /proc/cpuinfo indicate 8 physical processors and psutil.cpu_count() returns 8 but psutil.cpu_count(logical=False) returns None.

[giampaolo]

So, this is a patch which relies on lines such as processor: 1 and assume they are physical cores. I'm gonna put it here and do not commit this yet because I am not 100% convinced this is the right approach.

diff --git a/psutil/_pslinux.py b/psutil/_pslinux.py
index ecc4c703..faf954cb 100644
--- a/psutil/_pslinux.py
+++ b/psutil/_pslinux.py
@@ -642,8 +642,26 @@ def cpu_count_physical():
                     key, value = line.split(b'\t:', 1)
                     current_info[key] = int(value)
 
+    found = sum(mapping.values())
+    if not found:
+        # Perhaps an ARM system, see:
+        # https://github.com/giampaolo/psutil/issues/1359
+        found = set()
+        with open_binary('%s/cpuinfo' % get_procfs_path()) as f:
+            for line in f:
+                line = line.strip().lower()
+                if line.startswith(b'processor'):
+                    key, value = line.split(b'\t:', 1)
+                    try:
+                        value = int(value)
+                    except ValueError:
+                        pass
+                    else:
+                        found.add(value)
+        found = len(found)
+
     # mimic os.cpu_count()
-    return sum(mapping.values()) or None
+    return found or None
 
 
 def cpu_stats():
diff --git a/psutil/tests/test_linux.py b/psutil/tests/test_linux.py
index 4b72f725..042d1a7d 100755
--- a/psutil/tests/test_linux.py
+++ b/psutil/tests/test_linux.py
@@ -679,6 +679,44 @@ class TestSystemCPU(unittest.TestCase):
             self.assertIsNone(psutil._pslinux.cpu_count_physical())
             assert m.called
 
+    def test_cpu_count_physical_arm_mocked(self):
+        # See: https://github.com/giampaolo/psutil/issues/1359
+        def open_mock(name, *args, **kwargs):
+            if name == '/proc/cpuinfo':
+                return io.BytesIO(textwrap.dedent("""\
+                    Processor\t: AArch64 Processor rev 4 (aarch64)
+                    processor\t: 4
+                    model name\t: AArch64 Processor rev 4 (aarch64)
+                    BogoMIPS\t: 26.00
+                    BogoMIPS\t: 26.00
+                    Features\t: fp asimd evtstrm aes pmull sha1 sha2 crc32
+                    CPU implementer\t: 0x41
+                    CPU architecture: 8
+                    CPU variant\t: 0x0
+                    CPU part\t: 0xd03
+                    CPU revision\t: 4
+
+                    processor\t: 5
+                    model name\t: AArch64 Processor rev 4 (aarch64)
+                    BogoMIPS\t: 26.00
+                    BogoMIPS\t: 26.00
+                    Features\t: fp asimd evtstrm aes pmull sha1 sha2 crc32
+                    CPU implementer\t: 0x41
+                    CPU architecture: 8
+                    CPU variant\t: 0x0
+                    CPU part\t: 0xd03
+                    CPU revision\t: 4
+                    """).encode())
+            else:
+                return orig_open(name, *args, **kwargs)
+
+        orig_open = open
+        patch_point = 'builtins.open' if PY3 else '__builtin__.open'
+        with mock.patch(patch_point, create=True, side_effect=open_mock) as m:
+            ret = psutil.cpu_count(logical=False)
+            assert m.called
+            self.assertEqual(ret, 2)
+
     @unittest.skipIf(not HAS_CPU_FREQ, "not supported")
     def test_cpu_freq_no_result(self):
         with mock.patch("psutil._pslinux.glob.glob", return_value=[]):

[giampaolo]

What's the output of lscpu?

[fungi]

This is from an 8 processor Linux AArch64/arm64 virtual machine:

Architecture: aarch64
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
NUMA node0 CPU(s): 0-7

[giampaolo]

That is weird. I don't see how it could determine "8" given the content of your /proc/cpuinfo file. Maybe that means lscpu relies on something else other than /proc/cpuinfo? Could you please strace it, put the output in a txt file and attach it here?

strace lscpu > out.txt

[fungi]

I suspect you also want 2>&1 but it's a few hundred lines of text. Clearly indicates it's reading from lots of things under the /sys/devices/system/cpu/ tree. Do you really want the full strace posted into an issue comment?

[mtreinish]

@giampaolo the machine @fungi is using is a different arm system than mine. I expect if I ran lscpu on mine it would return 4 cpus. I just don't have that box handy right now. When I do I'll paste the contents of lscpu from it too.

[giampaolo]

I suspect you also want 2>&1 but it's a few hundred lines of text.

Correct. Thanks.

Do you really want the full strace posted into an issue comment?

I think github allows you to attach it as a text file.

[fungi]

strace-lscpu.txt

[mtreinish]

@giampaolo I just ran lscpu on my system and got:

Architecture:         aarch64
Byte Order:           Little Endian
CPU(s):               10
On-line CPU(s) list:  0-3
Off-line CPU(s) list: 4-9
Thread(s) per core:   0
Core(s) per socket:   2
Socket(s):            2
Vendor ID:            ARM
Model:                4
Model name:           Cortex-A53
Stepping:             r0p4
CPU max MHz:          2002.0000
CPU min MHz:          221.0000
BogoMIPS:             26.00
Flags:                fp asimd evtstrm aes pmull sha1 sha2 crc32

[mtreinish]

I also just spun up a raspberry pi 3 b+ running a current aarch64 kernel (v4.19.1) and ran the commands on that too for comparison:

• /proc/cpuinfo contents:

processor	: 0
BogoMIPS	: 38.40
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 1
BogoMIPS	: 38.40
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 2
BogoMIPS	: 38.40
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 3
BogoMIPS	: 38.40
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

• python -c "print(repr(open('/proc/cpuinfo').read()))":

'processor\t: 0\nBogoMIPS\t: 38.40\nFeatures\t: fp asimd evtstrm crc32 cpuid\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nprocessor\t: 1\nBogoMIPS\t: 38.40\nFeatures\t: fp asimd evtstrm crc32 cpuid\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nprocessor\t: 2\nBogoMIPS\t: 38.40\nFeatures\t: fp asimd evtstrm crc32 cpuid\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nprocessor\t: 3\nBogoMIPS\t: 38.40\nFeatures\t: fp asimd evtstrm crc32 cpuid\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\n'

• lscpu:

Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
Vendor ID:           ARM
Model:               4
Model name:          Cortex-A53
Stepping:            r0p4
BogoMIPS:            38.40
Flags:               fp asimd evtstrm crc32 cpuid

[ddelange]

FYI, I cannot reproduce this error on the Amazon Graviton2 processor and a psutil aarch64 wheel built from source:

Python 3.8.10 (default, Mar 15 2022, 12:22:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from packaging.markers import default_environment
>>> default_environment()
{'implementation_name': 'cpython', 'implementation_version': '3.8.10', 'os_name': 'posix', 'platform_machine': 'aarch64', 'platform_release': '5.13.0-1017-aws', 'platform_system': 'Linux', 'platform_version': '#19~20.04.1-Ubuntu SMP Mon Mar 7 12:55:31 UTC 2022', 'python_full_version': '3.8.10', 'platform_python_implementation': 'CPython', 'python_version': '3.8', 'sys_platform': 'linux'}
>>> import psutil
>>> (psutil.cpu_count(logical=False), psutil.cpu_count(logical=True))
(4, 4)

CPU info

# cat /proc/cpuinfo
processor       : 0
BogoMIPS        : 243.75
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x3
CPU part        : 0xd0c
CPU revision    : 1

processor       : 1
BogoMIPS        : 243.75
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x3
CPU part        : 0xd0c
CPU revision    : 1

processor       : 2
BogoMIPS        : 243.75
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x3
CPU part        : 0xd0c
CPU revision    : 1

processor       : 3
BogoMIPS        : 243.75
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x3
CPU part        : 0xd0c
CPU revision    : 1

# lscpu
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       ARM
Model:                           1
Model name:                      Neoverse-N1
Stepping:                        r3p1
BogoMIPS:                        243.75
L1d cache:                       256 KiB
L1i cache:                       256 KiB
L2 cache:                        4 MiB
L3 cache:                        32 MiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; CSV2, BHB
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs