platform.processor() and platform.machine() report inconsistent values under Rosetta on M1 macOS

[rossng]

Bug report

When I run the Python interpreter in x86_64 mode on an M1 MacBook, I get an unexpected result from platform.machine().

In arm64 mode:

ross@Rosss-MBP ~ % arch -arm64 python   
Python 3.10.6 (main, Sep 21 2022, 15:43:11) [Clang 14.0.0 (clang-1400.0.29.102)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.processor(), platform.machine()
('arm', 'arm64')

In x86_64 mode:

ross@Rosss-MBP ~ % arch -x86_64 python
Python 3.10.6 (main, Sep 21 2022, 15:43:11) [Clang 14.0.0 (clang-1400.0.29.102)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.processor(), platform.machine()
('i386', 'arm64')

Not that the processor is correct, but the machine is not.

Using an x86_64 cling binary gives the expected result:

ross@Rosss-MBP ~ % file /usr/local/Cellar/cling/0.9/bin/cling
/usr/local/Cellar/cling/0.9/bin/cling: Mach-O 64-bit executable x86_64
ross@Rosss-MBP ~ % /usr/local/Cellar/cling/0.9/bin/cling 

[cling]$ #include <sys/utsname.h>
[cling]$ struct utsname u;
[cling]$ uname(&u);
[cling]$ u.machine
(char [256]) "x86_64\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"

As does calling out to the uname utility as a subprocess:

ross@Rosss-MBP ~ % arch -x86_64 python
Python 3.10.6 (main, Sep 21 2022, 15:43:11) [Clang 14.0.0 (clang-1400.0.29.102)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output(['uname', '-p'], stderr=subprocess.DEVNULL, text=True,)
'i386\n'
>>> subprocess.check_output(['uname', '-m'], stderr=subprocess.DEVNULL, text=True,)
'x86_64\n'

It looks like the implementation of platform.process() works the same way as this - by calling out to a subprocess - whereas platform.machine() uses the uname syscall directly.

Not sure what causes this - perhaps it's a fundamental problem for Universal Binaries? In any case, it might be a good idea to patch for this case (since the output makes no sense at the moment).

Edit: not a universal binary issue - see below.

Your environment

• CPython versions tested on: 3.10.6, installed through pyenv
• Operating system and architecture: macOS 12.6, M1 MacBook Pro

[rossng]

A quick experiment shows that this does not appear to be inherent to universal binaries:

main.c:

#include <stdio.h>
#include <sys/utsname.h>

int main(int argc, const char * argv[]) {
    struct utsname u;
    uname(&u);
    printf("uname machine: %s\n", u.machine);
    return 0;
}

ross@Rosss-MBP bin % file uname-tester          
uname-tester: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64
- Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64
- Mach-O 64-bit executable arm64]
uname-tester (for architecture x86_64):	Mach-O 64-bit executable x86_64
uname-tester (for architecture arm64):	Mach-O 64-bit executable arm64
ross@Rosss-MBP bin % arch -arm64 ./uname-tester
uname machine: arm64
ross@Rosss-MBP bin % arch -x86_64 ./uname-tester
uname machine: x86_64

[rossng]

Aha - it seems that pyenv did not in fact install a universal Python binary, which would explain this behaviour.

I do still think the output is quite confusing, so maybe this is a case where it would be better to clarify the documentation.

[ronaldoussoren]

The documentation is pretty vague and doesn't consider the case where the program runs under emulation (such as rosetta on macOS, but likewise for binfmt* on Linux). Worse, it is far from clear what the specification should be: I have use cases for both behaviours (sometimes I need to know the real CPU, sometimes I need to know whatever CPU-type is used by the current proces) and sometimes I need to know which CPU-types are supported by the interpreter which is functionality that's not exposed in platform at all.

[kwsp]

Interesting... on my M1 mac with Python 3.8.9 (from /usr/bin/python3) that I assume is shipped as part of macOS Monterey, I get the expected (?) output:

~ /usr/bin/python3
Python 3.8.9 (default, Apr 13 2022, 08:48:06)
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.processor(), platform.machine()
('arm', 'arm64')

~ arch -x86_64 /usr/bin/python3
Python 3.8.9 (default, Apr 13 2022, 08:48:07)
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.processor(), platform.machine()
('i386', 'x86_64')

So this is possibly a regression. I can do a bisect when I get some time.

I agree with @ronaldoussoren that the documentation needs an update to clarify cases when the program runs under emulation. Possibly with the fix of this bug.

[kwsp]

@rossng if pyenv didn't install a universal binary, how did the binary even execute under x86_64? When I try this on a arm64 binary, I get

~ file $(which python3)
/opt/homebrew/bin/python3: Mach-O 64-bit executable arm64
~ arch -x86_64 python3
arch: posix_spawnp: python3: Bad CPU type in executable

With python 3.10.6 installed from pyenv,

~ arch -x86_64 .pyenv/versions/3.10.6/bin/python3
arch: posix_spawnp: .pyenv/versions/3.10.6/bin/python3: Bad CPU type in executable

I also built a universal binary from the current main branch with

./configure --with-pydebug --enable-universalsdk=/ --with-universal-archs=universal2
make -s -j

and cannot reproduce the behaviour:

~/code/cpython (main ✗) arch -x86_64 ./python.exe
Python 3.12.0a0 (heads/main:c051d55ddb, Oct 18 2022, 14:12:47) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.processor(), platform.machine()
('i386', 'x86_64')

[rossng]

I think the problem may stem from my default python3 executable being the one from miniconda, which appears to be an x86-64 binary. Seems like I thought I was using the pyenv-installed Python binary, but Conda actually hijacked my PATH with its own (higher-priority) entry.

❯ python3
Python 3.9.12 (main, Apr  5 2022, 01:53:17) 
[Clang 12.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.machine()
'x86_64'

❯ which python3
/usr/local/Caskroom/miniconda/base/bin/python3

❯ objdump -a /usr/local/Caskroom/miniconda/base/bin/python3
/usr/local/Caskroom/miniconda/base/bin/python3:	file format mach-o 64-bit x86-64

I just ran into this issue again when trying to compile Blender - also reported before at https://developer.blender.org/T78710#1160643