_dl.*.so is an x86_64 binary in an arm64 pytorch installation

[victor-shepardson]

🐛 Describe the bug

This appears to be an issue with pytorch which surfaces when trying to compile pytorch programs using nuitka (see Nuitka/Nuitka#1774) on macOS 12.4 with arm64 hardware.

I have a perfectly functional arm64 pytorch install, but it contains an x86 binary called _dl.cpython-310-darwin.so. nuitka chokes on this. it is possible to trigger an error directly with python -c "import torch._dl", though this has not been a problem for normal use of pytorch nor for linking C++ programs against the same pytorch install.

using python, pytorch and nuitka installed into a conda environment using mamba:

mamba create -n nuitka-test -c pytorch nuitka pytorch

  + bzip2                  1.0.8  h3422bc3_4          conda-forge/osx-arm64     Cached
  + ca-certificates    2022.6.15  h4653dfc_0          conda-forge/osx-arm64     Cached
  + libcxx                14.0.6  h2692d47_0          conda-forge/osx-arm64        1MB
  + libffi                 3.4.2  h3422bc3_5          conda-forge/osx-arm64     Cached
  + libpython-static      3.10.6  h2e04ded_0_cpython  conda-forge/osx-arm64        8MB
  + libsqlite             3.39.2  h2c9beb0_1          conda-forge/osx-arm64     Cached
  + libzlib               1.2.12  ha287fd2_2          conda-forge/osx-arm64     Cached
  + ncurses                  6.3  h07bb92c_1          conda-forge/osx-arm64     Cached
  + nuitka                 1.0.6  pyhd8ed1ab_0        conda-forge/noarch        Cached
  + openssl                3.0.5  h7aea29f_1          conda-forge/osx-arm64     Cached
  + pip                   22.2.2  pyhd8ed1ab_0        conda-forge/noarch        Cached
  + python                3.10.6  hae75cb6_0_cpython  conda-forge/osx-arm64       13MB
  + pytorch               1.12.1  py3.10_0            pytorch/osx-arm64         Cached
  + readline               8.1.2  h46ed386_0          conda-forge/osx-arm64     Cached
  + setuptools            65.3.0  pyhd8ed1ab_1        conda-forge/noarch         801kB
  + tk                    8.6.12  he1e0b03_0          conda-forge/osx-arm64     Cached
  + typing_extensions      4.3.0  pyha770c72_0        conda-forge/noarch        Cached
  + tzdata                 2022c  h191b570_0          conda-forge/noarch        Cached
  + wheel                 0.37.1  pyhd8ed1ab_0        conda-forge/noarch        Cached
  + xz                     5.2.6  h57fd34a_0          conda-forge/osx-arm64     Cached

> file ~/mambaforge/envs/nuitka-test/lib/python3.10/site-packages/torch/_dl.cpython-310-darwin.so
/Users/victor/mambaforge/envs/nuitka-test/lib/python3.10/site-packages/torch/_dl.cpython-310-darwin.so: Mach-O 64-bit bundle x86_64
> file ~/mambaforge/envs/nuitka-test/lib/python3.10/site-packages/torch/_C.cpython-310-darwin.so
/Users/victor/mambaforge/envs/nuitka-test/lib/python3.10/site-packages/torch/_C.cpython-310-darwin.so: Mach-O 64-bit bundle arm64

what is _dl.cpython-310-darwin.so and what is it doing there?

Versions

Collecting environment information...
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.4 (arm64)
GCC version: Could not collect
Clang version: 13.0.1
CMake version: version 3.23.2
Libc version: N/A

Python version: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:38:29) [Clang 13.0.1 ] (64-bit runtime)
Python platform: macOS-12.4-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] torch==1.12.1
[conda] pytorch 1.12.1 py3.10_0 pytorch

cc @ezyang @gchanan @zou3519 @malfet @seemethere

[malfet]

Thank you very much for reporting. Grabbing for myself. It affects both wheel and conda packaging.
Oneliner would fix the packaging, but I wonder if it'll be better to just delete this logic altogether, as all python3 os modules should have RTLD_GLOBAL

[malfet]

Please note, that its very unlikely that we are going to publish 1.12.2 (or update 1.12.1 for osx-arm64) to fix this issue, but the fix should be available in the nightly builds.