Error "module compiled against ABI version" when using a device on MacBook Pro M2 with MacOS Sonoma 14.4

[trevorstr]

🐛 Describe the bug

• I created a brand new virtual environment and Python script containing the following code.
• I installed the beta / nightly version of Pytorch, using the directions on Apple's website for MPS (M2) support.

pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu

import torch
mps = torch.device("mps")

When I run the code, I get this error:

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0b1 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled against NumPy 2.0.

If you are a user of the module, the easiest solution will be to
either downgrade NumPy or update the failing module (if available).

Traceback (most recent call last):  File "/Users/trevor.sullivan/git/pytorch-audio-classifier/main.py", line 7, in <module>
    mps = torch.device("cpu")
/Users/trevor.sullivan/git/pytorch-audio-classifier/main.py:7: DeprecationWarning: numpy.core._multiarray_umath is deprecated and has been renamed to numpy._core._multiarray_umath. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core._multiarray_umath._ARRAY_API.
  mps = torch.device("cpu")
/Users/trevor.sullivan/git/pytorch-audio-classifier/main.py:7: UserWarning: Failed to initialize NumPy: module compiled against ABI version 0x1000009 but this version of numpy is 0x2000000 (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
  mps = torch.device("cpu")

How can I go about resolving this?

Versions

PyTorch version: 2.4.0.dev20240317
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.4 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.3.9.4)
CMake version: Could not collect
Libc version: N/A

Python version: 3.12.2 (main, Feb  6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)] (64-bit runtime)
Python platform: macOS-14.4-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M2 Pro

Versions of relevant libraries:
[pip3] numpy==2.0.0b1
[pip3] torch==2.4.0.dev20240317
[pip3] torchaudio==2.2.0.dev20240317
[pip3] torchvision==0.18.0.dev20240317
[conda] Could not collect

cc @seemethere @malfet @osalpekar @atalman @mruberry @rgommers @kulinseth @albanD @DenisVieriu97 @razarmehr

[malfet]

[Edit] Indeed PyTorch build against Numpy-2.0 release candidate is backward compatible with Numpy-1.X

Let's test it throughout our config permutations: #122157

[malfet]

@trevorstr as comment message says, your workaround is to uninstall numpy (or install nunmpy 1.x)

[atalman]

@trevorstr We are currently working on enabling numpy 2.x on our CI/CD, tracked here: #107302

[trevorstr]

@trevorstr as comment message says, your workaround is to uninstall numpy (or install nunmpy 1.x)

I'm confused though, because I didn't install numpy 2.x specifically. It must have been automatically installed when I ran the pip command to install torch and torchvision.

Does that mean that the PyTorch package is referencing the wrong version of numpy in its dependencies?

[malfet]

I'm confused though, because I didn't install numpy 2.x specifically. It must have been automatically installed when I ran the pip command to install torch and torchvision.

torchvision depends on the NumPy and does not have a constraint on version to 1.x, only. And although NumPy-2.x is not out yet, as you are installing with --pre flag it will pick it.

[trevorstr]

That makes perfect sense, thank you for helping me understand why that occurred.

I can confirm that uninstalling numpy 2.x, and installing the latest version of 1.x explicitly, the code now works.

(venv) trevor.sullivan@computer pytorch-audio-classifier % pip3 uninstall numpy
Found existing installation: numpy 2.0.0b1
Uninstalling numpy-2.0.0b1:
  Would remove:
    /Users/trevor.sullivan/git/pytorch-audio-classifier/venv/bin/f2py
    /Users/trevor.sullivan/git/pytorch-audio-classifier/venv/bin/numpy-config
    /Users/trevor.sullivan/git/pytorch-audio-classifier/venv/lib/python3.12/site-packages/numpy-2.0.0b1.dist-info/*
    /Users/trevor.sullivan/git/pytorch-audio-classifier/venv/lib/python3.12/site-packages/numpy/*
Proceed (Y/n)? y
  Successfully uninstalled numpy-2.0.0b1
(venv) trevor.sullivan@computer pytorch-audio-classifier % pip3 install numpy==1.26.4
Collecting numpy==1.26.4
  Downloading numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl.metadata (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.1/61.1 kB 514.8 kB/s eta 0:00:00
Downloading numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl (13.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.7/13.7 MB 12.1 MB/s eta 0:00:00
Installing collected packages: numpy
pSuccessfully installed numpy-1.26.4
(venv) trevor.sullivan@computer pytorch-audio-classifier % python3 main.py
(venv) trevor.sullivan@computer pytorch-audio-classifier %

Feel free to close this ticket if you want, or leave it open since you're working on integrating numpy 2.x. Thank you

[malfet]

Actually, not sure there is anything we want to fix this, as neither of those are an errors, but rather a warnings, albeit an annoying ones

[rgommers]

I'll note that the verbose warning is due to pybind11 digging into private internals of numpy (which now mostly works, but may give incorrect behavior for some pybind11 functionality in combination with numpy 2.0), and it'll likely turn into a hard error very soon.

The likely next things to happen are:

• The fix for pybind11 gets merged (API: Make numpy.h compatible with both NumPy 1.x and 2.x pybind/pybind11#5050),
• pybind11 does a new minor release,
• numpy 2.0.0rc1 turns the warning into an error (see Handle warning about private API and pybind11 before 2.0.0rc1 numpy/numpy#26010).

[malfet]

@rgommers but is it safe to assume, that if one to build using current version of PyBind and numpy-2.0.0b1 API, it will be both forward compatible with the upcoming 2.0.0 and backward compatible with 1.x? Or it's better to constrain PyTOrch-2.3 to NumPy-1.x?

[rgommers]

I think not. There is two parts to that:

1. Is building against numpy 2.0.0b1 going to be ABI compatible with 2.0 and 1.x? Yes (we shouldn't be changing ABI anymore, and the official freeze in 2.0.0rc1 is days away)
2. Is using pybind11 2.11.1 going to result in binaries that are runtime-compatible with numpy 2.0? Likely not

For (2), Pybind11 accesses API (e.g., https://github.com/pybind/pybind11/blob/v2.11/include/pybind11/numpy.h#L266) that may disappear or start raising.

Or it's better to constrain PyTOrch-2.3 to NumPy-1.x?

As of right now, yes, it'd be safer to have a runtime dependency numpy<2. However, there is a month left to the 2.3 release date. If a pybind11 2.12.0 release and numpy 2.0.0rc1 will be available within a week or so, it'd be really useful to update pybind11 in the 2.3 release branch and not have an upper bound on numpy.

I understand that there is some risk to doing something like that so late in the release cycle, since pybind11 is an important dependency. But it's only a risk, and if something breaks somewhere even though all CI is green, surely it can be fixed. If we have PyTorch incompatible with the latest NumPy until 2.4.0 in the second half of the year though, that is definitely going to result in annoying problems (e.g., a user doing pip install torch numpy>=2.0 is going to then get latest numpy and a torch downgraded to 2.2.2, because 2.2.2 doesn't have the upper bound on numpy).

[malfet]

2. Is using pybind11 2.11.1 going to result in binaries that are runtime-compatible with numpy 2.0? Likely not

For (2), Pybind11 accesses API (e.g., https://github.com/pybind/pybind11/blob/v2.11/include/pybind11/numpy.h#L266) that may disappear or start raising.

But we are not querying any NumPy APIs thru PyBind, so that shouldn't be a problem.

As of right now, yes, it'd be safer to have a runtime dependency numpy<2

Tricky part is: PyTorch does not depend on NumPy, but say TorchVision does. So we can put a constraint there, but as you've said it just results in us installing older versions of Torch, which is not ideal

[rgommers]

But we are not querying any NumPy APIs thru PyBind, so that shouldn't be a problem.

Ah okay, then it's perhaps safer - but hard to be sure without trying. I'd say that if you see the warning in the issue description (A module that was compiled using NumPy 1.x ...), there is likely to be a problem. It's coming from

pytorch/torch/csrc/utils/tensor_numpy.cpp

Lines 64 to 85 in d2a8d38

	bool is_numpy_available() {
	static bool available = []() {
	if (_import_array() >= 0) {
	return true;
	}
	// Try to get exception message, print warning and return false
	std::string message = "Failed to initialize NumPy";
	// NOLINTNEXTLINE(cppcoreguidelines-init-variables)
	PyObject *type, *value, *traceback;
	PyErr_Fetch(&type, &value, &traceback);
	if (auto str = value ? PyObject_Str(value) : nullptr) {
	if (auto enc_str = PyUnicode_AsEncodedString(str, "utf-8", "strict")) {
	if (auto byte_str = PyBytes_AS_STRING(enc_str)) {
	message += ": " + std::string(byte_str);
	}
	Py_XDECREF(enc_str);
	}
	Py_XDECREF(str);
	}
	PyErr_Clear();
	TORCH_WARN(message);
	return false;

, so the end result is that is_numpy_available returns false, and numpy interop will not work.

[rgommers]

Tricky part is: PyTorch does not depend on NumPy, but say TorchVision does. So we can put a constraint there, but as you've said it just results in us installing older versions of Torch, which is not ideal

Agreed. That won't work well in practice. Maybe we should give it a few days, and see if the needed pybind11 fixes are in by Monday?

[atalman]

@rgommers Nightly change for wheels just landed: pytorch/builder@65f8e7d We can test these tomorrow morning. Is there particular smoke test or anything like this we can run to test if it works correctly ?

[rgommers]

Great! ABI compat should be fine; if not a very basic test like this should catch it:

import torch
import numpy as np  # with numpy 1.26 for example

x = np.arange(5)
t = torch.tensor(x)

test/test_numpy_interop.py is a useful small subset to run, with numpy 1.26 and 2.0.0b1

That's ABI compat; doesn't mean any regular Python API usage is 100% fine (it probably is), but a regular test suite run in an env with the nightly and numpy 2.0.0b1 will turn that up soon enough.

[rgommers]

Maybe we should give it a few days, and see if the needed pybind11 fixes are in by Monday?

Quick update: Pybind11 master is now in good shape, pybind/pybind11#5050 just got merged.

[rgommers]

Pybind11 2.12.0 is out now: https://pypi.org/project/pybind11/2.12.0/. Upgrading to that will fix this issue.

[rgommers]

@trevorstr I'll reopen this, the upgrade actually does need to happen in the pybind11 vendored by PyTorch.

[trevorstr]

Cool, thank you! I appreciate you taking ownership and driving the resolution.