ImportError "No module named numpy.core._multiarray_umath" on PPC64le

[st-pasha]

The error that I'm seeing is this:

error message

$ python
Python 3.6.8 (default, Jun 11 2019, 14:54:15) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "/home/jenkins/tmpenv/lib/python3.6/site-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/home/jenkins/tmpenv/lib/python3.6/site-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/home/jenkins/tmpenv/lib/python3.6/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jenkins/tmpenv/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/home/jenkins/tmpenv/lib/python3.6/site-packages/numpy/core/__init__.py", line 54, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.6 from "/home/jenkins/tmpenv/bin/python",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.18.1" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: No module named 'numpy.core._multiarray_umath'

Since the message says "you can open a new issue ...", I decided why not -- I mean, who doesn't like to deal with obscure installation errors? Some more details about the overall setup:

- how you installed Python
To be honest, I didn't, the python was already pre-installed on the machine. I merely created a virtual environment (virtualenv tmpenv --python=/usr/bin/python3.6 && source tmpenv/bin/activate). The python version is displayed as Python 3.6.8 (default, Jun 11 2019, 14:54:15)

- how you installed numpy

$ pip install numpy -i https://h2oai.github.io/py-repo/
Looking in indexes: https://h2oai.github.io/py-repo/
Collecting numpy
  Downloading http://artifacts.h2o.ai.s3.amazonaws.com/releases/ai/h2o/thirdparty/numpy/numpy-1.18.1-cp36-cp36m-linux_ppc64le.whl (15.4 MB)
     |████████████████████████████████| 15.4 MB 503 kB/s 
Installing collected packages: numpy
Successfully installed numpy-1.18.1

This is a pre-built wheel, which was essentially built using pip wheel numpy inside the official docker image for manylinux2014-powerpc64le. Full details are here: https://github.com/h2oai/py-repo/tree/master/numpy.

- your operating system
The OS is Red Hat Enterprise Linux 7.7 on a PowerPC64le machine.

- whether or not you have multiple versions of Python installed
I'm working in a virtual environment, so it is equivalent to having only one python.

- if you built from source, your compiler versions and ideally a build log
Sure, why not. I've added a -vv flag to the script linked above, and here's the output:
numpy-build-log.txt

---

As it happens, I even know why the error occurs. The following files are installed in the numpy directory:

$ ls tmpenv/lib/python3.6/site-packages/numpy/core/*.so
tmpenv/lib/python3.6/site-packages/numpy/core/_multiarray_tests.cpython-36m-powerpc64le-linux-gnu.so
tmpenv/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-powerpc64le-linux-gnu.so
tmpenv/lib/python3.6/site-packages/numpy/core/_operand_flag_tests.cpython-36m-powerpc64le-linux-gnu.so
tmpenv/lib/python3.6/site-packages/numpy/core/_rational_tests.cpython-36m-powerpc64le-linux-gnu.so
tmpenv/lib/python3.6/site-packages/numpy/core/_struct_ufunc_tests.cpython-36m-powerpc64le-linux-gnu.so
tmpenv/lib/python3.6/site-packages/numpy/core/_umath_tests.cpython-36m-powerpc64le-linux-gnu.so

At the same time, the python says the following:

>>> import sysconfig
>>> sysconfig.get_config_var("EXT_SUFFIX")
'.cpython-36m-ppc64le-linux-gnu.so'

As you can see, there is a mismatch between the extension on the files that are installed, and the extension that the python expects: powerpc64le vs ppc64le. Renaming these (and all other .so files in other directories) so that they have "ppc64le" in their name solves the problem: numpy works. However, this is a one-time solution.

What I really don't understand is whose responsibility it is to fix this problem permanently?

• Is this a faulty python distribution (though upgrading python on a server might be hard)?
• Should it be pip's responsibility to rename the files during install?
• The numpy itself could notice the mismatch in tags and do something about it?
• Wheel created incorrectly? (however, I tried on a different PPC64 machine, and the same wheel works perfectly fine there. That machine has python 3.6.5)

[charris]

Thanks for taking the time to track down the problem. I think this is probably a problem with the python in the docker image, which is distributed by https://github.com/pypa/manylinux, so you might ask there. Maybe it is configurable: @tylerjereddy, @mattip, @matthew-brett Thoughts?

[mattip]

If you enter the ppc64le manylinux docker image, what does the python there report for sysconfig.get_config_var("EXT_SUFFIX")?

[st-pasha]

There are 4 different pythons there, they each report

'.cpython-35m-powerpc64le-linux-gnu.so'
'.cpython-36m-powerpc64le-linux-gnu.so'
'.cpython-37m-powerpc64le-linux-gnu.so'
'.cpython-38-powerpc64le-linux-gnu.so'

[charris]

So the docker image is the immediate source of the discrepancy. The EXT_SUFFIX seems to be set by the configure run prior to building Python. AFAICT with a quick look, it uses powerpc64* form. Did you get your python directly from Redhat?

Might be interesting what happens if you download a Python source and build it on your machine, or at least run ./configure and look at the config.log.

[frenzymadness]

I have the same problem in my project and I've found this issue. I think that it's a bug in manylinux container images so I've filed an issue: pypa/manylinux#687

[mattip]

It is not a bug in the manylinux container, rather someone is messing with the cpython sources (which hardcode this value) or providing a command line option to override this value when compiling CPython. Please refer back to your CPython provider, as suggested in the comment above.

[frenzymadness]

You are right. It seemed strange to me because manylinux images are based on Centos and my problem occurred on RHEL. The truth is that RHEL/Centos/Fedora have a patch to change powerpc64le → ppc64le but manylinux compiles its own Pythons and uses the default setting.

[mattip]

Please open an issue there.

[frenzymadness]

I don't think this is something we can easily change there. IMHO Python there just respects what the whole distributions have. But we can make importlib in Python or manylinux wheels more universal. See my last comment.

[mattip]

Let's keep the conversation on pypa/manylinux since this is not a numpy-specific problem.

[seberg]

Not sure about the status, but it seems like some provider of the cpythong executable is using the wrong extension for things, and not something that NumPy can do about. The pypa issue probably more relevant anyway.