Cannot import numpy in Python 3.7 or Python 3.8 - Ubuntu 18.04

[jrisi256]

Hello! Hope all the devs are staying safe during corona time and thanks for making such a great package.

I manage a group of users so I'm trying to figure out how I can install numpy for all of them to use without having everyone install the package themselves.

I have Python 2.7.17 and Python 3.6.9 (installed by default through Ubuntu 18.04). Additionally, I have installed Python 3.7.5, Python 3.8.0, and numpy using the following.

sudo apt install python3.7
sudo apt install python3.8
sudo apt install python3-numpy

If I run python or python3 or python3.6 which (invokes python 2.7.17 or python 3.6.9), I can run the following command just fine.

import numpy

However, if I try to run the above after running python3.7 or python3.8 I get the following error message:

ImportError: cannot import name 'multiarray' from 'numpy.core' (/usr/lib/python3/dist-packages/numpy/core/__init__.py)

When I install numpy it gets installed to /usr/lib/python3/dist-packages which should be fine since it's on the sys.path() for all versions of Python.

Could it not be working because the version of numpy being pulled using apt is only for Python 3.6.9? If that's the case, how should I install numpy for all users for ALL versions of Python?

[rossbar]

I can't say how you could solve this problem with the system package manager (apt), but making use of Python's venv module (or any of the tools that make it more user-friendly, such as the virtualenv package) might be useful and generally offers greater explicit control when managing multiple Python versions.

[jrisi256]

It was my understanding that venv and virtualenv are designed for individuals or a team of individuals working on a specific project, and they weren't really designed for managing packages for all users of a system. If I'm mistaken can you point to some documentation that points to specifically how to do that?

[rossbar]

In the case of having multiple Python versions installed, virtualenv is just one way to help map specific environments to the appropriate Python installation. Your use-case is rather specific, but you could:

1. Create a folder for storing the virtual environments somewhere on the system that is accessible by your users (with appropriate r-x permissions)
2. Create the environments you want, e.g. with virtualenv and virtualenvwrapper:

   mkvirtualenv -p `which python3.X` py3X
   workon py3X
   pip install <whatever packages, e.g. numpy==1.18.2>

   Where 'X' should be replaced with the specific minor version
3. Then your users could use the appropriate Python version by activating the appropriate environment (again, with virtualenvwrapper): workon py3X

There are many ways to solve this problem. My suggestion is not necessarily the best and may not work for your specific case. I'd recommend PyPA's docs on virtualenv to see if this fits your needs.

Ultimately this question has more to do with general Linux system administration than NumPy installation - I would also recommend the Arch wiki as an excellent resource in that regard.

[rgommers]

When I install numpy it gets installed to /usr/lib/python3/dist-packages which should be fine since it's on the sys.path() for all versions of Python.

That's definitely not fine, NumPy contains C code and compiled Python extensions are specific to a minor Python version - you cannot mix 3.6 / 3.7 / 3.8

Could it not be working because the version of numpy being pulled using apt is only for Python 3.6.9?

Correct

If that's the case, how should I install numpy for all users for ALL versions of Python?

You need to install a Python version per Python version. Probably as simple as:

sudo apt install python3.7
sudo python3.7 -m pip install numpy
sudo apt install python3.8
sudo python3.8 -m pip install numpy

and actually probably better to get rid of your apt-installed python3-numpy and do sudo python3.6 -m pip install numpy for symmetry

[rgommers]

It was my understanding that venv and virtualenv are designed for individuals or a team of individuals working on a specific project, and they weren't really designed for managing packages for all users of a system.

I think that is correct.

[jrisi256]

sudo apt install python3.7
sudo python3.7 -m pip install numpy
sudo apt install python3.8
sudo python3.8 -m pip install numpy

I always thought it was STRONGLY discouraged to use sudo and pip together.

[jrisi256]

@rossbar Quick question. If it's too off-topic, I understand. Is it best practice to use venv or virtualenv. I see in official Python documentation they recommend venv but in practice I see most people use virtualenv so I'm a little confused.

[charris]

I always thought it was STRONGLY discouraged to use sudo and pip together.

I sudo install distro numpy for the library dependencies, then

$ python3 -m pip install numpy --user

That way the system numpy is left alone. In a virtual environment you don't need the --user.

PS: I want the system libraries because I always build from the repo.

[jrisi256]

$ python3 -m pip install numpy --user

@charris Unfortunately this will only install numpy for me. I need to install it for all users.

[rossbar]

Is it best practice to use venv or virtualenv. I see in official Python documentation they recommend venv but in practice I see most people use virtualenv so I'm a little confused.

I'm not really an expert here - from my perspective, venv is nice because it is built into Python so there's no need to install extra dependencies; OTOH virtualenv (especially when coupled with virtualenvwrapper) works really well and is perhaps slightly more customizable. I personally use both approaches for different scenarios.

PEP 405 is a nice read on this topic if you're interested in the design/motivations for venv and how it relates to virtualenv.

[mattip]

Another option is to use conda to install multiple environments.

[matthew-brett]

For venv vs virtualenv - since a recent comprehensive refactor, virtualenv can now use venv under the. hood - see : https://virtualenv.pypa.io/en/stable/user_guide.html#creators

[jrisi256]

Well thank you all for the guidance all on venv vs. virtualenv. As well as for clearing up why I was having an issue. I realize this isn't a numpy problem so i'll have to ask elsewhere and try out different tools to fix it.

One final question. And if it's too, too off-topic that's fine. I see in many places people saying to never use sudo pip install as it introduces a lot of security risks. However, I see a lot of other people (as well as in documentation for installing some things) using sudo pip install. So I'm a little confused as to its status.

[charris]

@jrisi256 I think the recommendation is to not use sudo unless you know what you are doing. It isn't terrible in itself, it's just that it gives you enough power to get in trouble :) For my single user case it simply isn't needed, for an administrator it may be required. That said, there may be good reasons that I don't know about.