numpy headers symlink breaks using numpy in a venv

[tornaria]

For some (unspecified) reason the numpy template contains this in its post_install():

	# create compat symlinks for .h files
	vmkdir ${py3_inc}
	ln -sfr ${DESTDIR}/${py3_sitelib}/numpy/core/include/numpy \
		${DESTDIR}/${py3_inc}

This creates a symlink /usr/include/python3.10/numpy -> ../../lib/python3.10/site-packages/numpy/core/include/numpy

In effect this means that whenever one uses -I/usr/include/python3.10 to add python headers, one is also adding system numpy headers.

The problem arises when one uses a different version of numpy installed in a venv. To compile against numpy headers one would add -I$VENV/lib/python3.10/site-packages/numpy/core/include to use those headers. However, when the CFLAGS has something like -I/usr/include/python3.10 -I$VENV/lib/python3.10/site-packages/numpy/core/include the symlink installed in the python include dir will take precedence.

This gives a runtime error, e.g. when system numpy is 1.22 (as in void) and venv numpy is 1.21 (as in sagemath) since those versions are binary-incompatible (the numpy.ndarray structure changed size).

A workaround is to ensure the numpy include dir always comes before the python include dir, but the issue is bound to resurface since controlling the order of stuff placed on CFLAGS is kind of difficult.

As a concrete example: sage-9.5 (current release) used to build ok on void linux but after numpy upgraded to 1.22 it no longer builds (this does NOT affect our template for sagemath -- we don't use a venv). See: https://trac.sagemath.org/ticket/33473.

Arguably, nothing else than python headers should be installed in /usr/include/python3.10; installing headers from other packages there is forcing the use of a particular version of the package.

Note: debian/ubuntu also adds this symlink. I didn't find other distros that do (I only looked at a few).

[tornaria]

BTW, the numpy include dir should not be hardcoded, it is supposed to be obtained in setup.py with:

>>> import numpy; print(numpy.get_include())
/usr/lib/python3.10/site-packages/numpy/core/include

(this is when running system python without venv)

>>> import numpy; print(numpy.get_include())
/home/tornaria/src/sage/sage-9.5/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/numpy/core/include

(this is when running system python with a venv)

If a python package doesn't compile without the symlink, this is a bug in the package which should use numpy.get_include() to know which numpy headers to use. This is the only way to make sure the package uses the correct headers when using numpy from a venv (because numpy.get_include() should be called from setup.py in the venv so it will give the correct answer, meaning the one that matches the numpy package in the venv).

[tornaria]

Note: debian/ubuntu also adds this symlink. I didn't find other distros that do (I only looked at a few).

Fedora installs the symlink as /usr/include/numpy, which I guess is ok for the current issue but would cause a conflict between python-numpy and python3-numpy (cf https://bugzilla.redhat.com/show_bug.cgi?id=185079 and https://src.fedoraproject.org/rpms/numpy/blob/f36/f/numpy.spec#_142)

[tornaria]

@ahesford I'd appreciate to know your point of view on this issue.

[ahesford]

I agree that these links are problematic.

This is probably a hack around path issues for cross-builds that compile C extensions with Numpy headers that is no longer necessary. I took some care to properly order CFLAGS in the numpy build helper; hopefully we can just drop the links.

I think there is a new Numpy release; we can bump the version, drop the symlinks and test some packages that rely on the build helper to confirm that everything works as expected.