Upgrading from older Python versions doesn't work

[jdemeyer]

This has been reported several time on sage-devel but now I finally managed to reproduce it.

The builds ends with

gcc -pthread -shared -L. -L/home/jdemeyer/sage-git/local/lib -Wl,-rpath,/home/jdemeyer/sage-git/local/lib -L. -L/home/jdemeyer/sage-git/local/lib -Wl,-rpath,/home/jdeme
yer/sage-git/local/lib build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/_ctypes.o build/temp.linux-ppc
64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/callbacks.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var
/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/callproc.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modu
les/_ctypes/stgdict.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/cfield.o build/temp.linux-ppc64
le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/prep_cif.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/l
ocal/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/closures.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2
-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/ffi.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctyp
es/libffi/src/powerpc/ffi_sysv.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/f
fi_linux64.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/sysv.o build/temp.lin
ux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/ppc_closure.o build/temp.linux-ppc64le-2.7/home/
jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/linux64.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/
var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/linux64_closure.o -L/home/jdemeyer/sage-git/local/lib -L/usr/lib/powerpc64le-linux-gnu -L/us
r/local/lib -L. -lpython2.7 -o build/lib.linux-ppc64le-2.7/_ctypes.so
*** WARNING: renaming "_ctypes" since importing it failed: build/lib.linux-ppc64le-2.7/_ctypes.so: undefined symbol: _PySlice_Unpack

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _ssl               _tkinter        
bsddb185           dbm                dl              
gdbm               imageop            sunaudiodev     
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_ctypes            _elementtree       _multiprocessing
array              mmap                               

running build_scripts
creating build/scripts-2.7
copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Tools/scripts/pydoc -> build/scripts-2.7
copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Tools/scripts/idle -> build/scripts-2.7
copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Tools/scripts/2to3 -> build/scripts-2.7
copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Lib/smtpd.py -> build/scripts-2.7
changing mode of build/scripts-2.7/pydoc from 664 to 775
changing mode of build/scripts-2.7/idle from 664 to 775
changing mode of build/scripts-2.7/2to3 from 664 to 775
changing mode of build/scripts-2.7/smtpd.py from 664 to 775
make[3]: Leaving directory '/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src'
Testing importing of various modules...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Lib/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: No module named _ctypes
ctypes module failed to import

The problem is that the Python build system imports ctypes at build-time but this will use the wrong version of libpython. The build system tried to work around that with

# Make sure -L. is placed before -L$SAGE_LOCAL/lib so that python and extension
# modules are linked with the right libpython; we pass this in at make time
# only, since we don't want -L. to be saved as one of the default LDFLAGS
# used for building third-party extension modules
$MAKE LDFLAGS="-L. $LDFLAGS"

However, this affects building only, not running. And even using LD_LIBRARY_PATH won't work since an rpath takes priority over that.

CC: @embray

Component: packages: optional

Author: Jeroen Demeyer

Branch/Commit: 379adfd

Reviewer: Erik Bray

Issue created by migration from https://trac.sagemath.org/ticket/24605

[jdemeyer]

Branch: u/jdemeyer/upgrading_from_older_python_versions_doesn_t_work

[jdemeyer]

New commits:

4a64f3e

Remove old Python libraries *before* build

[jdemeyer]

Commit: 4a64f3e

[embray]

comment:5

I don't like the idea of removing the existing libs before building. Let me think about this a bit.

[jdemeyer]

comment:6

OK, I'll let you think a bit. But if you cannot come up with a better solution, please accept this one which works.

[embray]

comment:7

It is annoying that, in this sense, Python's build process is basically incompatible with setting a custom rpath. I wonder if this has been discussed anywhere else.

[jdemeyer]

comment:8

The unusual thing about Python is that it actually runs itself as part of the build process...

[jdemeyer]

comment:9

Just wanted to point out that we should fix this before the next Python upgrade. But as there isn't even a plan for 2.7.15 at the moment, that doesn't look urgent.

[jdemeyer]

comment:10

Python 2.7.15 is planned for the end of this month.

[embray]

Reviewer: Erik Bray

[embray]

comment:11

I'm okay with it as a temporary solution. I think a better solution is needed in the long term, but that could come more or less automatically with #22510 (which would simplify completely uninstalling a package before upgrading it). I'm working on updating that work again so that we can start using it soon, hopefully.

[fchapoton]

Author: Jeroen Demeyer

[jhpalmieri]

comment:13

I feel like I'm missing something: the ticket description deals with Python 2 but the branch only deals with Python 3. What's going on?

[jdemeyer]

comment:14

There is a symlink from the Python 2 build scripts to the Python 3 build scripts.

[sagetrac-git]

Changed commit from 4a64f3e to 379adfd

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. This was a forced push. New commits:

379adfd

Remove old Python libraries *before* build

[jdemeyer]

comment:16

Rebased to 8.2.rc3

[vbraun]

Changed branch from u/jdemeyer/upgrading_from_older_python_versions_doesn_t_work to 379adfd