Find and uninstall broken installed SPKGs and wheels

[mkoeppe]

When system packages are upgraded, shared libraries that a Sage installation links to can disappear, rendering the Sage installation broken. This is a common problem, reported for example in https://groups.google.com/g/sage-devel/c/Iz8ZsmQM3Pg/m/CPl9CbHmBgAJ

To avoid a full rebuild (make distclean && make build), we can try to find the specific broken packages.

The new command make -j list-broken-packages assists the user with this:

$ make -j list-broken-packages
make --no-print-directory auditwheel_or_delocate-no-deps
sage-logger -p 'sage --pip install -r "/Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt"' '/Users/mkoeppe/s/sage/sage-rebasing/logs/pkgs/auditwheel_or_delocate.log'
[auditwheel_or_delocate] Ignoring auditwheel: markers 'sys_platform != "darwin"' don't match your environment
[auditwheel_or_delocate] Requirement already satisfied: delocate in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from -r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.10.2)
[auditwheel_or_delocate] Requirement already satisfied: wheel>=0.32.0 in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from delocate->-r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.37.1)
[auditwheel_or_delocate] Requirement already satisfied: typing-extensions in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from delocate->-r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (3.10.0.0)
make[2]: warning: -jN forced in submake: disabling jobserver mode.
# Checking /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/installed/bliss-0.73+debian-1+sage-2016-08-02.p0
...
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/lib/libumfpack.dylib'
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib'
Error during installcheck of 'suitesparse': /Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib
Uninstall broken packages by typing:


    make antic-SAGE_LOCAL-uninstall;
    make e_antic-SAGE_LOCAL-uninstall;
    make lcalc-SAGE_LOCAL-uninstall;
    make ratpoints-SAGE_LOCAL-uninstall;
    make normaliz-SAGE_LOCAL-uninstall;
    make r-SAGE_LOCAL-uninstall;
    make backports_zoneinfo-SAGE_VENV-uninstall;
    make fastjsonschema-SAGE_VENV-uninstall;
    make pytz_deprecation_shim-SAGE_VENV-uninstall;
    make stack_data-SAGE_VENV-uninstall;
    make tinycss2-SAGE_VENV-uninstall;
    make tzdata-SAGE_VENV-uninstall;
    make suitesparse-SAGE_LOCAL-uninstall;

real	2m25.787s
user	4m54.270s
sys	4m0.796s

The core functionality is implemented by the new command sage-spkg-installcheck (implemented in sage_bootstrap):

$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose givaro'
Checking shared library file 'lib/libgivaro.9.dylib'
Checking shared library file 'lib/libgivaro.dylib'
$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose matplotlib $SAGE_VENV'
Checking wheel file 'var/lib/sage/wheels/matplotlib-3.5.1-cp310-cp310-macosx_12_0_x86_64.whl'
$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose e_antic'
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/lib/libeantic.0.dylib'
ERROR [libsana|get_dependencies:137]: 
/usr/local/opt/flint/lib/libflint-16.dylib not found:
  Needed by: /Users/mkoeppe/s/sage/sage-rebasing/local/lib/libeantic.0.dylib
Error during installcheck of 'e_antic': Could not find all dependencies.

It inspects the installation records in $SAGE_LOCAL/var/lib/sage/installed/ and all shared libraries and platform wheels listed there.

On macOS, instead of using otool directly, we use delocate-listdeps from https://pypi.org/project/delocate/

On Linux, instead of using ldd directly, we use auditwheel.lddtree.lddtree (https://github.com/pypa/auditwheel/blob/main/src/auditwheel/lddtree.py)

(Both packages are also used by cibuildwheel)

See also: #34205 Optionally run auditwheel/delocate on wheels before installing them

Depends on #29097

CC: @dimpase @jhpalmieri @nbruin @EmmanuelCharpentier

Component: build

Author: Matthias Koeppe

Branch/Commit: 3c70a38

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/34203

[mkoeppe]

Branch: u/mkoeppe/find_and_uninstall_broken_installed_spkgs_and_wheels

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

562fcab

build/bin/sage-spkg-installcheck, build/sage_bootstrap/installcheck.py: New

[sagetrac-git]

Commit: 562fcab

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

5faa8cb

build/pkgs/auditwheel_or_delocate: New

[sagetrac-git]

Changed commit from 562fcab to 5faa8cb

[sagetrac-git]

Changed commit from 5faa8cb to 9265416

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

9265416

build/sage_bootstrap/installcheck.py: Implement for macOS

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

a18b205	build/sage_bootstrap/installcheck.py: Do not warn if there are no files to check
ad92af4	build/sage_bootstrap/installcheck.py: Skip pure Python wheels
c4e3371	build/make/Makefile.in (list-broken-packages): New

[sagetrac-git]

Changed commit from 9265416 to c4e3371

[mkoeppe]

Author: Matthias Koeppe

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

dbdf33b	build/sage_bootstrap/installcheck.py: Only warn when delocate is not available
08388f7	build/make/Makefile.in (list-broken-packages): Parallelize it

[sagetrac-git]

Changed commit from c4e3371 to 08388f7

[mkoeppe]

comment:17

Ready for testing on macOS.

[sagetrac-git]

Changed commit from 08388f7 to f2dba63

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

f2dba63

build/sage_bootstrap/uninstall.py: Do not refuse to uninstall packages that do not exist in the source tree

[sagetrac-git]

Changed commit from f2dba63 to b439429

[mkoeppe]

comment:45

Replying to John Palmieri:

Can you explain what "package" means in list-broken-packages?

An SPKG

[mkoeppe]

comment:46

Replying to John Palmieri:

Is stack_data actually broken, or is the "usage" message indication of something else going wrong? And what is stack_data, since it's not an spkg?

It's broken as follows: You have an installation of it (from some other branch), but it's not (per what is currently in build/pkgs) an SPKG.

[mkoeppe]

comment:47

In 9.7.rc1, stack_data of course is a package, so after merging 9.7.rc1, this will no longer be reported as "broken".

[sagetrac-git]

Changed commit from ef9226b to 0bbba44

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

0bbba44

Merge tag '9.7.rc1' into t/34203/find_and_uninstall_broken_installed_spkgs_and_wheels

[mkoeppe]

comment:49

And here's that merge commit.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

9a03d7f

build/sage_bootstrap/installcheck.py: Make error message more specific

[sagetrac-git]

Changed commit from 0bbba44 to 9a03d7f

[jhpalmieri]

comment:51

Should this have a nonzero return code if some packages are broken?

[mkoeppe]

comment:52

From its name list-broken-packages, I tend to say that it should not indicate a failure like that

[jhpalmieri]

comment:53

I think I agree but I wanted to ask.

[jhpalmieri]

comment:54

It is possible that some could want to do make -j list-broken-packages && do something else, but I don't see a concrete example.

[mkoeppe]

comment:55

Yes, I thought about this, but I'd say that this whole thing is a strictly manual operation, which doesn't lend itself to scripting.

[jhpalmieri]

comment:56

For the new package auditwheel_or_delocate, it seems that potentially both are used on Linux, only delocate on OS X. If that's right, then the SPKG.rst should be modified.

Installing auditwheel via pip also installs pyelftools, and patchelf is listed as a dependency on the PyPI page. Do we need to create a new package pyelftools, too? Is patchelf really required, or is it only necessary if one is trying to fix files, not just detect whether they're broken?

[jhpalmieri]

comment:57

The comment "and delete the rest" does not seem to be accurate (and I think it's fine not to delete the old stamp files here).

[sagetrac-git]

Changed commit from 9a03d7f to 6ceb44e

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

6ceb44e

build/pkgs/auditwheel_or_delocate/SPKG.rst: Explain that we use delocate also on Linux

[mkoeppe]

comment:59

Replying to John Palmieri:

Installing auditwheel via pip also installs pyelftools, and patchelf is listed as a dependency on the PyPI page. Do we need to create a new package pyelftools, too?

Not needed - that's what makes optional "pip" packages so convenient.

Is patchelf really required, or is it only necessary if one is trying to fix files, not just detect whether they're broken?

Right, it's only needed for repairing wheels. But it's best not try to circumvent the declared dependencies.

The scipy upgrade (#34081) will bring a patchelf, by the way.

[sagetrac-git]

Changed commit from 6ceb44e to 3c70a38

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

3c70a38

build/sage_bootstrap/installcheck.py: Update comment on stamp files

[jhpalmieri]

comment:61

Great, I think this is ready to go now.

[jhpalmieri]

Reviewer: John Palmieri

[mkoeppe]

comment:62

Thank you!

[vbraun]

Changed branch from u/mkoeppe/find_and_uninstall_broken_installed_spkgs_and_wheels to 3c70a38