Check SPKGs more vigorously for common problems

[qed777]

When we build new or updated Sage packages (spkgs), SAGE_LOCAL/bin/sage-pkg runs a few checks for common problems. For example,

$ sage -pkg foo/
Creating Sage package foo/
Warning: no version number detected

Created package foo.spkg.

    NAME: foo
 VERSION:
    SIZE: 17.8M
 HG REPO: Good
SPKG.txt: Good

Please test this package using
[...]

But we could add several new, more detailed tests for spkg-install, etc. Or put them in a sage-spkg-{check,checker,lint} script.

Component: packages: standard

Reviewer: Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/9622

[qed777]

comment:1

Possibilities, some of which could simply give warnings or reminders:

• Check for spkg-check and spkg-install and that they're executable.
• Check the first line of scripts for #!/usr/bin/env (cf. Fix first line of pari-2.3.5.p1's spkg-install #9597).
• Check for an unfinished Mercurial patch queue.
• Check for the presence of set -e.
• Check for the presence of $MAKE and other important variables.
• Check for Python scripts and whether Python is a dependency in spkg/standard/deps (cf. sage-4.5.alpha3 fails on bsd.math #9435, if spkg-install is a Python script, first check that SAGE_LOCAL/bin/python exists. #9507).

I'm sure there are others!

[nexttime]

comment:2

Yep, but two comments:

• I wonder how many people actually use sage-pkg to create spkgs (rather than just use tar).

• In a perfect world, the reviewer(s) would do this job (i.e. look at the code e.g., too, not just try to install a new package and then give positive review, or sometimes the other way around).

I was thinking of a good spkg template (-install, -check, SPKG.txt), too. And perhaps some minimalistic shell programming guide targeted at Sage scripts.

[nexttime]

comment:3

I've created a (vaguely) related ticket:

Update and extend "SPKG Tracking" Wiki page, #9626.

(sage-pkg could spit out some reminder...)

Comments there welcome.

[nexttime]

comment:4

We could also formalize/code some aspects of SPKG.txt's Special Update/Build Instructions, e.g. in (a) new (optional) script(s) like spkg-cleanup-upstream (deleting parts not needed by Sage, upstream repo files, etc.), contained in the spkg itself, and perhaps run automatically.

Some upstream managers also tend to copy files without preserving mtime, so we could more or less automatically check for files like configure, make sure they are newer than their sources, and if not, touch them before preparing the spkg.

The application of patches to the upstream source tree (i.e. currently copying of patched files) could also be "normalized", allowing to perform some spkg sanity checks automatically, too, and to simplify spkg-installs.

[qed777]

comment:5

Replying to @nexttime:

• I wonder how many people actually use sage-pkg to create spkgs (rather than just use tar).

Good question. I usually use sage -spkg, an alias of sage -pkg.

• In a perfect world, the reviewer(s) would do this job (i.e. look at the code e.g., too, not just try to install a new package and then give positive review, or sometimes the other way around).

Just a quick thought: We could also run the spkg checks, or a subset of them, during installation.

[sagetrac-drkirkby]

comment:6

Replying to @qed777:

Possibilities, some of which could simply give warnings or reminders:

• Check for spkg-check and spkg-install and that they're executable.

Good idea

• Check the first line of scripts for #!/usr/bin/env (cf. Fix first line of pari-2.3.5.p1's spkg-install #9597).

Yes.

• Check for an unfinished Mercurial patch queue.

Yes

• Check for the presence of set -e.

William will not have that. He is very against the use of set -e.

• Check for the presence of $MAKE and other important variables.

What's important in one package is not in another. That might be difficult to do.

• Check for Python scripts and whether Python is a dependency in spkg/standard/deps (cf. sage-4.5.alpha3 fails on bsd.math #9435, if spkg-install is a Python script, first check that SAGE_LOCAL/bin/python exists. #9507).

I'm sure there are others!

If the package is called foobar-x.y.z, then SPKG.txt should have the string foobar.x.y.z somewhere in it. Many times commits get made, with no entry in SPKG.txt

Dave

[sagetrac-drkirkby]

comment:7

One other thing, make sure all the required sections from SPKG.txt exist. i.e. none are missing. Even if there are no special build instructions, the section should exist and simply say "none".

One could also check that entries in SPKG.txt have something in each section. But this could get a lot of work.

[jdemeyer]

Reviewer: Jeroen Demeyer

[jdemeyer]

comment:8

Close because we no longer use spkgs.