Do not generate pipestatus from spkg/install

[vbraun]

The $SAGE_ROOT/spkg/install has a here document that creates pipestatus. With the sage root repository, this is no longer needed as pipestatus is tracked by the repository. The patch removes it from $SAGE_ROOT/spkg/install.

---

Patch moved to #11073.

CC: @jdemeyer @nexttime

Component: build

Keywords: sd32

Reviewer: Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/10970

[vbraun]

Attachment: trac_10970_do_not_generate_pipestatus.patch.gz

Patch to SAGE_ROOT repo

[sagetrac-mariah]

comment:2

Patch applied, then patched file moved to new source directory. 'make testlong' passed all tests. Positive review.

[sagetrac-mariah]

Reviewer: Mariah Lenox

[vbraun]

comment:3

I've recently played around with automatic testing upgrades and I think this might break upgrades. Its really a bigger issue, the sage_root repository is only upgraded rather late in the upgrade process. I think this needs to be fixed first. I will do some more testing when I have time.

I'll set this ticket to "needs info" until I (or somebody else) tests upgrading.

[jhpalmieri]

comment:4

My guess is that if this causes problems with upgrading, it should only happen when upgrading from versions of Sage which don't already include the file "pipestatus", which would mean some version between 4.4.4 (which doesn't have it) and 4.5.3 (which does). I haven't actually tested anything, though.

Maybe the root repository should be upgraded earlier. In an install from scratch, it has to be installed somewhat late because it depends on mercurial, but when upgrading, mercurial should already be present. This would require altering the upgrade process so that it doesn't just run spkg/install. We could install the root repo by hand (in sage-upgrade) before running spkg/install.

[vbraun]

comment:5

Replying to @jhpalmieri:

Maybe the root repository should be upgraded earlier. In an install from scratch, it has to be installed somewhat late because it depends on mercurial, but when upgrading, mercurial should already be present.

Those were exactly my thoughts, too.

I was mostly motivated by #11073. We can't stuff all files in the spkg/base repo into a here document :-)

Do we have an official policy on how old a Sage version should still be able to upgrade?

[williamstein]

comment:7

Replying to @vbraun:

Do we have an official policy on how old a Sage version should still be able to upgrade?

I declare "1 year" as the official policy, since that's the official policy for deprecation. I think it is a reasonable amount of time.

Since sage-4.5.3 had pipestatus, and was released 2010-09-08 and the next sage release won't be for two weeks (or so), it is fine to deprecate upgrading from pre sage-4.5.3 releases.

-- William

[jdemeyer]

comment:8

pipestatus exists since sage-4.5, released 16 july 2010.

[williamstein]

comment:9

I mentioned this to people at Sage Days and they point out that "sage -upgrade" rarely works anyways, so we shouldn't be be too worried about breaking "sage -upgrade".

[jdemeyer]

comment:10

Replying to @williamstein:

I mentioned this to people at Sage Days and they point out that "sage -upgrade" rarely works anyways^[citation needed]^, so we shouldn't be be too worried about breaking "sage -upgrade".

We do test upgrading on various systems from various Sage versions and we almost never have problems.

Anothe question is: is upgrading really used? I mean, do normal Sage users sometimes do sage -upgrade? If not, then why do we support upgrading?

[williamstein]

comment:11

1. Jdemeyer -- thank you for doing testing. I greatly appreciate it. This would be a great topic for our "testing sage days".

Replying to @jdemeyer:

Anothe question is: is upgrading really used? I mean, do normal Sage users sometimes
do sage -upgrade? If not, then why do we support upgrading?

I use sage -upgrade a lot on my servers that have lots of optional packages installed.

[williamstein]

comment:12

I did some testing of this... and I say: positive review.

[williamstein]

Changed keywords from none to sd32

[nexttime]

comment:14

Replying to @williamstein:

I mentioned this to people at Sage Days and they point out that "sage -upgrade" rarely works anyways, so we shouldn't be be too worried about breaking "sage -upgrade".

We should be worried that (or why they feel) upgrading doesn't work for them though. I guess they just didn't try since last year (when it started working better). ;-)

---

However, I see no real point in (or any compelling reason for) removing pipestatus' creation.

If we do so, despite of that it [just] breaks upgrading from older releases, spkg/install should give an appropriate error message in case pipestatus isn't already present. Unfortunately this would happen after dozens of spkgs to upgrade have been downloaded, which is a bit odd.

If at all, we IMHO should remove it in some "major" release, like e.g. Sage 5.0, or perhaps Sage 4.8, since people are probably less likely to upgrade to such a version (with -upgrade), or at least more likely to read its release notes in advance.

[nexttime]

comment:15

P.S.: The current patch does not even give an [early, intentional] error message in case pipestatus is missing.

I also think supporting sage -upgrade is valuable; note that it still issues a scary warning message we should IMHO change.

[nexttime]

comment:19

I still think removing pipestatus for no reason at this point is simply superfluous.

The patch should IMHO at least test for pipestatus's presence, and issue an appropriate error message (with instructions how to fix this) if it doesn't exist; though quite late in the upgrade process, i.e. after all updated packages have been downloaded, of course.

P.S.: We could even attempt to download it (with e.g. curl or wget) from the script. Or install the root repository "manually" in advance on upgrading, of course from the updated spkg/install, since the other scripts are those of the old version.

[jhpalmieri]

comment:21

In addition to leif's comments, note that the pipestatus script says

# IMPORTANT: if you edit this file, be sure to also edit spkg/install
#            where this file is created for upgrading.

This comment should probably be deleted.

[nexttime]

comment:22

Replying to @jhpalmieri:

In addition to leif's comments, note that the pipestatus script says

# IMPORTANT: if you edit this file, be sure to also edit spkg/install
#            where this file is created for upgrading.

This comment should probably be deleted.

Do you agree that this ticket needs works, w.r.t. the comment, and -- IMHO more important -- the lack of an error check + message on how to fix this?

[vbraun]

comment:23

I think the elephant in the room is #11073: remove the spkg/base repo, add a mechanism that adds the absolutely necessary files to the tarball (or directly downloads them for upgrades) without depending on bzip2/mercurial/.... Then pipestatus would just one of those "base" scripts and everything would be taken care of automatically. So perhaps we should make this ticket just contingent upon #11073 and focus our energy there.

[jhpalmieri]

comment:24

I'm attaching a referee patch printing an error message if pipestatus is missing. We could also patch the scripts repo:

diff --git a/sage-update b/sage-update
--- a/sage-update
+++ b/sage-update
@@ -400,6 +400,8 @@ def do_update():
     os.chdir(SPKG_ROOT)
     download_file("install")
     os.system('chmod +x install')
+    download_file("pipestatus")
+    os.system('chmod +x pipestatus')
     os.chdir(os.path.join(SPKG_ROOT, "standard"))
     download_file("standard/deps")
     download_file("standard/newest_version")
@@ -408,6 +410,7 @@ def do_update():
     os.chdir(SAGE_ROOT)
     if root_repo_intact:
         for file in ['spkg/install',
+                     'spkg/pipestatus',
                      'spkg/standard/deps',
                      'spkg/standard/newest_version']:
             subprocess.call('./sage -hg commit %s -m ' % file +

but I'm not sure it's worth it: pipestatus is not a script which should change a lot, so if you have an out-of-date version, it's probably good enough, and it will get replaced during the upgrade process and used the next time around.

[jhpalmieri]

Attachment: trac_10970-error-msg.patch.gz

apply on top of other patch

[jdemeyer]

comment:25

Replying to @vbraun:

I've recently played around with automatic testing upgrades and I think this might break upgrades. Its really a bigger issue, the sage_root repository is only upgraded rather late in the upgrade process. I think this needs to be fixed first. I will do some more testing when I have time.

Any results? Which version failed to upgrade?

Concerning #11073: I don't see why it should affect this ticket. I think we should be able to merge this ticket now (if it doesn't break upgrades).

[jdemeyer]

Changed reviewer from Mariah Lenox to Jeroen Demeyer, Mariah Lenox

[jdemeyer]

comment:27

The only annoying thing here is that spkg/install is executed twice: if the upgrade fails the first time, it is run again a second time. There is no way for spkg/install to say "this is really a failure, no point trying again". That would require a change to local/bin/sage-upgrade (of the old Sage version!).

[jdemeyer]

comment:28

• Upgrading from 4.4.4 failed (with the appropriate error message)

• Upgrading from 4.5 and 4.6 also failed, but while installing boehm_gc-7.2.alpha6.p0. This looks unlikely to be related to this ticket:

/bin/bash ./libtool --tag=CC   --mode=link gcc -fexceptions -g -O2  -version-info 1:3:0 -no-undefined  -o libgc.la -rpath /mnt/usb1/scratc
h/jdemeyer/sage-4.6/local/lib allchblk.lo alloc.lo blacklst.lo checksums.lo dbg_mlc.lo dyn_load.lo finalize.lo gc_dlopen.lo gcj_mlc.lo hea
ders.lo malloc.lo mallocx.lo mark.lo mark_rts.lo misc.lo new_hblk.lo obj_map.lo os_dep.lo pcr_interface.lo ptr_chck.lo real_malloc.lo recl
aim.lo specific.lo stubborn.lo typd_mlc.lo backgraph.lo thread_local_alloc.lo pthread_start.lo pthread_support.lo pthread_stop_world.lo
atomic_ops.lo mach_dep.lo -lpthread -ldl
libtool: link: gcc -shared  -fPIC -DPIC  .libs/allchblk.o .libs/alloc.o .libs/blacklst.o .libs/checksums.o .libs/dbg_mlc.o .libs/dyn_load.
o .libs/finalize.o .libs/gc_dlopen.o .libs/gcj_mlc.o .libs/headers.o .libs/malloc.o .libs/mallocx.o .libs/mark.o .libs/mark_rts.o .libs/mi
sc.o .libs/new_hblk.o .libs/obj_map.o .libs/os_dep.o .libs/pcr_interface.o .libs/ptr_chck.o .libs/real_malloc.o .libs/reclaim.o .libs/spec
ific.o .libs/stubborn.o .libs/typd_mlc.o .libs/backgraph.o .libs/thread_local_alloc.o .libs/pthread_start.o .libs/pthread_support.o .libs/
pthread_stop_world.o .libs/atomic_ops.o .libs/mach_dep.o   -lpthread -ldl  -O2   -Wl,-soname -Wl,libgc.so.1 -o .libs/libgc.so.1.0.3
libtool: link: (cd ".libs" && rm "libgc.so.1" && ln -s "libgc.so.1.0.3" "libgc.so.1")
rm: cannot remove `libgc.so.1': No such file or directory
make[2]: *** [libgc.la] Error 1
make[2]: Leaving directory `/mnt/usb1/scratch/jdemeyer/sage-4.6/spkg/build/boehm_gc-7.2.alpha6.p0/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/mnt/usb1/scratch/jdemeyer/sage-4.6/spkg/build/boehm_gc-7.2.alpha6.p0/src'
Error building BoehmGC.

real    0m38.188s
user    0m13.470s
sys     0m5.540s
sage: An error occurred while installing boehm_gc-7.2.alpha6.p0
Please email sage-devel http://groups.google.com/group/sage-devel
explaining the problem and send the relevant part of
of /mnt/usb1/scratch/jdemeyer/sage-4.6/install.log.  Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/mnt/usb1/scratch/jdemeyer/sage-4.6/spkg/build/boehm_gc-7.2.alpha6.p0 and type 'make check' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
(cd '/mnt/usb1/scratch/jdemeyer/sage-4.6/spkg/build/boehm_gc-7.2.alpha6.p0' && '/mnt/usb1/scratch/jdemeyer/sage-4.6/sage' -sh)
When you are done debugging, you can type "exit" to leave the
subshell.
make: *** [installed/boehm_gc-7.2.alpha6.p0] Error 1

[jdemeyer]

comment:30

Since this ticket fits well with #11073, I will merge it in #11073.

[jdemeyer]

Changed author from Volker Braun to none

[jdemeyer]

Changed reviewer from Jeroen Demeyer, Mariah Lenox to Jeroen Demeyer