Put more files under revision control.

[jhpalmieri]

Put the text files in $SAGE_ROOT, and also the text files in spkg, under revision control. (See the discussion at the end of #9351.)

Here are the instructions:

---

• apply attachment: trac_9433-sage-repo.2.patch
• apply attachment: trac_9433-scripts.v9.patch to the scripts repository
• move the attached file attachment: hgignore to $SAGE_ROOT/.hgignore (note that this is a new file)
• move the attached file attachment: root-spkg-install.v4 to $SAGE_ROOT/spkg/root-spkg-install (note that this is a new file)
• apply attachment: 9433_install.diff to $SAGE_ROOT/spkg/install
• apply attachment: 9433_deps.diff to $SAGE_ROOT/spkg/standard/deps

Then from $SAGE_ROOT, run the attached script attachment: 9433_hg_script.sh to create the Mercurial repository.

---

Testing: see http://sage.math.washington.edu/home/release/sage-4.7.alpha0/

CC: @williamstein @dandrake @kcrisman @nexttime

Component: distribution

Author: John Palmieri

Reviewer: Leif Leonhardy, Volker Braun, Jeroen Demeyer

Merged: sage-4.7.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/9433

[jhpalmieri]

comment:1

I'm marking this as "needs review", but consider this patch experimental.

[jhpalmieri]

comment:2

A little explanation: this patch creates a directory "other-scripts" in SAGE_ROOT/local/bin. This new directory contains a brief README.txt and also subdirectories "root" and "spkg". "root" contains the text files from SAGE_ROOT. The only one with any changes is README.txt which explains how these files are under revision control. Similarly, "spkg" contains various text files from SAGE_ROOT/spkg, and the only one with any changes is README.txt.

[jhpalmieri]

comment:3

This probably needs to be rebased. When people are ready to look at it, let me know and I'll see what I can do.

[jhpalmieri]

rebased against 4.5.alpha4 + #9456

[jhpalmieri]

comment:4

Attachment: trac_9433-scripts.2.patch.gz

New approach, after a discussion on sage-devel: create a new repo at the top level tracking the appropriate files. I'm attaching a new version of the patch for the scripts repo. Someone -- the release manager, I guess -- also needs to create the top level repo, because I don't know how to do this in such a way that it can be posted on a ticket. Here are the instructions:

• move the attached file "hgignore" to SAGE_ROOT/.hgignore
• cd $SAGE_ROOT
• hg init .
• hg add .hgignore COPYING.txt README.txt makefile sage sage-python
• cd spkg
• hg add README.txt gen_html install pipestatus
• cd standard
• hg add README.txt deps libdist_filelist newest_version
• hg add notes.txt numeric-24.2.txt

(I don't know if we really need these last two files, but this is probably not the ticket for making such decisions.) Finally, do

• hg commit

When you run "sage -sdist ..." it should add a tag for the new version of Sage.

This does not create a new spkg for the files in SAGE_ROOT, since those files have to be in place when you unpack the sage tar file. But it creates the repository so that people can post patches to the trac server, etc.

[williamstein]

comment:5

Looking with my eyes, this looks good. I don't have time to test right now. The test would be to take a clean Sage, do the above, then do "sage -sdist ..." and make sure that in the sdist the above is all still there.

[jhpalmieri]

main repo: add "hg_root" command to Sage

[jhpalmieri]

comment:6

Attachment: trac_9433-sage-repo.patch.gz

Replying to @williamstein:

The test would be to take a clean Sage, do the above, then do "sage -sdist ..." and make sure that in the sdist the above is all still there.

This works for me, but other people should definitely look at it carefully.

[jhpalmieri]

comment:7

This probably needs work: how will it work with "sage -upgrade"?

[jhpalmieri]

comment:8

Here's a new version of the patch for the scripts repo. I think this should deal with upgrading: the script "sage-upgrade" now runs "sage --hg branch" from SAGE_ROOT, and if this fails, it assumes that there is no Mercurial repository and creates it.

[jhpalmieri]

comment:10

This may need to be changed if #9609 is merged: the directions here, and also the modified "sage-upgrade" script, refer to files which would be deleted by #9609.

[jhpalmieri]

comment:11

New version, rebased against #9609.

[jhpalmieri]

new version, using "hg clone". apply to scripts repo.

[jhpalmieri]

Attachment: trac_9433-scripts.3.patch.gz

new version, using "hg clone". apply to scripts repo.

[jhpalmieri]

comment:13

Attachment: trac_9433-scripts.patch.gz

Note that if you use "hg clone ..." to copy the repo, you have to do it earlier in the process: it apparently needs to be done with an empty directory, so in sage-sdist it is now done right after creating $TMP.

[qed777]

comment:15

If I

• Build the forthcoming Sage 4.5.2 (which is just 4.5.2.rc1 + README.txt says " Sage builds with GCC >= 3.x" but it does NOT #9226) from source.
• Follow the steps in the description.
• ./sage -sdist 4.5.99

hg log in the "root" repo now gives

changeset:   1:0fea58e94942
tag:         tip
user:        Mitesh Patel <qed777@gmail.com>
date:        Fri Aug 06 21:40:23 2010 -0700
summary:     Added tag 4.5.99 for changeset 4c1f4320f743

changeset:   0:4c1f4320f743
tag:         4.5.99
user:        Mitesh Patel <qed777@gmail.com>
date:        Fri Aug 06 21:33:45 2010 -0700
summary:     Initial Sage "root" repository

The new 4.5.99 builds successfully from source and passes the long doctests. But hg log in the root repo for 4.5.99 lists just revision 0, and the root repo is missing from a binary distribution made with ./sage -bdist 4.5.99.

[qed777]

Author: John Palmieri

[qed777]

comment:16

I also noticed

• SAGE_ROOT/ipython and SAGE_ROOT/sage-README-osx.txt are missing from the new source and binary distributions.
• An extra SAGE_ROOT/sage-python after unpacking sage-4.5.99.tar.

[jhpalmieri]

comment:17

Replying to @qed777:

The new 4.5.99 builds successfully from source and passes the long doctests. But hg log in the root repo for 4.5.99 lists just revision 0, and the root repo is missing from a binary distribution made with ./sage -bdist 4.5.99.

Right, I didn't do this right. In the new patch, the root repo is not modified at all by sage-make_devel_packages; instead, sage-sdist clones it, tags it, and commits the new tag, while sage-bdist just clones it.

I also noticed

• SAGE_ROOT/ipython and SAGE_ROOT/sage-README-osx.txt are missing from the new source and binary distributions.

The missing ipython directory was an oversight. I think I've fixed it. The missing sage-README-osx.txt was intentional: this should only be included for binary distributions on OS X, and its presence there is taken care of by sage-bdist:

if [ "$UNAME" = "Darwin" ]; then
    ...
    cp sage/local/bin/sage-README-osx.txt README.txt
    ...

Perhaps we can close #6938 if this gets merged?

• An extra SAGE_ROOT/sage-python after unpacking sage-4.5.99.tar.

This was intentional. Before this, the file sage-python was stored in the scripts repo and then unpacked to the top level. I'm not sure why this was done, but I wanted to keep the end result as close as possible to what it was before.

[jdemeyer]

comment:143

John (and/or Volker), do you have time to work on this in the coming week / 2 weeks? If yes, I would like to merge this in sage-4.7.alpha0. This ticket might require some more work to get it right, so it would be nice to know whether you have time. Otherwise we can postpone this to a later Sage release.

[vbraun]

comment:144

I'm around and am in favor of merging it asap.

[jhpalmieri]

comment:145

I have a fix to this problem which I am testing right now. It looks good so far, and I'll post it later if it continues to look good. (The fix is as follows: the script sage-update downloads new versions of several files, like install and deps; I'm now adding a command to commit the Sage root repo after doing this download.)

[jhpalmieri]

comment:146

Okay, I have two options; which is better? In sage-update, the files install, deps, and newest_version might get changed. We could:

• commit the changes in sage-update, and then the changes will get committed again by root-spkg-install when the root repo spkg is installed.

• or in root-spkg-install, we can overwrite any changes: do hg pull ... to pull from the new repo and then hg update --clean to discard any other changes.

The first of these adds a redundant commit message to the log file if the relevant files are changed. The second discards all changes, including any other ones that the user may have made. I don't know Mercurial well enough, but perhaps there is a better third choice?

[vbraun]

comment:147

I'm in favor of the following third option:

1. The first thing that sage-update should do is to commit any outstanding changes and, in particular, abort if there is a patch queue applied. So you are safe even if you forgot to commit your changes.
2. Then sage-update overwrites some critical files for its operation.
3. Finally, the sage_root spkg is installed and its spkg-install overwrites the remaining non-critical files in the root repository. I'd rather be guaranteed to have a clean slate after update than dying in the middle of the update because the attempted merge fails.

So if you had any uncommitted changes before updating, you'll end up with two commit messages. The more individual commits the better. The only drawback is that if you had made modifications to the root repository that you want to keep and you are not using patch queues, then you now have to extract your changes as a patch from the repository and apply that patch. I think thats acceptable since, I think, most developers use patch queues and I don't expect too frequent edits to the root repository.

If that doesn't convince you, I'd prefer option number 1 as my second choice :-)

[jhpalmieri]

comment:148

Here is a new version of the scripts patch. The only change is in sage-update. I updated the instructions, also: I'm not worrying about making the repo compatible with older versions of Mercurial right now.

[jhpalmieri]

comment:149

Oh, also, I created some versions for testing upgrades:

• http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.2.X0/
• http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.2.X1/ (spkg/install changed)
• http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.2.X2/ (Makefile changed)
• http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.2.X3/ (Makefile and spkg/standard/deps changed)

The first of these is just 4.6.2.rc0 with the patches from this ticket applied. In each of the other ones, one or two files have been modified, as indicated. It's also worth testing by updating to the "X0" version, for example, then modifying a file tracked by the root repo (like README.txt) without committing the change, and then trying to update to "X1". The update should abort.

[jhpalmieri]

comment:150

I made a few minor cosmetic changes in the patch, and right now, the upgrade paths I mentioned use the old version, not the new one. I'll have them updated in a few minutes.

[jhpalmieri]

comment:151

Okay, everything's updated now.

[jdemeyer]

comment:152

I made new testing releases:

1. http://boxen.math.washington.edu/home/release/sage-4.7.alpha0/: sage-4.6.2.rc1 + this ticket
2. http://boxen.math.washington.edu/home/release/sage-4.7.alpha1/: sage-4.7.alpha0 + Remove weave from Sage #10688 + attachment: 9433_testing.patch

Upgrading sage-4.7.alpha0 -> sage-4.7.alpha1 succeeded this time.

[vbraun]

comment:153

I've tried the update and it works beautifully.

I think there is still one problem: If the user has no .hgrc (like many non-developers trying to upgrade), then mercurial will fail to commit with

[vbraun@volker-desktop sage-4.7.alpha0]$ mv ~/.hgrc ~/backup.hgrc
[vbraun@volker-desktop sage-4.7.alpha0]$ hg commit -m "test"
abort: no username supplied (see "hg help config")

So whenever we commit to the root repo, we should specify a username explicitly. For example,

hg commit -u "committed by sage -upgrade" -m "test"

[jhpalmieri]

patch for scripts repo

[jhpalmieri]

Attachment: trac_9433-scripts.v9.patch.gz

The file $SAGE_ROOT/spkg/root-spkg-install

[jhpalmieri]

comment:154

Attachment: root-spkg-install.v4.gz

Replying to @vbraun:

I think there is still one problem: If the user has no .hgrc (like many non-developers trying to upgrade), then mercurial will fail

I wouldn't have spotted this. Good catch. I have new patches which add "-u ..." to various commit commands: in sage-update and in root-spkg-install. I haven't bothered with sage-sdist or sage-make_devel_packages, since these are done by the release manager who had better have a .hgrc file. (Besides, there are already other "hg commit" commands in those scripts.)

I've also updated my versions (4.6.2.X0 etc.) for testing upgrades with these changes.

[vbraun]

comment:155

I'm happy with it now. I tested upgrades without ~/.hgrc and everything worked for me.

[jdemeyer]

comment:156

Agreed! This will get merged in sage-4.7.alpha0. I still expect breakage here and there, but that's what alpha versions are for.

[jdemeyer]

Merged: sage-4.7.alpha0