git-related doctest failures after moving sage installation

[dkrenn]

I've compiled sage in /local/data/krenn/sage-dev/sage-6.1.1 and then moved it to another directory. I am getting the following when testing sage:

File "src/sage/dev/test/config.py", line 43, in sage.dev.test.config.DoctestConfig
Failed example:
    DoctestConfig()
Expected:
    Config('''
    [trac]
    username = doctest
    ticket_cache = ...
    [UI]
    log_level = 1
    [git]
    ssh_key_set = True
    repository_anonymous = remote_repository_undefined
    repository = remote_repository_undefined
    src = ...
    dot_git = ...
    [sagedev]
    ''')
Got:
    warning: templates not found /local/data/krenn/sage-dev/sage-6.1.1/local/share/git-core/templates
    Config('''
    [trac]
    username = doctest
    ticket_cache = /home/krenn/.sage/temp/brown/8826/dir_0ABZH9/ticket_cache
    [UI]
    log_level = 1
    [git]
    ssh_key_set = True
    repository_anonymous = remote_repository_undefined
    repository = remote_repository_undefined
    src = /home/blub/.sage/temp/brown/8826/dir_0ABZH9/repo
    dot_git = /home/blub/.sage/temp/brown/8826/dir_0ABZH9/repo/.git
    user.name = doctest
    user.email = doc@test.test
    user_email_set = True
    [sagedev]
    ''')

There are a couple of other files with similar warnings.

CC: @ppurka

Component: misc

Keywords: git, relocate

Author: John Palmieri

Branch/Commit: 782e3f6

Reviewer: Peter Bruin

Issue created by migration from https://trac.sagemath.org/ticket/15901

[dkrenn]

comment:1

FYI: A manual fix is to use

rm local/var/lib/sage/installed/git-1.8.4.4
make

to rebuild git.

[nexttime]

comment:2

Replying to @dkrenn:

FYI: A manual fix is to use

rm local/var/lib/sage/installed/git-1.8.4.4
make

to rebuild git.

Or just

./sage -f git

[jhpalmieri]

Branch: u/jhpalmieri/git-relocate

[jhpalmieri]

comment:4

Here's an attempt at a patch. Works for me.

---

New commits:

62524f6

Get rid of warnings about missing git templates after relocating Sage

[jhpalmieri]

Commit: 62524f6

[jhpalmieri]

Author: John Palmieri

[ppurka]

comment:5

How do you compile sage after this?

sage --dev checkout --ticket 15901
sage -btp --long src/sage/dev

does not work.

Even the following doesn't work

cp {src,local}/bin/sage-env
sage -btp --long src/sage/dev

[nexttime]

comment:6

Replying to @ppurka:

How do you compile sage after this?

You have to touch the src/sage/dev/[test/] Python files (or remove their .pyc files).

(I think.)

[nexttime]

comment:7

Ahem, or maybe something else... 8-)

[jhpalmieri]

comment:8

Hmm. I guess I should have said "Worked for me yesterday on one machine." I can't get it to work today on a different machine (and haven't tried on the first one yet).

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

782e3f6

Get rid of warnings about missing git templates after relocating Sage

[sagetrac-git]

Changed commit from 62524f6 to 782e3f6

[jhpalmieri]

comment:10

Okay, here's another attempt. Please try this one. (I hope that just sage -tp src/sage/dev works. There are no changes to Python files, so sage -b should be unnecessary.)

[pjbruin]

comment:12

This must explain the failures in the first patchbot run of 6.2 (http://patchbot.sagemath.org/ticket/0/, 2014-05-06 20:45:32 +0100); I made a fresh Sage install for experimenting with the patchbot, but then decided to move it. Patchbot is now running in the original location, but I'll try this patch on another copy.

[pjbruin]

comment:13

The patchbot instance is now getting doctest failures in sage/plot/plot.py:

RuntimeError: Could not open facefile /home/pbruin/src/sage/local/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-linux-x86_64.egg/matplotlib/mpl-data/fonts/ttf/Vera.ttf; Cannot_Open_Resource

The directory it is looking in is where my other Sage copy was before moving that copy to test this ticket! The path seems to have been obtained from a cache living in my home directory: ~/.cache/matplotlib/fontList.cache. Does this mean that sage-location potentially has to worry about hardcoded paths existing in $DOT_SAGE?

[pjbruin]

comment:14

Curiously, sometime after the above failures in sage/plot, my patchbot copy of Sage seems to have updated the paths in ~/.cache/matplotlib. Anyway, I can confirm that the failures in sage/dev are fixed by this patch.

[pjbruin]

Reviewer: Peter Bruin

[pjbruin]

comment:15

Replying to @pjbruin:

I can confirm that the failures in sage/dev are fixed by this patch.

... which should be enough for a positive review. If there really is a problem with the matplotlib cache, it should be dealt with on a different ticket.

[nexttime]

comment:16

Replying to @pjbruin:

Replying to @pjbruin:

I can confirm that the failures in sage/dev are fixed by this patch.

... which should be enough for a positive review. If there really is a problem with the matplotlib cache, it should be dealt with on a different ticket.

Then we should probably also change the ticket's title accordingly.

[nexttime]

comment:17

Replying to @pjbruin:

If there really is a problem with the matplotlib cache, it should be dealt with on a different ticket.

This really deserves its own ticket (and btw. is pretty unrelated to relocating Sage; just imagine having different versions of Sage running at the same time).

IMHO Sage's matplotlib must not put its cache into $HOME (if at all, it should be in $DOT_SAGE/.cache/...), and it should somehow be versioned (with matplotlib's and/or Sage's version in the folder name).

[nexttime]

comment:18

Replying to @nexttime:

IMHO Sage's matplotlib must not put its cache into $HOME (if at all, it should be in $DOT_SAGE/.cache/...), and it should somehow be versioned (with matplotlib's and/or Sage's version in the folder name).

... or probably $SAGE_LOCAL/var/{lib,tmp}/matplotlib/...

[jhpalmieri]

comment:19

Does it need to be writeable? Anyway, it looks like we can modify this by setting the environment variable XDG_CACHE_HOME: see the functions _get_cachedir and _get_xdg_cache_dir in sage/local/lib/python/site-packages/matplotlib-1.3.1-py2.7-macosx-10.8-x86_64.egg/matplotlib/__init__.py. I also think this belongs on another ticket.

[pjbruin]

comment:20

Replying to @nexttime:

IMHO Sage's matplotlib must not put its cache into $HOME (if at all, it should be in $DOT_SAGE/.cache/...), and it should somehow be versioned (with matplotlib's and/or Sage's version in the folder name).

In fact that is where it used to be: I have an old directory ~/.sage/matplotlib-1.2.1/ lying around, containing fontList.cache and tex.cache/, just like ~/.cache/matplotlib/ looks now.

[jhpalmieri]

comment:21

By default, Sage's matplotlib should write to .sage/matplotlib-1.3.1. Do you have any environment variables set, like MPLCONFIGDIR? On my OS X machine, I just checked: MPLCONFIGDIR (which Sage sets to .sage/matplotlib-1.3.1 if it's unset) still gets fontList.cache and tex.cache/ added to it when you call plot from Sage. My $HOME/.cache directory only contains fontconfig/.

[nexttime]

comment:22

Replying to @jhpalmieri:

By default, Sage's matplotlib should write to .sage/matplotlib-1.3.1. Do you have any environment variables set, like MPLCONFIGDIR? On my OS X machine, I just checked: MPLCONFIGDIR (which Sage sets to .sage/matplotlib-1.3.1 if it's unset) still gets fontList.cache and tex.cache/ added to it when you call plot from Sage. My $HOME/.cache directory only contains fontconfig/.

Well, setting up MPL's config dir based on its version in sage-env in turn can indeed break upon relocation... :-)

(as it calls Python before sage-location is called, and the Python interpreter may try to load libraries that have moved, as I already mentioned elsewhere).

[nexttime]

comment:23

Replying to @nexttime:

Well, setting up MPL's config dir based on its version in sage-env in turn can indeed break upon relocation... :-)

(as it calls Python before sage-location is called, and the Python interpreter may try to load libraries that have moved, as I already mentioned elsewhere).

See for example #16044 comment:7 ff.

(Note that IIRC sage-location currently wouldn't help there anyway, as it doesn't mess with install_names.)

[pjbruin]

comment:24

Replying to @jhpalmieri:

By default, Sage's matplotlib should write to .sage/matplotlib-1.3.1. Do you have any environment variables set, like MPLCONFIGDIR? On my OS X machine, I just checked: MPLCONFIGDIR (which Sage sets to .sage/matplotlib-1.3.1 if it's unset) still gets fontList.cache and tex.cache/ added to it when you call plot from Sage. My $HOME/.cache directory only contains fontconfig/.

On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in sage -sh. This is despite `logs/pkgs/matplotlib-1.3.1.log saying

When Sage runs, MPLCONFIGDIR will be set to '$DOT_SAGE/matplotlib-1.3.1'.

[nexttime]

comment:25

Replying to @pjbruin:

On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in sage -sh.

True. B)

$ ./sage --sh -c 'env | grep MPL'; uname -sm
Linux x86_64

[pjbruin]

comment:26

Replying to @nexttime:

Replying to @pjbruin:

On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in sage -sh.

True. B)

It turns out that this is because the following command in sage-env fails (I removed the 2>/dev/null at the end:

"$SAGE_ROOT/local/bin/python" -c 'import pkg_resources; pkg_resources.get_distribution("matplotlib").version'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
zipimport.ZipImportError: can't decompress data; zlib not available

[nexttime]

comment:27

Replying to @pjbruin:

Replying to @nexttime:

Replying to @pjbruin:

On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in sage -sh.

True. B)

It turns out that this is because the following command in sage-env fails (I removed the 2>/dev/null at the end:

"$SAGE_ROOT/local/bin/python" -c 'import pkg_resources; pkg_resources.get_distribution("matplotlib").version'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
zipimport.ZipImportError: can't decompress data; zlib not available

I'm getting

.../local/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

So apparently LD_LIBRARY_PATH isn't set up properly to that time.

[pjbruin]

comment:28

The problem in comment:26 was on 6.2.rc0; with 6.2 on a different system I get

Traceback (most recent call last):
...
ImportError: cannot import name MAXREPEAT

I guess the MPLCONFIGDIR-relevant part of sage-env should be moved down to a point where all required setting up has been done.

[nexttime]

comment:29

Oh, just noticed:

# Use a matplotlib config directory specific to Sage and specific to
# the version number of matplotlib, by setting the environment
# variable MPLCONFIGDIR. Note that we can't find the version number by
# importing matplotlib, because that could create matplotlib's standard
# config directory. So we use pkg_resources.
"$SAGE_ROOT/local/bin/python" -c 'import pkg_resources; pkg_resources.get_distribution("matplotlib").version' 2>/dev/null
if [ $? -eq 0 ]; then
    MPLVERSION=`"$SAGE_ROOT/local/bin/python" -c 'import pkg_resources; print pkg_resources.get_distribution("matplotlib").version'`
    MPLCONFIGDIR="$DOT_SAGE/matplotlib-$MPLVERSION"
    export MPLCONFIGDIR
    # The directory is created when Sage starts (see sage.misc.misc).
fi

# Add some directories to $LD_LIBRARY_PATH:
# * lib/openmpi is needed for openmpi.
# * lib/R/lib is needed for R in case the Sage install is moved.
# * lib32 and lib64 are needed for GCC, see #12405.
for d in lib/openmpi lib/R/lib lib32 lib64 lib; do
    libdir="$SAGE_LOCAL/$d"
    # Add only existing directories
    if [ -d "$libdir" ]; then
        [ -z "$LD_LIBRARY_PATH" ] || LD_LIBRARY_PATH=":${LD_LIBRARY_PATH}"
        LD_LIBRARY_PATH="${libdir}$LD_LIBRARY_PATH"
    fi
done
export LD_LIBRARY_PATH

>8-O

[nexttime]

comment:30

(I meant to say it's done right after we try to call python; I've slightly edited the snippet.)

[pjbruin]

comment:31

I created #16305 for this.

[vbraun]

Changed branch from u/jhpalmieri/git-relocate to 782e3f6