ImportError: No module named sagenb on openSUSE

[qed777]

Florent Hivert reports on sage-release that Sage refuses to start after building 4.6.rc0 from scratch on openSUSE 11.3:

popcorn-*age/sage-4.6.rc0 $ ./sage 
[...]
    ImportError: No module named sagenb 

According to the thread, the problem seems to be that installing the sagenb package does not yield an

SAGE_ROOT/local/lib/python/site-packages/easy-install.pth

that contains any path to the Sage Notebook files.

This is a possible follow-up to #10097, which is supposed to replace the line ./sagenb-0.8.2-py2.6.egg --- inserted by setuptools into easy-install.pth --- with ../../../../devel/sagenb.

CC: @hivert @jasongrout @sagetrac-drkirkby @TimDumol @jdemeyer

Component: notebook

Reviewer: Leif Leonhardy

Issue created by migration from https://trac.sagemath.org/ticket/10176

[hivert]

comment:1

Hi there,

Actually I had the same error with a openSuSE 11.1 (64 bits if it maters).

[nexttime]

comment:3

Replying to @hivert:

Hi there,

Actually I had the same error with a openSuSE 11.1 (64 bits if it matters).

So we can exclude the possibility of a broken sed I guess...

I really would try just adding #!/usr/bin/env bash to (the generation of) SageNB's spkg-install, since apparently Sage 4.6.rc0 built fine and passed all tests on Skynet's OpenSuSE 11.1 box (menas).

(If one changes that, one could of course also add some debug code to see if and how the easy-install.pth gets modified; also, one should check more exit codes in the script.)

[nexttime]

comment:4

The attached patch to the SageNB spkg adds #!... and some error checking to its spkg-install, which now also prints the old and new SageNB path in easy-install.pth - at least for debugging...

Florent, please (apply,) build, test and report! ;-)

(I have no idea if that solves the problem, therefore a temporary patch and "needs info". At least we should hopefully be able to track this further down.)

---

To apply the patch, extract the original SageNB package (best from a Sage shell, since we need a suitable Python installation), cd to sagenb-0.8.7/src/sagenb/, then apply the patch and run ./spkg-dist to create a new sagenb-0.8.7-patched.spkg, located in sagenb-0.8.7/src/sagenb/dist/ (i.e., ./dist/).

[hivert]

comment:5

Florent, please (apply,) build, test and report! ;-)

Just to let you now: test in progress...

[hivert]

comment:6

The compilation of sage is not yet finished but from install.log

Finished processing dependencies for sagenb==0.8.7-patched
Old path: ""
New path: ""

real    0m25.564s
user    0m13.760s
sys     0m3.817s
Successfully installed sagenb-0.8.7-patched

[nexttime]

comment:7

Replying to @hivert:

The compilation of sage is not yet finished but from install.log

Finished processing dependencies for sagenb==0.8.7-patched
Old path: ""
New path: ""

real    0m25.564s
user    0m13.760s
sys     0m3.817s
Successfully installed sagenb-0.8.7-patched

Hmmm, so something else goes wrong (i.e. there's no sagenb path listed there at all, unless my grep failed for some reason...)

Btw, it should have been sufficient to just install the patched spkg.

Can you take a look at / post your $SAGE_ROOT/local/lib/python/site-packages/easy-install.pth?

Maybe we have a race condition in editing that? (You could retry with ./sage -f ... when your build has finished.)

[nexttime]

comment:8

Just for the record, I got:

...
Searching for setuptools==0.6c9
Best match: setuptools 0.6c9
Processing setuptools-0.6c9-py2.6.egg
setuptools 0.6c9 is already the active version in easy-install.pth
Installing easy_install script to /home/leif/Sage/sage-4.6.rc0-short-logs/local/bin
Installing easy_install-2.6 script to /home/leif/Sage/sage-4.6.rc0-short-logs/local/bin

Using /home/leif/Sage/sage-4.6.rc0-short-logs/local/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg
Finished processing dependencies for sagenb==0.8.7-patched
Old path: "/home/leif/Sage/sage-4.6.rc0-short-logs/devel/sagenb-main"
New path: "../../../../devel/sagenb"

real	0m8.624s
user	0m5.800s
sys	0m2.570s
Successfully installed sagenb-0.8.7-patched
Now cleaning up tmp files.
Making Sage/Python scripts relocatable...
Making script relocatable
Finished installing sagenb-0.8.7-patched.spkg

[nexttime]

comment:9

Replying to @nexttime:

Maybe we have a race condition in editing that? (You could retry with ./sage -f ... when your build has finished.)

Could you also take a look at easy-install.pth's modification time after reinstalling the spkg?

[nexttime]

comment:10

To brute-force-fix your problem without knowing the real cause, we could just add:

# Dave says Solaris' non-POSIX grep in the default path
# doesn't understand "-q" (which *is* POSIX):

if ! grep sagenb easy-install.pth >/dev/null; then
    echo 'No sagenb path found in easy-install.pth!'
    echo "Adding relative sagenb path to easy-install.pth"
    # Anyone recalling the correct syntax for *sed*?
    ed easy-install.pth >/dev/null <<EOF
$i
../../../../devel/sagenb
.
wq
EOF
fi

Opinions?

[nexttime]

comment:11

Ok, the ed command can be substituted by:

    sed -e '$ i \../../../../devel/sagenb' easy-install.pth > easy-install.$$
    mv -f easy-install.$$ easy-install.pth

[hivert]

comment:12

Maybe we have a race condition in editing that? (You could retry with ./sage -f ... when your build has finished.)

Could you also take a look at easy-install.pth's modification time after reinstalling the spkg?

Looks reasonable. It has been touched at the end of the build.

I tried a sage -f ./sage -f ../sagenb-0.8.7-patched.spkg and get no changes:

[...]
sing /home/data/Sage-Install/readline/sage-4.6.rc0/local/lib/python2.6/site-packages/Twisted-9.0.0-py2.6-linux-x86_64.egg
Searching for zope.interface==3.6.1
Best match: zope.interface 3.6.1
Processing zope.interface-3.6.1-py2.6-linux-x86_64.egg
zope.interface 3.6.1 is already the active version in easy-install.pth

Using /home/data/Sage-Install/readline/sage-4.6.rc0/local/lib/python2.6/site-packages/zope.interface-3.6.1-py2.6-linux-x86_64.egg
Searching for setuptools==0.6c9
Best match: setuptools 0.6c9
Processing setuptools-0.6c9-py2.6.egg
setuptools 0.6c9 is already the active version in easy-install.pth
Installing easy_install script to /home/florent/src/Sage/readline/sage-4.6.rc0/local/bin
Installing easy_install-2.6 script to /home/florent/src/Sage/readline/sage-4.6.rc0/local/bin

Using /home/data/Sage-Install/readline/sage-4.6.rc0/local/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg
Finished processing dependencies for sagenb==0.8.7-patched
Old path: ""
New path: ""

real    0m8.013s
user    0m5.983s
sys     0m2.016s
Successfully installed sagenb-0.8.7-patched
Now cleaning up tmp files.
Making Sage/Python scripts relocatable...
Making script relocatable
Finished installing sagenb-0.8.7-patched.spkg

I'm not sure what to do... If you have some time, maybe it could be easier to discuss on irc. I'm kind of busy tonight (it is 20:4 here) but I probably can find 1/2h if you need. I've no idea how this setup.py/easy-install stuff works and what could be the cause...

[nexttime]

comment:13

So the modification time did not change when running ./sage -f ...?

I'm not familiar with setuptools, easy-install etc. either... :(

I could perhaps upload a modified patch including the "brute force" fix later, but am busy now, too.

Dave just said he's going to install OpenSuSE 11.3 and / or 11.2 in a virtual machine; perhaps he can help, too, with debugging that.

[hivert]

comment:14

Replying to @nexttime:

So the modification time did not change when running ./sage -f ...?

On the contrary it did change as expected.

I'm not familiar with setuptools, easy-install etc. either... :(

I could perhaps upload a modified patch including the "brute force" fix later, but am busy now, too.

Ok !

Dave just said he's going to install OpenSuSE 11.3 and / or 11.2 in a virtual machine; perhaps he can help, too, with debugging that.

[nexttime]

comment:15

A last question for the moment:

How do you get the different paths /home/florent/src/... and /home/data/Sage-Install/...?

[hivert]

comment:16

Replying to @nexttime:

A last question for the moment:

How do you get the different paths /home/florent/src/... and /home/data/Sage-Install/...?

My install is in /home/data/Sage-Install with a symbolic link from /home/florent/src/.... I figured out that it could be the source of the problem. I though I have been careful not to go through the link but it seems I failed. However, on my openSuSE 11.1 I had the same failure without any links. So this is not the cause of the problem.

[nexttime]

Attachment: trac_10176-improve_spkg_dists_spkg_install_generation-spkg.patch.gz

SageNB spkg patch (0.8.7 -> 0.8.7.p1). Now includes an ugly work-around.

[nexttime]

Attachment: sagenb-0.8.7_vs._0.8.7.p1.diff.gz

Cumulative diff between SageNB 0.8.7 and 0.8.7.p1 - for reference / review only.

[nexttime]

Author: Leif Leonhardy

[nexttime]

comment:17

New spkg: http://spkg-upload.googlecode.com/files/sagenb-0.8.7.p1.spkg

md5sum: 6fe0e76e336fc24e71b5c591a112f8fb sagenb-0.8.7.p1.spkg

Florent, again: Please install / build, test and report! ;-)

This spkg now contains the work-around adding the relative sagenb path to easy-install.pth if (for whatever reason) there's no sagenb path at all in it...

I've also updated the spkg patch, and added a cumulative diff for reference / review (the Mercurial changeset contains two patches), but you can just download and install the new spkg.

(I actually made a mistake in the previously attached patch s.t. the #! in spkg-install ended up on the second line. I'm really curious what messages you will now get - either "Adding relative sagenb path to easy-install.pth" or "Making sagenb path in easy-install.pth relative"...)

I've again tested the spkg (only) on Ubuntu 10.04.

$ ./sage -f /path/to/sagenb-0.8.7.p1.spkg
$ ./sage -t -long -sagenb # Though there are no long tests IIRC

should be sufficient to test it. (Others should perhaps test it on different platforms as well.)

You could then do

$ egrep "Adding|Removing" spkg/logs/sagenb-0.8.7.p1.log
$ grep easy-install\.pth spkg/logs/sagenb-0.8.7.p1.log # more matches

(also with your previous logs, i.e. spkg/logs/sagenb-0.8.7.log or spkg/logs/sagenb-0.8.7-patched.log).

[qed777]

comment:18

Leif, thanks for making a new spkg with your improvements.

From spkg/standard/deps (cf. #8306):

# setuptools forgets to update easy-install.pth during parallel
# builds, so we build the relevant packages serially.

Perhaps forgets should be "forgets". Anyway, I'm mentioning this, in case anyone sees a connection with the behavior Florent sees.

[nexttime]

comment:59

Replying to @jdemeyer:

Replying to leif:

Just curious: Which 'sage' is supposed to be called here?
Should we check if SAGE_LOCAL or SAGE_ROOT is defined?
... and perhaps call $SAGE_ROOT/sage or $SAGE_LOCAL/bin/...?

I think it's not important which sage is called. It just needs sage -pkg, sage -hg and so on to do the Right Thing.

;-) Well, since SageNB moved from a more or less typical spkg to something more like the Sage library, I expect more people - potentially less familiar with the SageNB update process - to work on it.

So it's now IMHO more likely developers will have a broken (or outdated) Sage in their default path (i.e., outside a Sage subshell), in which case "arbitrary" things could happen.

If sage-env was sourced, i.e. some variables like SAGE_LOCAL or SAGE_ROOT are defined*, we can perhaps be more sure sage is the "proper" one, i.e. more recent.

---

• I prefer having SAGE_ENV_SOURCED anyway, to also avoid to (effectively) source it more than once. See Don't (effectively) source sage-env more than once #10469.

[nexttime]

comment:60

Replying to @jdemeyer:

Leif: my patch is the same as yours, expect that it's a hg patch and that it doesn't change the version number.

My Mercurial patch(es) contain a lot of commit messages with notes; the diff was not intended to get merged.

Replying to @jdemeyer:

Note that (as release manager) I don't care about the spkg, only about the diffs.

At the moment the whole purpose of the spkg is to aid debugging the problems mentioned on sage-devel; nobody opened a ticket I asked for, or replied on that thread yet.

In case the spkg helps, I suggest opening a new ticket with a description of the newly encountered problems, then "clean" Mercurial patches, and with just a back-reference to this ticket since the two patches from here will (most probably) be also included there.

[Error] messages given by my current spkg may lead to further work. I can't do that until I get some feedback and know what's really going wrong.

Testing upgrading with the current buildbot / scripts seems less useful; cf. sage-release.

[jdemeyer]

comment:62

Replying to @nexttime:

Replying to @jdemeyer:

Leif: my patch is the same as yours, expect that it's a hg patch and that it doesn't change the version number.

My Mercurial patch(es) contain a lot of commit messages with notes; the diff was not intended to get merged.

I know :-)

I just prefer qpushing a hg patch to applying a plain patch.

[jdemeyer]

comment:63

Leif, as far as I can tell, your patch fixes the issue. Could it be?

[nexttime]

comment:64

Replying to @jdemeyer:

Leif, as far as I can tell, your patch fixes the issue. Could it be?

Well, besides other things it removes the ln -snf which would not work as expected on Solaris, but you said you also failed to (re)install SageNB on sage.math previously...

[jdemeyer]

comment:65

Replying to @nexttime:

Replying to @jdemeyer:

Leif, as far as I can tell, your patch fixes the issue. Could it be?

Well, besides other things it removes the ln -snf which would not work as expected on Solaris, but you said you also failed to (re)install SageNB on sage.math previously...

True, but I can't reproduce this problem anymore with sage 4.6.1.rc0 (not officially released).

[jdemeyer]

comment:66

I have (finally) created ticket #10494 for the upgrade issue.

[jdemeyer]

comment:67

Patch copied to #10494.

[a-andre]

comment:72

Is this still an issue?

[jdemeyer]

Changed author from Leif Leonhardy to none

[jdemeyer]

Reviewer: Leif Leonhardy