Install Jupyter kernel spec and nbextensions in $SAGE_LOCAL #19371

jdemeyer · 2015-10-08T07:32:46Z

Instead of installing the Jupyter files in the user's home directory, we should install it in $SAGE_LOCAL (a "system" install). This has two main advantages:

it avoids conflicts in case multiple installations of Sage exist.
the Sage Jupyter notebook can be used more easily outside of Sage (e.g. using Jupyterhub)

We should not use versioned identifiers. For example, all these actually refer to the same Sage installation:

$ ls -l ~/.local/share/jupyter/kernels/
total 12
drwx------ 2 jdemeyer jdemeyer 4096 Sep 24 18:05 sage_6_9_beta7
drwx------ 2 jdemeyer jdemeyer 4096 Oct  7 22:41 sage_6_9_rc0
drwx------ 2 jdemeyer jdemeyer 4096 Oct  7 22:04 sage_6_9_rc3

I'm also replacing IPython -> Jupyter and Sage -> SageMath

Depends on #19373

Component: interfaces

Author: Jeroen Demeyer

Branch/Commit: 4f06d3d

Reviewer: Emmanuel Charpentier

Issue created by migration from https://trac.sagemath.org/ticket/19371

The text was updated successfully, but these errors were encountered:

jdemeyer · 2015-10-08T08:35:19Z

Branch: u/jdemeyer/install_jupyter_kernel_spec_in__sage_local

jdemeyer · 2015-10-08T09:43:55Z

New commits:

`7c916ef`	`Install Jupyter kernel spec in $SAGE_LOCAL`

jdemeyer · 2015-10-08T09:43:55Z

Commit: 7c916ef

egourgoulhon · 2015-10-08T13:45:25Z

comment:4

Looks good to me.
But it would be nice if someone more expert on Jupyter gives a look too.

egourgoulhon · 2015-10-08T13:45:25Z

Reviewer: Eric Gourgoulhon

kiwifb · 2015-10-08T22:33:43Z

comment:6

You are concerned with multiple SageMath installs I am more concerned with global install for multiple users (SMC and sage-on-gentoo) and I am not sure about that change. It gives me problem on install at the moment because it violates sandbox rules of packaging. It's more the fault of jupyter_core for not giving me a way of over-ridding ENV_JUPYTER_PATH than yours.

But I am more worried about the fact that on a global install people won't be able to touch that directory. If you don't need to put anything else in it at "user runtime" great. If not we are in a big no situation as far as I am concerned (and William will almost certainly not be happy either).

As for running multiple SageMath install side by side without bad interactions from the content of ~/.sage you can always define DOTSAGE. A bit inconvenient (unless you version it inside of sage) but it works.

jdemeyer · 2015-10-09T07:36:04Z

comment:7

Replying to @kiwifb:

It gives me problem on install at the moment because it violates sandbox rules of packaging.

I think you need to explain this more. The only possible issue I can see is that maybe the directory ENV_JUPYTER_PATH[0] is the "wrong" place in your Sage-on-Gentoo setup.

If ENV_JUPYTER_PATH[0] is some directory inside $SAGE_LOCAL (which it should be), I don't see why installing this one particular file in $SAGE_LOCAL is any different from installing the huge number of other files in $SAGE_LOCAL...

But I am more worried about the fact that on a global install people won't be able to touch that directory. If you don't need to put anything else in it at "user runtime" great.

We do not need to put anything else in that directory at runtime. Why do you think we do?

kiwifb · 2015-10-09T07:55:59Z

comment:8

Replying to @jdemeyer:

Replying to @kiwifb:

It gives me problem on install at the moment because it violates sandbox rules of packaging.

I think you need to explain this more. The only possible issue I can see is that maybe the directory ENV_JUPYTER_PATH[0] is the "wrong" place in your Sage-on-Gentoo setup.

ENV_JUPYTER_PATH is defined as os.path.join(sys.prefix, 'share', 'jupyter') which translates as /usr/share/jupyter (or $SAGE_LOCAL/share/jupyter) in concrete terms. The problem is the sage installation procedure is now trying to create and write files out of the sandbox.

Proper packaging in the sense of rpm, deb and most other doesn't allow you to write in the "live" system. Things are installed and written in a sandbox space and then copied (merged) to the system. Because of the way Andrew wrote env.py (and I was grateful to him for that) I can set most SAGE_* variables so sage will write things in my sandbox. If jupyter_core was allowing me to over-ride ENV_JUPYTER_PATH I could set in the appropriate place in the sandbox and it would be merged in the right place.

If ENV_JUPYTER_PATH[0] is some directory inside $SAGE_LOCAL (which it should be), I don't see why installing this one particular file in $SAGE_LOCAL is any different from installing the huge number of other files in $SAGE_LOCAL...

sys.prefix is SAGE_LOCAL, actually even in sage-on-gentoo, the difference is that you don't protect your live prefix during build.

But I am more worried about the fact that on a global install people won't be able to touch that directory. If you don't need to put anything else in it at "user runtime" great.

We do not need to put anything else in that directory at runtime. Why do you think we do?

Because I am uncertain of how it works. You are obviously creating something there as place holder but it was possibly silly of me.

jdemeyer · 2015-10-09T07:59:45Z

comment:9

Replying to @kiwifb:

sys.prefix is SAGE_LOCAL, actually even in sage-on-gentoo

So, if sys.prefix is $SAGE_LOCAL and you can set $SAGE_LOCAL to whatever you want, then I don't see the problem.

the difference is that you don't protect your live prefix during build.

I have absolutely no idea what this sentence means.

jdemeyer · 2015-10-09T08:01:41Z

comment:10

Replying to @kiwifb:

Because I am uncertain of how it works.

It's really just one configuration file. That's the whole thing I don't understand: what makes this one particular file so special? You apparently manage to install all of Sage, but not this one particular file?

kiwifb · 2015-10-09T08:18:37Z

comment:11

Two things:

the fact that sys.prefix is SAGE_LOCAL is incidental, it doesn't have to be.
Once jupyter_core is in place it doesn't matter what I set SAGE_LOCAL to, it doesn't change the value of sys.prefix jupyter_core or rather python is reporting.

The install procedure for sage is now trying to create /usr/share/jupyter/kernels/sagemath and any missing directory. rpm, deb and ebuild do not allow you to do that directly. You install $SOMESANDBOX/usr/share/jupyter/kernels/sagemath and later you merge into the live system. That's what DESTDIR is for in autotools, you set DESTDIR=$SOMESANDBOX and things magically install in the sandbox.

Now my main concern was the potential for user to do something to {/usr,$SAGE_LOCAL}/share/jupyter/kernels/sagemath which would be problematic for multi-user install. It is unfounded. The rest is an upstream problem and we can stop debating packaging tech in this ticket.

jdemeyer · 2015-10-09T08:22:41Z

comment:12

Replying to @kiwifb:

You install $SOMESANDBOX/usr/share/jupyter/kernels/sagemath and later you merge into the live system. That's what DESTDIR is for in autotools, you set DESTDIR=$SOMESANDBOX and things magically install in the sandbox.

Right, and how does this work with distutils? I assume (but I might be wrong) that distutils also uses sys.prefix to determine where to install stuff.

jdemeyer · 2015-10-09T08:26:00Z

comment:13

Replying to @kiwifb:

The rest is an upstream problem and we can stop debating packaging tech in this ticket.

It happens that I am debating packaging tech with upstream, so I'd like to understand your issue.

kiwifb · 2015-10-09T08:38:40Z

comment:14

Replying to @jdemeyer:

Replying to @kiwifb:

The rest is an upstream problem and we can stop debating packaging tech in this ticket.

It happens that I am debating packaging tech with upstream, so I'd like to understand your issue.

I have just read it ( before seeing your comment here). It is a bit messy but thinking more about it there must a bug of some kind. distutils as used from within portage should set a sandboxed prefix, therefore sys.prefix should point to a sandboxed location, so something is wrong there.

kiwifb · 2015-10-09T09:48:38Z

comment:15

I know what the problem is. We have three phases in order: build, test (optional) and install. Only the install phase is run with a prefix, this is the only time when it should be necessary. But the installation of the jupyter kernel is attempted during the build phase. So from my point of view that bit has to be restricted to install time.

sagetrac-git · 2015-10-09T12:56:10Z

Branch pushed to git repo; I updated commit sha1. New commits:

`2cc3a74`	`Fix dependencies; install kernel spec with custom install class`

sagetrac-git · 2015-10-09T12:56:10Z

Changed commit from 7c916ef to 2cc3a74

kiwifb · 2015-10-09T20:26:03Z

comment:21

Thank you for taking my concern on board.

jdemeyer · 2015-10-11T09:57:41Z

Dependencies: #19371

sagetrac-git · 2015-10-11T10:01:47Z

Changed commit from 2cc3a74 to 6c7261c

sagetrac-git · 2015-10-11T10:01:47Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`52a717a`	`Use relative links to local help`
`6c7261c`	`Install Jupyter kernel spec in $SAGE_LOCAL`

jdemeyer · 2015-10-11T10:02:06Z

Changed dependencies from #19371 to #19373

sagetrac-git · 2015-10-11T10:29:50Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`4f06d3d`	`Install Jupyter kernel and nbextensions in $SAGE_LOCAL`

sagetrac-git · 2015-10-11T10:29:50Z

Changed commit from 6c7261c to 4f06d3d

egourgoulhon · 2015-10-11T16:57:44Z

Changed reviewer from Eric Gourgoulhon to none

egourgoulhon · 2015-10-11T16:57:44Z

comment:28

Sorry I'm afraid I cannot review this one (don't feel competent about it).

EmmanuelCharpentier · 2015-10-15T14:36:03Z

comment:29

I had a problem on an Ubuntu machine wher I could get MathJax in sheets created in my home directory but nowhere else.

I checked the present patch and recompiled :

It seems to solve my Ubuntu-specific problem
It passes ptestlong.

Is that enough for a positive review ?

jdemeyer · 2015-10-15T14:41:20Z

comment:30

Replying to @EmmanuelCharpentier:

Is that enough for a positive review ?

Who are you asking? Essentially, that's up to you to decide.

jdemeyer · 2015-10-15T14:42:57Z

comment:31

Note that this depends on #19373. So, it would be nice if you could also review that.

EmmanuelCharpentier · 2015-10-15T14:59:57Z

comment:32

Replying to @jdemeyer:

Note that this depends on #19373. So, it would be nice if you could also review that.

Is this dependance somehow managed by git ?

In other terms, when I checked #19371 out, did that also applied #19373 ?

If so, that means that my tests show that (#19371+#19373) passes ptestlong. I cannot speak for a Jupyter hub, since I don't have such a beast in my zoo...

Otherwise, I'll have to check out #19373 and merge both branches, then recompile and ptestlong (about 4 hours on my Ubuntu netbook...).

Or would it be enough to positively review #19371, and leave #19373 to someone equipped to do so ?

jdemeyer · 2015-10-15T15:05:01Z

comment:33

Replying to @EmmanuelCharpentier:

In other terms, when I checked #19371 out, did that also applied #19373 ?

Yes, this branch here is based on #19373.

If so, that means that my tests show that (#19371+#19373) passes ptestlong.

Indeed.

EmmanuelCharpentier · 2015-10-15T15:23:00Z

Reviewer: Emmanuel Charpentier

EmmanuelCharpentier · 2015-10-15T15:23:00Z

comment:34

Replying to @jdemeyer:

Replying to @EmmanuelCharpentier:

In other terms, when I checked #19371 out, did that also applied #19373 ?

Yes, this branch here is based on #19373.

If so, that means that my tests show that (#19371+#19373) passes ptestlong.

Indeed.

OK. I give a positive review to #19371. I'll leave to someone having the possibility to really test #19373 the responsibility of validating it.

vbraun · 2015-10-18T19:11:25Z

Changed branch from u/jdemeyer/install_jupyter_kernel_spec_in__sage_local to 4f06d3d

jdemeyer added this to the sage-6.9 milestone Oct 8, 2015

jdemeyer added c: interfaces labels Oct 8, 2015

This comment has been minimized.

Sign in to view

jdemeyer added the s: needs review label Oct 8, 2015

egourgoulhon added s: positive review and removed s: needs review labels Oct 8, 2015

jdemeyer added p: blocker / 1 and removed p: major / 3 labels Oct 8, 2015

jdemeyer added s: needs work and removed s: positive review labels Oct 9, 2015

jdemeyer removed the s: needs work label Oct 9, 2015

jdemeyer added the p: major / 3 label Oct 9, 2015

jdemeyer modified the milestones: sage-6.9, sage-6.10 Oct 9, 2015

This comment has been minimized.

Sign in to view

jdemeyer changed the title ~~Install Jupyter kernel spec in $SAGE_LOCAL~~ Install Jupyter kernel spec and nbextensions in $SAGE_LOCAL Oct 11, 2015

This comment has been minimized.

Sign in to view

jdemeyer added s: needs review and removed s: needs work labels Oct 11, 2015

EmmanuelCharpentier mannequin added s: positive review and removed s: needs review labels Oct 15, 2015

vbraun removed the s: positive review label Oct 18, 2015

vbraun closed this as completed in 9f84c89 Oct 18, 2015

cheuberg mentioned this issue Aug 19, 2020

Jupyter dies on incorrect kernel specification #20121

Closed

Install Jupyter kernel spec and nbextensions in $SAGE_LOCAL #19371

Install Jupyter kernel spec and nbextensions in $SAGE_LOCAL #19371

Comments

jdemeyer commented Oct 8, 2015

This comment has been minimized.

jdemeyer commented Oct 8, 2015

jdemeyer commented Oct 8, 2015

jdemeyer commented Oct 8, 2015

egourgoulhon commented Oct 8, 2015

egourgoulhon commented Oct 8, 2015

kiwifb commented Oct 8, 2015

jdemeyer commented Oct 9, 2015

kiwifb commented Oct 9, 2015

jdemeyer commented Oct 9, 2015

jdemeyer commented Oct 9, 2015

kiwifb commented Oct 9, 2015

jdemeyer commented Oct 9, 2015

jdemeyer commented Oct 9, 2015

kiwifb commented Oct 9, 2015

kiwifb commented Oct 9, 2015

sagetrac-git mannequin commented Oct 9, 2015

sagetrac-git mannequin commented Oct 9, 2015

kiwifb commented Oct 9, 2015

jdemeyer commented Oct 11, 2015

sagetrac-git mannequin commented Oct 11, 2015

sagetrac-git mannequin commented Oct 11, 2015

jdemeyer commented Oct 11, 2015

This comment has been minimized.

sagetrac-git mannequin commented Oct 11, 2015

sagetrac-git mannequin commented Oct 11, 2015

This comment has been minimized.

egourgoulhon commented Oct 11, 2015

egourgoulhon commented Oct 11, 2015

EmmanuelCharpentier mannequin commented Oct 15, 2015

jdemeyer commented Oct 15, 2015

jdemeyer commented Oct 15, 2015

EmmanuelCharpentier mannequin commented Oct 15, 2015

jdemeyer commented Oct 15, 2015

EmmanuelCharpentier mannequin commented Oct 15, 2015

EmmanuelCharpentier mannequin commented Oct 15, 2015

vbraun commented Oct 18, 2015