# D4.5: Sage notebook / Jupyter notebook convergence #94

Closed
opened this Issue Sep 8, 2015 · 47 comments

Projects
None yet
7 participants
Contributor

### minrk commented Sep 8, 2015 • edited by nthiery Edited 1 time nthiery edited Sep 20, 2017 (most recent)

 WP4: User Interfaces Lead Institution: Université Paris-Sud Due: 2016-08-31 (month 12) Nature: Demonstrator Tasks: T4.1 (#69): Uniform notebook interface for all interactive components, T4.6 (#74) Structured documents Proposal: p. 48 Final report The Jupyter Notebook is a web application that enables the creation and sharing of executable documents which contain live code, equations, visualizations and explanatory text. Thanks to a modular design, Jupyter can be used with any computational system that provides a so-called Jupyter kernel implementing the Jupyter messaging protocol to communicate with the notebook. OpenDreamKit therefore promotes the Jupyter notebook as user interface of choice, in particular since it is particularly suitable for building modular web based Virtual Research Environments. A notebook interface is such a vital integrative component that, predating Jupyter, the open source mathematical system SageMath had developed as early as 2005 its own solution, which we refer to here as the legacy SageMath notebook. Development was fast tracked to ensure its availability to allow the project to move forward, and it served as major source of inspiration for Jupyter. Meanwhile, moving at a fast pace thanks to its much larger community, Jupyter had, by 2014, basically caught up with the legacy SageMath notebook in terms of functionality. Building on top of D4.4 (#93): Basic Jupyter interface for GAP, PARI/GP, SageMath, Singular, the goal of this deliverable was therefore to help phase out the legacy SageMath notebook in favour of Jupyter. Outsourcing this key but non disciplinary component is an important step toward the sustainability of ODK's ecosystem (Objective 5). Since 2014, a lot of work was put into this by the SageMath community, and in particular by Volker Braun. Recently, this work has been continued thanks to ODK. It included two aspects: ensuring that Jupyter included all important features of the legacy SageMath notebook, and enabling a smooth migration path for users: #20690, #22458: Live documentation @fcayre, @nthiery, @videlec Interacts/widgets: full support and backward compatibility with the legacy SageNB #21267 (closed installation issues: #21260 #21261 #20218 #21256) @jdemeyer #19877: Conversion of legacy notebooks to Jupyter notebooks @vbraun, @videlec, @marcinofulus #19740: Migration wizard, as a web application Future work: WYSIWYG editor for the markdown cells, such as TinyMCE (see this post for motivation) #20316: Add button to export SageNB notebooks to Jupyter "Search help" box Custom menus (e.g. for setting options like %display latex, automatic vars, enable/disable preparser; possibly menus to insert typical commands as in SMC)

Contributor

### jdemeyer commented Oct 27, 2015 • edited by slel Edited 1 time slel edited Jul 26, 2016 (most recent)

 See http://trac.sagemath.org/ticket/19128 Upgrade to IPython 4.0.0 + Jupyter http://trac.sagemath.org/ticket/19135 Restore Sage interrupt handler in Jupyter http://trac.sagemath.org/ticket/19371 Install Jupyter kernel spec and nbextensions in \$SAGE_LOCAL http://trac.sagemath.org/ticket/19374 LaTeX display broken in Jupyter output cells http://trac.sagemath.org/ticket/19469 Jupyter mathjax fails for notebooks in subdirectories jupyter/notebook#663 Interpret mathjax_url relative to base_url

Open

Open

Contributor

Contributor

### jdemeyer commented May 10, 2016

 Just adding some links... Use numbers abstract base class: jupyter-widgets/ipywidgets#236 Register Sage classes in numbers abc: http://trac.sagemath.org/ticket/19571 Interact API support for selection widgets: jupyter-widgets/ipywidgets#288
Contributor

### nthiery commented Jul 26, 2016 • edited by slel Edited 1 time slel edited Jul 26, 2016 (most recent)

 About interacts: I had a chat with @SylvainCorlay (main author of ipywidgets; aka Jupyter's interacts) over the phone yesterday: he is now based in Paris and happy to give a hand; he probably will visit one day for Sage Days 74 in Cernay; following discussions with @jdemeyer at Sage Days 70 in Berkeley, the new ipywidgets version should be close to backward compatible with Sage's interacts in the legacy notebook. @marcinofulus has a huge collection of legacy sage notebooks filled with interacts. Marcin is eager to convert them to Jupyter notebooks. So we have a large test set and a motivated beta tester. @jdemeyer: maybe you want to open a separate ticket/issue to trac progress on this specific point.

### SylvainCorlay commented Jul 26, 2016

 @nthiery thanks for the ping. (I am not "the main author" of ipywidgets though, jdfreder worked a lot on it before he left jupyter).
Contributor

### jdemeyer commented Aug 17, 2016

 I started working with interacts at https://trac.sagemath.org/ticket/21267

Closed

Merged

Contributor

### bpilorget commented Aug 30, 2016

 Is the report close to an end?
Contributor

### minrk commented Aug 30, 2016

 @nthiery Paris-Sud is the leader on this. What is the state of the report? Let me know what I can do to help.
Contributor

### bpilorget commented Aug 30, 2016

 The README gives info on how to write deliverables (scroll down): https://github.com/OpenDreamKit/OpenDreamKit
Contributor

### nthiery commented Aug 31, 2016

 Ah right, that's a good point @minrk. Jeroen is coming to Orsay next Monday. I'll write the report with him then, as I need to touch base with him to check on the status of a couple things. It should be short anyway. I'll update the PO that we will be late by a couple days.
Contributor

### jdemeyer commented Aug 31, 2016 • edited by minrk Edited 1 time minrk edited Feb 27, 2017 (most recent)

 I have continued to work on the interact convergence. It's not done, but I would like somebody to review the following first: It would be great if this could be done before or while I'm in Orsay next week.

Closed

Contributor

### nthiery commented Sep 11, 2016

 Hi Jeroen, How did the discussion go with Sylvain? What timeline do you foresee at this stage for interacts?

Merged

Contributor

### jdemeyer commented Sep 12, 2016

 Hi Jeroen, How did the discussion go with Sylvain? I would say that the meeting went quite well. My main request for upstream ipywidgets was to add the needed hooks that we can override in Sage. I am currently waiting for review of jupyter-widgets/ipywidgets#725 and jupyter-widgets/ipywidgets#762 What timeline do you foresee at this stage for interacts? Most interacts already work with the work that I have done, so I consider it almost done. Maybe I need a few more days.

### SylvainCorlay commented Sep 12, 2016

 Same here. I imagine the features could be code-complete and merged quite soon. The release of a version of ipywidgets including those changes is a different matter. There are a few things that we need to fix before 6.0. We are going to sprint on ipywidgets integration into jupyterlab next week with Jason Grout and Brian Granger among other subjects.
Contributor

### jdemeyer commented Oct 21, 2016

 Any idea on the release date of ipywidgets 6?

Closed

### SylvainCorlay commented Oct 21, 2016

 @jasongrout released one of the main blockers, so we should be able to sprint to have a proper release now. The remaining item is a usability / css restyling...
Contributor

### dimpase commented Oct 24, 2016 • edited Edited 1 time dimpase edited Oct 24, 2016 (most recent)

 There is a lot of demand (say, on sage-support list) for a guide to use jupyterhub in a multiuser setting where users are not authenticated via dedicated Unix accounts, something akin to the SageNB authentication model (this is a setting most convenient for teaching a class using SageMath). How to set this up, and whether this is possible at all, is not clear ATM. See e.g. here.
Contributor

# Deliverables update

@minrk As you know , during the interim review in Bremen, reviewers gave advice to improve deliverables. UPSud tried to follow their advice by updating D5.1 #107
The main critic is that deliverables should be static document. Therefore we went through the #107 report to check if all links were still working and also if the information contained on the links was necessary for the understanding of the report.
As a result when it was necessary annexes were added to the #107 report.

Can you please do the same for this deliverable and add annexes when need be?

Contributor

### minrk commented Oct 24, 2016

 Will do, thanks.

Closed

Closed

Contributor

### minrk commented Nov 24, 2016

 @nthiery I don't see a report on the status of this deliverable. Do you have one?

### jasongrout commented Nov 24, 2016

 Relevant recent discussion: https://groups.google.com/forum/?fromgroups#!topic/sage-devel/t11JSxxCgpw (see the end where it talks about Jupyter/SageNB convergence), and https://trac.sagemath.org/ticket/21267
Contributor

### nthiery commented Nov 24, 2016

 On Thu, Nov 24, 2016 at 11:45:59AM -0800, Min RK wrote: ***@***.*** I don't see a report on the status of this deliverable. Do you have one? Not yet. That's indeed becoming urgent. I'll try to work on it next week, but help is welcome. Cheers,

Closed

Contributor

### jdemeyer commented Jan 6, 2017

 Relevant Sage ticket which needs review: https://trac.sagemath.org/ticket/22125
Contributor

### jdemeyer commented Jan 19, 2017

 I just volunteered to lead this deliverable.
Contributor

### jdemeyer commented Feb 1, 2017

 Volker made me a collaborator of sagenb_export (https://github.com/vbraun/ExportSageNB). I am testing this on real-world examples and fixing various issues.
Contributor

### nthiery commented Feb 6, 2017

 Dear M12 deliverable leaders, Just a reminder that reports are due for mid-february, to buy us some time for proofreading, feedback, and final submission before February 28th. See our README for details on the process. In practice, I'll be offline February 12-19, and the week right after will be pretty busy. Therefore, it would be helpful if a first draft could be available sometime this week, so that I can have a head start reviewing it. Thanks in advance!
Contributor

### minrk commented Feb 9, 2017

 Let me know if there's anything I can do to help get this report finished on time.
Contributor

### jdemeyer commented Feb 11, 2017

 I'll have a look at this on monday.
Contributor

### nthiery commented Feb 27, 2017

 Hi @jdemeyer! I guess that's the next thing on your plate, right? Do you need help for how to best finalize the report, and describe where we stand?

### SylvainCorlay commented Feb 27, 2017

 For those who may have missed it, we have published a release candidate for ipywidgets 6 which includes the work of @jdemeyer on the convergence with Sage.
Contributor

### jdemeyer commented Feb 27, 2017

 I pushed a first version of the report. I doesn't mention yet the live documentation, which is unfortunately broken (due to a missing XSRF cookie): https://trac.sagemath.org/ticket/22458
Contributor

### jdemeyer commented Feb 27, 2017

 For those who may have missed it, we have published a release candidate for ipywidgets 6 which includes the work of @jdemeyer on the convergence with Sage. As far as I can tell: 1 other person besides me tried it and it didn't work, possibly because of [IPKernelApp] WARNING | The installed widget Javascript is the wrong version. It must satisfy the semver range ~2.1.0. 

### SylvainCorlay commented Feb 27, 2017 • edited Edited 1 time SylvainCorlay edited Feb 27, 2017 (most recent)

 This almost certainly due to the fact that he has another version of the js installed from a previous installation. I have written a Jupyter Enhancement proposal on how to improve the extension mechanism to make this sort of confusing situation impossible. jupyter/enhancement-proposals#21
Contributor

### jdemeyer commented Feb 27, 2017

 This almost certainly due to the fact that he has another version of the js installed from a previous installation. Shouldn't an installation of a newer version overwrite the old version?

### SylvainCorlay commented Feb 27, 2017

 This almost certainly due to the fact that he has another version of the js installed from a previous installation. Shouldn't an installation of a newer version overwrite the old version? It is actually a bit more subtle than that. widgetsnbextension, which holds the js, is installed in the notebook prefix. ipywidgets has no static assets and is installed in the kernel prefix the static assets of the notebook extension can be installed / enabled / disabled at different levels of precedence (system, user, sys-prefix)... The current semantics for notebook extension is bad and lies outside of the ipywidgets project. The above JEP is the proposal for better semantics in this area.
Contributor

### minrk commented Feb 27, 2017

 I've been working on the Thebe issue, and it seems relatively difficult to resolve. Thebe ships with a copy of old notebook javascript, which doesn't set the xsrf token that Jupyter requires to protect against cross-site request forgeries. I tried a simple update in Thebe, but it hasn't worked as I hoped. The simplest (insecure) fix is to disable xsrf checking, e.g. by launching a notebook server with: c.NotebookApp.disable_check_xsrf = True Both JupyterLab and nteract now provide javascript libraries that can be used to make writing a new version of Thebe much simpler and more maintainable. This isn't within scope for today, though.
Contributor

### jdemeyer commented Feb 28, 2017

 I just pushed a new version of the report, adding live documentation. I didn't mention that it actually doesn't work currently.
Contributor

### jdemeyer commented Feb 28, 2017

 @fcayre, @nthiery: I added you as authors too, so maybe you want to read the report? For me, it can be submitted.
Contributor

### nthiery commented Feb 28, 2017

 I am finishing up submitting D3.2, and I'll hop on this one. Thanks for the notice!
Contributor

### nthiery commented Feb 28, 2017

 I fleshed up a bit the github issue description, did some minor reordering in the report for consistency with the feature order in the description, and enlarged the figures in landscape mode for more readability. I committed the final pdf. I'll submit in half an hour unless someone requests some change!
Contributor

### jdemeyer commented Feb 28, 2017 • edited Edited 1 time jdemeyer edited Feb 28, 2017 (most recent)

 Fixed typo: open souce :-) (open source sauce?)
Contributor

### jdemeyer commented Feb 28, 2017

 There is some inconsistency between SageMath, Sage and \Sage. In the github description, I changed to using SageMath consistently but I have not changed the report.
Contributor

### nthiery commented Feb 28, 2017

 Ok. I'll update the report. Anything else, or should I just submit when done?
Contributor

### jdemeyer commented Feb 28, 2017

 I have no further comments. So you can submit if you want.
Contributor

### nthiery commented Feb 28, 2017

 Submitted! Thanks Jeroen for leading this deliverable and its reporting, and for all the hard work on interacts, migration wizard, and the zillion other little details that make the difference for our users. Thanks @vbraun too for pushing so hard on the Jupyter Sage kernel! And @SylvainCorlay, @fcayre, @dimpase, @videlec, @seblabbe, Thierry, and many others for the help on the Jupyter migration. It is a major progress which we have all been looking forward!

Contributor

### nthiery commented Mar 13, 2017

 Hi @minrk, I am sitting with @jdemeyer, and we are discussing the Thebe issue. Thanks for investigating it! We now need to decide on whether to disable xsrf checks for the local sage-jupyter notebook, as a temporary workaround. If we had to guess when there will be a new Thebe based on JupyterLab, what would be your guesstimate? How bad a security hole would this be in you opinion? Is this something we could afford temporarilly? Cheers,
Contributor

### minrk commented Mar 14, 2017

 If we had to guess when there will be a new Thebe based on JupyterLab, what would be your guesstimate? I think we can have it in the next few months. I can work on this. How bad a security hole would this be in you opinion? Is this something we could afford temporarily? The known issue is relatively small, discussed here. Only some browsers are affected, and only limited actions can be taken.

### vbraun added a commit to vbraun/sage that referenced this issue Mar 14, 2017

 Trac #22458: Temporarily disable Jupyter XSRF check in local notebook… 
…s to fix live documentation in Thebe

When running a local Jupyter notebook, the live documentation does not
work at all. The notebook log reports
{{{
[...]
[W 12:55:00.384 NotebookApp] 403 POST /api/kernels (::1): '_xsrf'
argument missing from POST
[W 12:55:00.385 NotebookApp] 403 POST /api/kernels (::1) 2.54ms referer=
http://localhost:8888/kernelspecs/sagemath/doc/prep/Quickstarts/Interact
.html
}}}

One workaround, [OpenDreamKit/OpenDreamKit#94#
issuecomment-286408387 suggested by Min R. K.] is to disable the XSRF
security check in Jupyter.

This is meant as a temporary measure until Thebe will have been
refactored on top of JupyterLab (tentatively a couple months???).

'''Upstream report''': oreillymedia/thebe#93

URL: https://trac.sagemath.org/22458
Reported by: jdemeyer
Ticket author(s): Jeroen Demeyer
Reviewer(s): Julian Rüth
 6f35f28 

Closed