User-configurable JupyterHub / BinderHub launch buttons

[choldgraf]

Description / Summary

We currently hard-code the JupyterHub/BinderHub that a book points to via the author's configuration. However, some readers may wish to bring their own JupyterHubs with them, particularly for content that is meant to be generically useful.

In those cases, it would be useful if they could manually specify a JupyterHub at the client side. For example, they could type in the Hub URL as part of the "launch buttons" dropdown. This would then generate a "launch button" for their hub, and persist between pages.

Value / benefit

This would encourage the (re)use of material across different JupyterHubs, which is an interesting future use-case to consider. It is unclear if this is a particularly common need now, but may become more common as more organizations use JupyterHubs for their work.

Implementation details

This would probably need to an improvement somewhere around here:

https://github.com/executablebooks/sphinx-book-theme/blob/43506962d4f90e6fc74d209d72503929900d84e7/sphinx_book_theme/topbar/launchbuttons.html#L3-L32

and might depend on changes that are being proposed in #1253

Tasks to complete

• Iterate and discuss the best way to do this
• ...fill in as we understand more

[jlperla]

@choldgraf I would check with @mmcky and @AakashGfude to see how they did this for the QE lectures and the associated devops there (e.g. the notebook repo needs to be configured, etc.)

But I don't see how this can work prior to getting jupyter support in myst? Or would this still only launch things that are already ipnb notebooks?

[choldgraf]

But I don't see how this can work prior to getting jupyter support in myst? Or would this still only launch things that are already ipnb notebooks?

Yep - either the source files would need to be ipynb, or the hub a user was adding themselves would need to know how to read .md files as Jupyter Notebooks. So I think the question is getting MyST notebooks support in Jupyter, and documenting how hubs could configure themselves to use MyST notebooks (e.g. via jupytext, or one day via a direct integration with JupyterLab)

[jlperla]

or the hub a user was adding themselves would need to know how to read .md files as Jupyter Notebooks
one day via a direct integration with JupyterLab)

I think those are both the wrong approach, even in the longterm. I (and any student I can imagine) want ipynb as an output format. Notebooks are a great output, and I don't want the underlying markup for a book executed by the students any more than I would want them to compile the .tex file for the PDFs themselves.

This is most important for making these books accessible and simple for people who have never programmed before. I wouldn't want someone who has never programmed before to see a myst file with a bunch of complicated metadata for publication any more than I would want them to see a bunch of scary .tex file and macros if I just want to teach them some theory.

documenting how hubs could configure themselves to use MyST notebooks

I think that is also the wrong approach. Not everything is a hub... end users download these things onto their desktops, google colab, vscode juptyer, nteract, etc. Even if you got support for myst on all of them, I still say it is the wrong thing for an introductory user to be looking at. (and I wouldn't want jupytext running on the hub any more than I woudl want the student to compile their own PDF from a tex file on the hub).

Yep - either the source files would need to be ipynb, or the hub a user was adding themselves would need to know how to read .md files as Jupyter Notebooks.

Enough of my complaining, how about a workable solution: What if you just let the content developers choose their notebook repository URL and then it is up to them to have the notebooks in the correct folder structure/etc. (and as ipynb so nbgitpuller/etc all work). They are responsible for that conversion and worflow.

That is, there are existing workflows with jupytext that try to hack around this issue (e.g. https://github.com/QuantEcon/lecture-python.myst/blob/main/.github/workflows/publish.yml and many others) maybe there is an intermediate step here to get us closer to a world with jupyter support from all myst.

What if the notebook repository URL is made more formal and integrated into jupyterbook instead of hacked on top of it with a theme? @mmcky can explain how they did it better, but

• https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/_config.yml#L52 has a configuration that could become part of jupyterbooks setup directly instead of a theme option.
• That is, an optional setting in the config here: https://jupyterbook.org/interactive/launchbuttons.html#common-launch-button-configuration
• Then the binder, jupyterub, etc. would all use this prefix themselves if available to generate links rather than the source repository (e.g. https://jupyterbook.org/interactive/launchbuttons.html#add-a-launch-on-binder-button and https://jupyterbook.org/interactive/launchbuttons.html#add-a-launch-on-jupyterhub-button )
• Similarly, the default download notebook button could then use that prefix rather than the one associated with the source repository.

If I understand this correctly, this feature would eliminate the hacked up themes which re-implements these core executable book features which are not currently supported in myst - but wouldn't eliminate the custom devops and delivery required to generate and publish the notebooks to a repo. I think that is a fine place to start.

[mmcky]

I think those are both the wrong approach, even in the longterm. I (and any student I can imagine) want ipynb as an output format.

either the source files would need to be ipynb, or the hub a user was adding themselves would need to know how to read .md files as Jupyter Notebooks.

I think the crux of the issue here is you want a set of notebooks (ipynb) that you can be used for students that don't contain author specific markup that jupyter-book uses for formating etc. Essentially you're differentiating between:

1. source notebooks / myst:md files (which have a 1-to-1 mapping)
2. published notebooks (that are adjusted for use by students / users)

Some elements for published notebooks are items like:

1. removed solutions to exercises
2. removed publishing markup such as unittests or directives and roles that are rendered
3. updated links so notebooks point back to website for images and figures
4. ...

Some of this is coming with javascript plugins for jupyter notebook that will render the myst markup in the notebook -- but I do think that there are certainly cases that do require a set of publish notebooks. As you know -- we have a similar workflow currently on quantecon projects using sphinx_tojupyter but that is a sub-optimal approach (technically) given it builds notebooks from the sphinx.ast which is ultimately a lossy process and needs ongoing maintenance as sphinx changes over time.

how about a workable solution

Thanks for this suggestion. I guess this would essentially boil down to adding notebook_repository option to the config that can be used generally by sphinx/themes etc:

# Information about where the book exists on the web
repository:
  url                       : https://github.com/yourusername/yourbookrepo  # Online location of your book
  path_to_book              : path/to/book  # Optional path to your book, relative to the repository root
  branch                    : master  # Which branch of the repository should be used when creating links (optional)

# Information about where companion notebooks exist on the web
notebook_repository:
  url                       : https://github.com/yourusername/younotebookskrepo  # Online location of your book
  path_to_notebooks             : path/to/notebooks  # Optional path to your notebooks, relative to the repository root
  branch                    : master  # Which branch of the repository should be used when creating links (optional)

and then setting up standard for any html theme that wants to support notebook execution to check this general configuration.

---

Another Option (jupyter-book as a coordinator for multiple asset builds):

At ANU we have been discussing another option which would be to use jupyter-book as a coordinator in the case of organising multiple output / generated assets (such as html, pdf (via LaTeX), and ipynb sets) by introducing a website builder. This would then integrate tightly with a set of standard for html themes for showing assets like:

1. pdf downloads (via LaTeX or HTML/Javascript)
2. ipynb downloads / execution buttons

jb build ./ --builder=website

this would then take a lot of the configuration complexity away by enabling high level options such as:

website:
  pdflatex: true/false  #build pdf via LaTeX

then jupyter-book would coordinate (based on set rules) locations for assets that themes would need to conform to as a standard.

So for the example above:

1. jupyter-book would request a pdflatex build and copy _build/latex/book.pdf outputs to _build/website/_pdf
2. jupyter-book would request an html build with supporting assets in set locations that themes support to show the buttons etc.

For the case with launch_buttons we could then focus on the service layer config

Any supporting notebooks can be produced by jupyter-book and options could include various technology routes for producing the ipynb as an output:

1. copy the source files (some authors may want this)
2. use jupytex + some filtering layer for lightweight manipulation of notebooks to produce ipynb
3. use sphinx_tojupyter (or some other extension to build a notebook)
4. ...

[jlperla]

Thanks @mmcky Great summary.

source notebooks / myst:md files (which have a 1-to-1 mapping)

Yes. To me, that 1 to 1 is not useful since I don't see why I would ever want to write in ipynb/jupyterlab, but I understand others prefer jupyter notebooks as an editing environment and file format. But I think those have essentially nothing to do with with the use of notebooks by students (because even if they aren't myst, they would end up with a bunch of author-centric metadata and stuff releated to the themes).

Some of this is coming with javascript plugins for jupyter notebook that will render the myst markup in the notebook -- but I do think that there are certainly cases that do require a set of publish notebooks

Yes, I would say essentially all user-facing cases. Content editing is a different issue entirely. For that, I would prefer to have myst extensions working well in vscode (which would require something like a javascirpt parser) but these are lower priority that anything user facing.

Thanks for this suggestion. I guess this would essentially boil down to adding notebook_repository option to the config that can be used generally by sphinx/themes etc:

Well, I don't understand the architecture all that well, but I think it would be used (if provided!) as the url prefix for anything currently connected to just the repositories URL.

That is, the current colab_url would use the notebook repo if provided rather than the content repo, as would the notebook download links, etc. If that sort of code is all in the themes, then that is where it makes sense to check those things. Basically: for anything notebook related in the jupyterbook repo, if they provide notebook_repository: it uses that info instead of the repository: for generating links. The source-editing links would use the repository: one regardless. I think this should be an all-or-nothing thing. Either everthing notebook related comes from that alternative link, or none do.

At ANU we have been discussing another option which would be to use jupyter-book as a coordinator in the case of organising multiple output / generated assets (such as html, pdf (via LaTeX), and ipynb sets) by introducing a website builder.

That all sounds great, though I don't entirely understand it entirely. But it doesn't sound like it would be a replacement for what we I suggested, just that it would provide a better production workflow given the myst source? If so, sounds great to me. I am relatively agnostic on those steps, except that it would be great if they were easy to maintain and standardized rather than custom code. The critique in this issue is narrowly around https://jupyterbook.org/interactive/launchbuttons.html being unable to work with generated notebooks from the content (even ones generated through an external workflow like the one you are describing). I believe making a distinction between the source repo and the notebook repo would alleviate that issue, but I could be wrong.

[choldgraf]

Hey all - could I ask that we move conversation about "launch buttons that point to generated notebooks" to a different issue? This issue is specifically about "users that want to configure a book to point to their own JupyterHub", which I think is related but not the same as the topic that is now generating most of the discussion in this issue. We should definitely cross-ref them though

[jlperla]

Sure, create another issue with the content, or we can copy it over. Of course, this issues has an upstream dependency. The "user-configurable launch buttons" needs stuff to launch.