Allow multiple JupyterHub/BinderHub instances in launch buttons

[jlperla]

Description

Currently it is only possible to specify a single JupyterHub in the drop-down "launch" buttons.
It behaves like this:

launch_buttons:
  jupyterhub_url: "your-hub-url"  # The URL for your JupyterHub. (e.g., https://datahub.berkeley.edu)

However, many organizations have multiple JupyterHubs to which they direct readers.
For example, an organization-wide JupyterHub and a class-based JupyterHub where only subsets of users have access to each.

It should be possible for people to specify multiple JupyterHub (or BinderHub) instances, rather than only a single one.

Benefit

This would benefit any organization that has multiple hubs running for their content. It would probably most strongly benefit organizations like universities and classes.

More generally, this could be an opportunity to streamline our launch buttons so that it was a bit simpler and more easily extendable.

Implementation

Here is our logic around populating the launch buttons, this is likely where we'd need to make a change:

https://github.com/executablebooks/sphinx-book-theme/blob/43506962d4f90e6fc74d209d72503929900d84e7/sphinx_book_theme/topbar/launchbuttons.html#L3-L32

There are two options here:

1. Make it possible for each configuration to take a list of URLs. Then you could specify a list of hub URLs like so:

   launch_buttons:
     jupyterhub_url:
       - https://hub.university1.edu
       - https://hub.university2.edu

2. Make launch buttons take a list, where each item must follow a spec and corresponds to one hub. This would be a bigger change, but would allow us to generalize and extend more easily. In this case, launch_buttons would take entries like so:

   launch_buttons:
   - type: jupyterhub
     url: https://hub.university1.edu
     name: Some hub name 
   - type: binderhub
     url: https://binderhub.university1.edu
     name: Some other hub name
   - type: jupyterhub
     url: https://anotherhub.university1.edu
     name: A third hub name

   and so on.

   The launch buttons would then be a flat list that followed the entries that were in launch_buttons.

Some examples of how this has looked elsewhere:

See the quantecon datascience one:

and the quantecon julia one:

Tasks to complete

• Decide which kind of implementation we'd like to see above
• Implement and review

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[choldgraf]

I think this is a good idea, we could define a spec for the launch buttons, something like

{name: service_name, address: service_address, **kwargs)

where kwargs could be defined by the service in question (e.g., if you wanted to use the nbgitpuller pattern mentioned, then you'd need to also specify the "environment repo" you want used, whereas this wouldn't be applicable to a JupyterHub).

Then the configuration could just be:

launch_buttons:
  - name: jupyterhub
    address: datahub.berkeley.edu
  - name: binderhub
    address: mybinder.org
  - name: binderhub
    address: someotherbinder.org
    environment_repo: https://someotherrepo

and so-on.

[jlperla]

Sounds great. I guess the only question is whether a user editable field is feasible. For example, if a university wanted to use data8 or the quantecon lectures without getting us to regenerate the whole document to add them to the list. Or maybe that could be a second stage of the feature.

[jlperla]

@choldgraf More generally, you might want to consider how this would interact with having jupyter output (which I think is intended to use https://github.com/jupyter/jupyter-sphinx or is that wrong @mmcky ?) I kind of feel that the current feature which seems to only work with content stored as .ipynb is not very useful and should be superceded or discouraged. But maybe that is just me.

The reason I bring this up is that the configuration of nbgitpuller/etc. is heavily dependent on a notebook repository, so it seems like the launch stuff needs to be organized around that as well. is that what you had in mind with the environment_repo?

[jlperla]

Sorry, is this the repo that you are planning to integrate in @choldgraf and @mmcky ? https://github.com/QuantEcon/sphinx-tojupyter

I have to say that it is paradoxical that jupyterbook doesn't actually support jupyter natively at this point. Seems like a first-order feature :-)

[choldgraf]

if a university wanted to use data8 or the quantecon lectures without getting us to regenerate the whole document to add them to the list

Ah you mean you'd want people to be able to "bring their own binder configs"? That does sound useful but also complicated.

doesn't actually support jupyter natively at this point

can you explain what you mean by this? ipynb files work in jupyter book just fine, as does execution and integration of their outputs...

[jlperla]

Ah you mean you'd want people to be able to "bring their own binder configs"? That does sound useful but also complicated.

Not exactly, more like BYOBU (i.e. bring your own binder/nbgitpuller url).

I don't think it is that difficult, in fact it is already working on those sites I posted which we tested with PIMS and syzygy . It is just making it flexible to provide text that users could copy/paste in as an option instead of forcing them to use a hardcoded one. If you are already storing the users chosen launch option in the list you proposed as a cookie, then it is not a massive change. But I think this is largely a strategic decision for not just jupyterbook but the broader jupyter community as a whole: do we see executable notebooks/books/etc. as something where the user should be able to configure where they want them to execute? I hate the idea that people get trapped into colab, github codespaces, or even a hardcoded jupyterhub myself.

can you explain what you mean by this? ipynb files work in jupyter book just fine, as does execution and integration of their outputs...

Well, if something is already in a jupyter notebook then I can execute it, but I could before. But then jupyterbook is of limited use as all it does is a prettier version of nbconvert by embedding the notebook in a html frame. Moreover, nobody wants to author the content of a textbook with jupyter notebooks if they can help it for anything nontrivial.

As soon as I use any of the amazing jupyterbook features embedded in myst/etc, I can no longer have the book "executable" So unless I am missing something, I don't think it is an exaggeration (though I am admittedly prone to hyperbole :-) ) to say that jupyterbook doesn't have first-class support for jupyter notebooks at this point - or, in the absence of installing a particular extension in jupyterlab, the output isn't even executable. But my apologies if I am missing something

Is the idea to push myst as its own format and have the plugin in jupyterlab/etc. as a parallel to jupyter notebooks? If so, I see the strategy but it seems a pretty long road to success given that you would need to get colab/sagemaker/azure notebooks/vscode jupyter extension/etc. all to support it---and I don't see why they would have incentives to support a new format. More philosophically, I don't see why a user should be executing the underlying markup language designed for the books themselves as it was written to have all sorts of complicated stuff for publication.

@mmcky is finding ways around this limitation, but the toolchain is very complicated rather than something central to executable books. To me, it really seems like it should be integrated into the core of the product instead of as an external
and fragile toolchain - especially given all of the features like https://jupyterbook.org/interactive/launchbuttons.html#launch-buttons-for-interactivity that people will expect will work for any myst code and unless they read the docs carefully will be pretty confused.

If that is unclear, then I am happy to talk about this on a call at some point (and I have discussed this at length with @colliand previously) to explain my perspective more fully.

[phaustin]

+1 for this -- it would significantly help in sharing jupyter-books for graduate/upper-year courses and research publications that require multiple cores and a lot of memory to execute.

[choldgraf]

I think it's worth teasing apart these two goals:

1. Make it possible for authors to configure a variety of JupyterHub/Binder options for a book. This I think is pretty straightforward to do right now, we just need to define a spec like Allow multiple JupyterHub/BinderHub instances in launch buttons #1253 (comment) and add the proper UI around it. It'd probably be somewhere around these sections: https://github.com/executablebooks/sphinx-book-theme/blob/72f417b07f08816e7179d85dff2e5c7b2e5f41e7/sphinx_book_theme/launch.py#L97
2. Make it possible for users to configure their own JupyterHub/Binder for a book. This sounds like a longer-term goal, though potentially the QuantEcon lectures could be used as inspiration for how to do this.

If anybody wants to give a shot at implementation for 1 I am happy to review PRs! Another thought is that we might wanna pull the URL creation out of the Sphinx Book Theme and into something like MyST-NB so that other themes could use the same logic.

[jlperla]

I would add in goal number 0 as jupyter notebook output. Otherwise the additional two goals aren't very useful since it means we couldn't use the built-in jupyterbook support in the first place. Part of number 0 is that there is a separation of the notebook repo and the html source in practice.

Certainly these are two separate goals that can come in stages, but I think that they need to be designed together. Basically, I think the flexible stuff for goal 2 requires it to be done on the frontend in javascript, so starting along that path could make it a seamless transition.

It may come down to where in the code the URL specification for sphinx is combined with the actual notebook path/etc. in the front-end javascript or in the generated sphinx. So instead of https://github.com/executablebooks/sphinx-book-theme/blob/72f417b07f08816e7179d85dff2e5c7b2e5f41e7/sphinx_book_theme/launch.py#L98-L102 it would take the URL specification and do the substiution in javascript on the frontend.
For example, the possible binders could just be a list of strings intended for substitution at execution time which get generated in the list at sphinx runtime from the yaml specification. That is, something like #1253 (comment) could instead generate javascript for configuration like

let repo_url ="https://github.com/QuantEcon/quantecon-notebooks-datascience"
let repo = "quantecon-notebooks-datascience"
let branch = "main"
let org = "QuantEcon"
let launch_urls = [
    `https://mybinder.org/hub/user-redirect/git-pull?repo=${repo_url}&urlpath=${ui_pre}/${repo}/\${path_rel_repo}&branch=${branch}`,
    `https://colab.research.google.com/github/${org}/${repo}/blob/${branch}/\${path_rel_repo}`
]

Note that I left the repo_url, repo, org, etc. in there even though it could technically be substituted at the time of generating the links. Also notice that I left the ${path_rel_repo} as a special case because it changes per page.

So in one way or another, that could could lead to a configuration choice such as the following generated string in the frontend, and saved in a cookie

`https://mybinder.org/hub/user-redirect/git-pull?repo=https://github.com/QuantEcon/quantecon-notebooks-datascience&urlpath=lab/tree/quantecon-notebooks-datascience/${path_rel_repo}&branch=main`

So then on the launch, javascript frontend knows the path_rel_repo for the page and substitutes that into the string to find the URL to launch.

If you go with a variation on that design, then you can let the user provide their own direct strings at the configuration diaglog given that there teacher would tell them what to substitute) directly or there could be a simple GUI to make it less error prone. Basically, they would just give them a https://colab.research.google.com/github/${org}/${repo}/blob/${branch}/\${path_rel_repo} etc. string But that is all just front-end UI that could evolve.

Anyways, not saying that this design is the best but (1) that it works - as I think this is the basic approach used in the quantecon lectures; (2) you have to design the stage 2 in conjunction with stage 1 (even if you don't implement stage 2 immediately) because it changes where the URLs are constructed.

[choldgraf]

I think it's a good idea to separate this out into at least two issues otherwise it will be hard for others to understand the concrete steps needed to implement improvements. Agreed that a client side js implementation would be useful, but it would take more work and a different skillset to accomplish for whoever does it.

I still don't know what you mean by step 0, but I think that this issue already has too many intermingled topics in it so it'd be helpful if instead you opened up a discussion in meta if you want to talk about ways that jupyter book should support notebooks better. It would be helpful if you can provide an example of a thing that you want to do, and how jupyter book cannot do this, or even better an example repository, to help others understand.

[jlperla]

Not sure which repo you mean for me to add in an issue, but happy to. But it is pretty simple: jupyterbook does not generate jupyter notebooks in its core design - unless I am missing something fundamental, and I apologise if so. So any discussion of adding something in the core design for better configuration to launch notebooks is premature until that point. Not just because it is of limited use, but more because it might need to fundamentally change after you natively generate jupyter notebooks as a core feature - if that is your intention.

[choldgraf]

Ohh you're talking about generating notebooks not ingesting them, ok I understand now. When you said "Jupyter Output" above, I thought you were talking about cell outputs in notebooks, not "outputting entire Jupyter notebooks". So it's not so much that jupyter book does not have notebook support, but that the launch buttons don't have great notebook support when the notebooks aren't written in ipynb. That is indeed an interesting but complex topic. In the short term the answer is to tell book authors to make sure book authors have jupytext installed on any hub or binder they want links to point to. This becomes much more challenging when we want readers to define their own binder sources and would require a different long term solution

[mmcky]

@jlperla @choldgraf sorry I am a bit late to this overnight party. First I just want to add that I also like the idea of building better support for easier integration with execution environments 👍 in addition to support for ipynb outputs that can be consumed by end users rather than authors (as transformation of the source content).

A few comments on the thread above grouped loosely by theme:

Sorry, is this the repo that you are planning to integrate in @choldgraf and @mmcky ? https://github.com/QuantEcon/sphinx-tojupyter

The sphinx-tojupyter repo is only intended to support the generation of ipynb as output in the short to medium term as move integrated solutions are developed within the jupyter-book community. It essentially builds notebooks from the sphinx.AST which isn't a perfect source of information (about the source notebook) as there are transforms, and post-transforms within Sphinx that modify states etc. The benefit of the extension at the moment is that it translates all the myst extension objects (such as directives and roles) into base markdown which produces more readable notebooks that if you use jupytext to convert from md:myst to ipynb.

The project is working on jupyter support (via extensions) that would render MyST natively in the browser, and there are also open issues for thinking on how to add filters to notebook conversion to more simply translate source ipynb files to support other notebooks that have links to webpages rather than local links to files for example.

I have to say that it is paradoxical that jupyterbook doesn't actually support jupyter natively at this point

as @choldgraf indicates it is fully supported as a source to jupyter-book so you can author everything in notebooks if you like and then use jb to build html, pdf etc. I think the main issue is getting access to a notebook that can be used to support things like download notebook of the page, live notebooks on binder etc. which requires either transformation of the initial source notebook OR construction of ipynb from Sphinx representation (i.e. sphinx-tojupyter). Ultimately the shortest path is the best technical solution as it typically involves less information loss along the way through various software layers.

As soon as I use any of the amazing jupyterbook features embedded in myst/etc, I can no longer have the book "executable" So unless I am missing something

This isn't the case. You can use jupytex to convert a md:myst text file and it will essentially split the file into chunks of markdown and code-cells. When you open the notebook however not all the markdown is rendered (such as directives and roles) so it is visually compromised but executes. We are currently having a debate about how to supported nested code-cells in directives (which is currently prohibited) and the current design requires code-cell's to be at the root level of a document.

it seems a pretty long road to success given that you would need to get colab/sagemaker/azure notebooks/vscode jupyter extension/etc. all to support it

I think the main issue is with rendering. I might be wrong on this but the aim would be to develop a loadable extension that the jupyter notebook / jupyter lab ecosystem can load (by the user) to render myst syntax more fully. (But this is a way off yet to my knowledge)

In addition to the two items listed by @choldgraf:

3. build better support for end-user notebook support (as transformations, or built notebooks). There is current support for this is via sphinx-tojupyter + github actions to automatically update a notebook repository. @mmcky to write up a guide on the sphinx-tojupyter docs to help in the short term. This is also help fix some current issues: Downloadable Notebooks with asset links converted to URLs #886

[jlperla]

@mmcky Thanks so much for the update on the current set of software. When I said there wasn't native support for executable notebooks from myst, I meant "out of the box" and integrated with the launch buttons/etc. The ex-post toolchain seems to patch that, but I don't think it is connected to the launching that is the core of this problem in this issue.

@choldgraf

Ohh you're talking about generating notebooks not ingesting them, ok I understand now. When you said "Jupyter Output" above, I thought you were talking about cell outputs in notebooks, not "outputting entire Jupyter notebooks".

My apologies for a lack of clarity. To provide my perspective to ensure we are on the same page: I see 3 basic ways to write the content:

1. Write content using an ipynb
2. Write content in myst within jupyterlab
3. Write content in myst in the editor of choice

To me at least, (1) is the last place I would want to edit content. Forgetting the ipynb editing experience itself, the beauty of myst is that it is that you can write maintainable content, have book-related features. (2) is probably the second last place I would want to write content. I want to use a proper IDE with all of the standard bells and whistles (GitLens etc.) Jupyterlab is great for students to use for working with generated ipynb content, but not sure why I would want to write it there in the first place. So I would want vscode/vim/etc. instead (https://marketplace.visualstudio.com/items?itemName=ExecutableBookProject.myst-highlight is almost a year since last commit but I can wait on that of course)

OK, so if that is the way to write content (and I think only 3. is really necessary, but that is just an opinion). It seems like the goal of the project for execution is:

1. Statically generated HTML
2. Static PDF
3. Executable jupyter notebooks which can be used Interact, classic jupyter, jupyterlab, vscode's jupyter extension, colab, azure notebooks, etc.
4. easily launchable notebooks from the HTML

For these, (1) is already superb right now; (2) is in progress and doesn't seem to have anything really holding it back or architectural issues; but (3) doesn't work at all unless you tell end-users to run an extension for book editing in jupyterlab; or you avoid using myst; and (4) while it looks like these things are integrated when you scan the docs, the fine print says that it doesn't work for myst content.

So it's not so much that jupyter book does not have notebook support, but that the launch buttons don't have great notebook support when the notebooks aren't written in ipynb.

I think the first order issue to me is that juypterbook doesn't natively generate jupyter notebooks . If it did, then having the launch buttons point to it would be relatively easy.

That is indeed an interesting but complex topic.

To me this is complex but it seems central to the design. That is why I said that I see getting jupyter notebook as a first-class output is a requirement before launch buttons/etc. are worthwhile which is where this started. Until then, there is nothing for them to hook up to. I also think that if/when this stuff is integrated into the core of the product, there will be all sorts of things such as having the notebook repo to export to, etc.

In the short term the answer is to tell book authors to make sure book authors have jupytext installed on any hub or binder they want links to point to. This becomes much more challenging when we want readers to define their own binder sources and would require a different long term solution

I am not sure I understand. But if this is part of an ex-post toolchain, then I think I understand. Not sure what you mean though about binder/jupyterhub. This isn't for the users, right? For users of the notebooks, they could be in github codespaces, colab, etc.

No response is necessary unless there is something that I can help clarify. But let me also say that despite my feeling that the jupyter support is just not possible without a fragile ex-post toolchain, I also feel that the HTML stuff is outstanding so far! Thank you!

[jlperla]

Hi @choldgraf I was curious if you have decided on any plans for the timeline for jupyter (output) support for jupyterbook out of the box and without a custom theme/toolchain (i.e. @AakashGfude and @mmcky have hacked this sort of stuff into a theme for quantecon, but it isn't a jupyerbook feature)? Is this a planned feature at this point, or is the intention instead to focus on getting myst viewing/editing support as extensions t in the jupyter front-ends instead as the first-goal?

[fmaussion]

I think this is a good idea, we could define a spec for the launch buttons, something like

{name: service_name, address: service_address, **kwargs)

where kwargs could be defined by the service in question (e.g., if you wanted to use the nbgitpuller pattern mentioned, then you'd need to also specify the "environment repo" you want used, whereas this wouldn't be applicable to a JupyterHub).

Then the configuration could just be:

launch_buttons:
  - name: jupyterhub
    address: datahub.berkeley.edu
  - name: binderhub
    address: mybinder.org
  - name: binderhub
    address: someotherbinder.org
    environment_repo: https://someotherrepo

and so-on.

At OGGM (e.g. https://oggm.org/oggm-edu-notebooks, https://oggm.org/tutorials), we now have two hubs (one for users, hub.oggm.org, and one for classes, classroom.oggm.org). I would be thrilled to have the options described here:

• link more than one jupyterhub
• the possibility to separate content repo from environment repo in mybinder (see Allow to launch from binder with nbgitpuller executablebooks/sphinx-book-theme#282 and Allow use of nbgitpuller with binderhub_url launch button #1235 from @scottyhq).

What are the steps involved to get this feature into either Jupyter-Book or sphinx-book-theme? I can try to allocate the resources in order to find someone to work on this...

[choldgraf]

@fmaussion I've updated the top comment with some extra information and suggestions for implementation. I think we have roughly two options to take - happy to iterate and discuss on the best path forward there. I've also re-scoped this issue to be a bit tighter, and opened up #1444 to track user-configurable JupyterHubs

[choldgraf]

cc' also @jakubzitny who recently opened up #1434 to extend launch buttons to Deepnote as well. Do you have thoughts on whether the second configuration structure above would be more sensible / easier to implement for you?

[jakubzitny]

Hey, sorry to be late to the party @choldgraf. I think the second structure is a good way to go. In my case, I don't think it makes it radically easier, but for the reasons listed above (multiple JupyterHub instances) I think it makes sense.

One simplification could be: In Google Colab case I don't think it's necessary to have the user provide Colab URL, because that's always one hosted SaaS instance with the same URL which could be hardcoded in this repo. Not sure how relevant that is here, but wanted to share anyway :)