-
Notifications
You must be signed in to change notification settings - Fork 658
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow multiple JupyterHub/BinderHub instances in launch buttons #1253
Comments
Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗 |
I think this is a good idea, we could define a spec for the launch buttons, something like
where kwargs could be defined by the service in question (e.g., if you wanted to use the nbgitpuller pattern mentioned, then you'd need to also specify the "environment repo" you want used, whereas this wouldn't be applicable to a JupyterHub). Then the configuration could just be: launch_buttons:
- name: jupyterhub
address: datahub.berkeley.edu
- name: binderhub
address: mybinder.org
- name: binderhub
address: someotherbinder.org
environment_repo: https://someotherrepo and so-on. |
Sounds great. I guess the only question is whether a user editable field is feasible. For example, if a university wanted to use data8 or the quantecon lectures without getting us to regenerate the whole document to add them to the list. Or maybe that could be a second stage of the feature. |
@choldgraf More generally, you might want to consider how this would interact with having jupyter output (which I think is intended to use https://github.com/jupyter/jupyter-sphinx or is that wrong @mmcky ?) I kind of feel that the current feature which seems to only work with content stored as The reason I bring this up is that the configuration of nbgitpuller/etc. is heavily dependent on a notebook repository, so it seems like the launch stuff needs to be organized around that as well. is that what you had in mind with the |
Sorry, is this the repo that you are planning to integrate in @choldgraf and @mmcky ? https://github.com/QuantEcon/sphinx-tojupyter I have to say that it is paradoxical that |
Ah you mean you'd want people to be able to "bring their own binder configs"? That does sound useful but also complicated.
can you explain what you mean by this? ipynb files work in jupyter book just fine, as does execution and integration of their outputs... |
Not exactly, more like BYOBU (i.e. bring your own binder/nbgitpuller url). I don't think it is that difficult, in fact it is already working on those sites I posted which we tested with PIMS and syzygy . It is just making it flexible to provide text that users could copy/paste in as an option instead of forcing them to use a hardcoded one. If you are already storing the users chosen launch option in the list you proposed as a cookie, then it is not a massive change. But I think this is largely a strategic decision for not just jupyterbook but the broader jupyter community as a whole: do we see executable notebooks/books/etc. as something where the user should be able to configure where they want them to execute? I hate the idea that people get trapped into colab, github codespaces, or even a hardcoded jupyterhub myself.
Well, if something is already in a jupyter notebook then I can execute it, but I could before. But then jupyterbook is of limited use as all it does is a prettier version of nbconvert by embedding the notebook in a html frame. Moreover, nobody wants to author the content of a textbook with jupyter notebooks if they can help it for anything nontrivial. As soon as I use any of the amazing jupyterbook features embedded in myst/etc, I can no longer have the book "executable" So unless I am missing something, I don't think it is an exaggeration (though I am admittedly prone to hyperbole :-) ) to say that jupyterbook doesn't have first-class support for jupyter notebooks at this point - or, in the absence of installing a particular extension in jupyterlab, the output isn't even executable. But my apologies if I am missing something Is the idea to push myst as its own format and have the plugin in jupyterlab/etc. as a parallel to jupyter notebooks? If so, I see the strategy but it seems a pretty long road to success given that you would need to get colab/sagemaker/azure notebooks/vscode jupyter extension/etc. all to support it---and I don't see why they would have incentives to support a new format. More philosophically, I don't see why a user should be executing the underlying markup language designed for the books themselves as it was written to have all sorts of complicated stuff for publication. @mmcky is finding ways around this limitation, but the toolchain is very complicated rather than something central to executable books. To me, it really seems like it should be integrated into the core of the product instead of as an external If that is unclear, then I am happy to talk about this on a call at some point (and I have discussed this at length with @colliand previously) to explain my perspective more fully. |
+1 for this -- it would significantly help in sharing jupyter-books for graduate/upper-year courses and research publications that require multiple cores and a lot of memory to execute. |
I think it's worth teasing apart these two goals:
If anybody wants to give a shot at implementation for 1 I am happy to review PRs! Another thought is that we might wanna pull the URL creation out of the Sphinx Book Theme and into something like MyST-NB so that other themes could use the same logic. |
I would add in goal number 0 as jupyter notebook output. Otherwise the additional two goals aren't very useful since it means we couldn't use the built-in jupyterbook support in the first place. Part of number 0 is that there is a separation of the notebook repo and the html source in practice. Certainly these are two separate goals that can come in stages, but I think that they need to be designed together. Basically, I think the flexible stuff for goal 2 requires it to be done on the frontend in javascript, so starting along that path could make it a seamless transition. It may come down to where in the code the URL specification for sphinx is combined with the actual notebook path/etc. in the front-end javascript or in the generated sphinx. So instead of https://github.com/executablebooks/sphinx-book-theme/blob/72f417b07f08816e7179d85dff2e5c7b2e5f41e7/sphinx_book_theme/launch.py#L98-L102 it would take the URL specification and do the substiution in javascript on the frontend. let repo_url ="https://github.com/QuantEcon/quantecon-notebooks-datascience"
let repo = "quantecon-notebooks-datascience"
let branch = "main"
let org = "QuantEcon"
let launch_urls = [
`https://mybinder.org/hub/user-redirect/git-pull?repo=${repo_url}&urlpath=${ui_pre}/${repo}/\${path_rel_repo}&branch=${branch}`,
`https://colab.research.google.com/github/${org}/${repo}/blob/${branch}/\${path_rel_repo}`
] Note that I left the So in one way or another, that could could lead to a configuration choice such as the following generated string in the frontend, and saved in a cookie `https://mybinder.org/hub/user-redirect/git-pull?repo=https://github.com/QuantEcon/quantecon-notebooks-datascience&urlpath=lab/tree/quantecon-notebooks-datascience/${path_rel_repo}&branch=main` So then on the launch, javascript frontend knows the If you go with a variation on that design, then you can let the user provide their own direct strings at the configuration diaglog given that there teacher would tell them what to substitute) directly or there could be a simple GUI to make it less error prone. Basically, they would just give them a Anyways, not saying that this design is the best but (1) that it works - as I think this is the basic approach used in the quantecon lectures; (2) you have to design the stage 2 in conjunction with stage 1 (even if you don't implement stage 2 immediately) because it changes where the URLs are constructed. |
I think it's a good idea to separate this out into at least two issues otherwise it will be hard for others to understand the concrete steps needed to implement improvements. Agreed that a client side js implementation would be useful, but it would take more work and a different skillset to accomplish for whoever does it. I still don't know what you mean by step 0, but I think that this issue already has too many intermingled topics in it so it'd be helpful if instead you opened up a discussion in meta if you want to talk about ways that jupyter book should support notebooks better. It would be helpful if you can provide an example of a thing that you want to do, and how jupyter book cannot do this, or even better an example repository, to help others understand. |
Not sure which repo you mean for me to add in an issue, but happy to. But it is pretty simple: jupyterbook does not generate jupyter notebooks in its core design - unless I am missing something fundamental, and I apologise if so. So any discussion of adding something in the core design for better configuration to launch notebooks is premature until that point. Not just because it is of limited use, but more because it might need to fundamentally change after you natively generate jupyter notebooks as a core feature - if that is your intention. |
Ohh you're talking about generating notebooks not ingesting them, ok I understand now. When you said "Jupyter Output" above, I thought you were talking about cell outputs in notebooks, not "outputting entire Jupyter notebooks". So it's not so much that jupyter book does not have notebook support, but that the launch buttons don't have great notebook support when the notebooks aren't written in ipynb. That is indeed an interesting but complex topic. In the short term the answer is to tell book authors to make sure book authors have jupytext installed on any hub or binder they want links to point to. This becomes much more challenging when we want readers to define their own binder sources and would require a different long term solution |
@jlperla @choldgraf sorry I am a bit late to this overnight party. First I just want to add that I also like the idea of building better support for easier integration with execution environments 👍 in addition to support for A few comments on the thread above grouped loosely by theme:
The sphinx-tojupyter repo is only intended to support the generation of The project is working on
as @choldgraf indicates it is fully supported as a
This isn't the case. You can use
I think the main issue is with rendering. I might be wrong on this but the aim would be to develop a loadable extension that the In addition to the two items listed by @choldgraf:
|
@mmcky Thanks so much for the update on the current set of software. When I said there wasn't native support for executable notebooks from myst, I meant "out of the box" and integrated with the launch buttons/etc. The ex-post toolchain seems to patch that, but I don't think it is connected to the launching that is the core of this problem in this issue.
My apologies for a lack of clarity. To provide my perspective to ensure we are on the same page: I see 3 basic ways to write the content:
To me at least, (1) is the last place I would want to edit content. Forgetting the ipynb editing experience itself, the beauty of myst is that it is that you can write maintainable content, have book-related features. (2) is probably the second last place I would want to write content. I want to use a proper IDE with all of the standard bells and whistles (GitLens etc.) Jupyterlab is great for students to use for working with generated ipynb content, but not sure why I would want to write it there in the first place. So I would want vscode/vim/etc. instead (https://marketplace.visualstudio.com/items?itemName=ExecutableBookProject.myst-highlight is almost a year since last commit but I can wait on that of course) OK, so if that is the way to write content (and I think only 3. is really necessary, but that is just an opinion). It seems like the goal of the project for execution is:
For these, (1) is already superb right now; (2) is in progress and doesn't seem to have anything really holding it back or architectural issues; but (3) doesn't work at all unless you tell end-users to run an extension for book editing in jupyterlab; or you avoid using myst; and (4) while it looks like these things are integrated when you scan the docs, the fine print says that it doesn't work for myst content.
I think the first order issue to me is that juypterbook doesn't natively generate jupyter notebooks . If it did, then having the launch buttons point to it would be relatively easy.
To me this is complex but it seems central to the design. That is why I said that I see getting jupyter notebook as a first-class output is a requirement before launch buttons/etc. are worthwhile which is where this started. Until then, there is nothing for them to hook up to. I also think that if/when this stuff is integrated into the core of the product, there will be all sorts of things such as having the notebook repo to export to, etc.
I am not sure I understand. But if this is part of an ex-post toolchain, then I think I understand. Not sure what you mean though about binder/jupyterhub. This isn't for the users, right? For users of the notebooks, they could be in github codespaces, colab, etc. No response is necessary unless there is something that I can help clarify. But let me also say that despite my feeling that the jupyter support is just not possible without a fragile ex-post toolchain, I also feel that the HTML stuff is outstanding so far! Thank you! |
Hi @choldgraf I was curious if you have decided on any plans for the timeline for jupyter (output) support for jupyterbook out of the box and without a custom theme/toolchain (i.e. @AakashGfude and @mmcky have hacked this sort of stuff into a theme for quantecon, but it isn't a jupyerbook feature)? Is this a planned feature at this point, or is the intention instead to focus on getting myst viewing/editing support as extensions t in the jupyter front-ends instead as the first-goal? |
At OGGM (e.g. https://oggm.org/oggm-edu-notebooks, https://oggm.org/tutorials), we now have two hubs (one for users, hub.oggm.org, and one for classes, classroom.oggm.org). I would be thrilled to have the options described here:
What are the steps involved to get this feature into either Jupyter-Book or sphinx-book-theme? I can try to allocate the resources in order to find someone to work on this... |
@fmaussion I've updated the top comment with some extra information and suggestions for implementation. I think we have roughly two options to take - happy to iterate and discuss on the best path forward there. I've also re-scoped this issue to be a bit tighter, and opened up #1444 to track user-configurable JupyterHubs |
cc' also @jakubzitny who recently opened up #1434 to extend launch buttons to Deepnote as well. Do you have thoughts on whether the second configuration structure above would be more sensible / easier to implement for you? |
Hey, sorry to be late to the party @choldgraf. I think the second structure is a good way to go. In my case, I don't think it makes it radically easier, but for the reasons listed above (multiple JupyterHub instances) I think it makes sense. One simplification could be: In Google Colab case I don't think it's necessary to have the user provide Colab URL, because that's always one hosted SaaS instance with the same URL which could be hardcoded in this repo. Not sure how relevant that is here, but wanted to share anyway :) |
Description
Currently it is only possible to specify a single JupyterHub in the drop-down "launch" buttons.
It behaves like this:
However, many organizations have multiple JupyterHubs to which they direct readers.
For example, an organization-wide JupyterHub and a class-based JupyterHub where only subsets of users have access to each.
It should be possible for people to specify multiple JupyterHub (or BinderHub) instances, rather than only a single one.
Benefit
This would benefit any organization that has multiple hubs running for their content. It would probably most strongly benefit organizations like universities and classes.
More generally, this could be an opportunity to streamline our launch buttons so that it was a bit simpler and more easily extendable.
Implementation
Here is our logic around populating the launch buttons, this is likely where we'd need to make a change:
https://github.com/executablebooks/sphinx-book-theme/blob/43506962d4f90e6fc74d209d72503929900d84e7/sphinx_book_theme/topbar/launchbuttons.html#L3-L32
There are two options here:
Make it possible for each configuration to take a list of URLs. Then you could specify a list of hub URLs like so:
Make launch buttons take a list, where each item must follow a spec and corresponds to one hub. This would be a bigger change, but would allow us to generalize and extend more easily. In this case,
launch_buttons
would take entries like so:and so on.
The launch buttons would then be a flat list that followed the entries that were in
launch_buttons
.Some examples of how this has looked elsewhere:
See the quantecon datascience one:
and the quantecon julia one:
Tasks to complete
The text was updated successfully, but these errors were encountered: