Integrating the Case Studies from different repos

[sgibson91]

Summary

I was trying to work out logistically how case studies would be integrated into The Turing Way Jupyter Book given that the source code will be hosted in other repos.

I came across the following resource which details how the binder buttons can be configured to point to other repos (and even other BinderHubs!): https://jupyter.org/jupyter-book/features/interact.html#creating-interact-buttons-for-binderhub

What needs to be done?

• Binder links created for each case study
• Integrate these Binder links into the Jupyter Book
• What will the actual page of the book where the Binder link lives look like?
• Document how contributors can provide future case studies in this way.

Who can help?

• @choldgraf as creator of Jupyter Books may have some other advice :)

---

Updates

[sgibson91]

The different BinderHubs aspect of this will be relevant for #401

[dingaaling]

Hello from October 2022 @sgibson91 ! Hope all is well with you, many months later. The TTW Core team is following up on issues in our backlog and wanted to check if we now should close this issue? If so, feel free to comment or 👍 this comment. If not, please leave a comment on how we could revive this thread and how we can support this work.

We'll leave the issue open for now, and if we don't hear from you by mid-November we'll mark it as closed - but we are always happy to reopen it! Thank you!

[sgibson91]

This is really an upstream issue regarding Jupyter Book's features and/or something important to bear in mind if/when we bring interactive content into the book.

Concept

Jupyter Book can rely on mybinder.org or a known JupyterHub to provide interactive pages of the site. This was one vision of the Turing Way early on that we could have interactive examples.

Jupyter Book's role

At the time when this issue was opened, Jupyter Book allowed a configuration that not only set where the environment-building/code-execution would happen (e.g. mybinder.org, my-awesome-hub.com, etc) but also the source of the content to be pulled from, i.e., another repo. When the Jupyter Book moved out of beta, they deprecated this feature (as you can see, the docs link I provided is now a 404).

The Effect

Now, if we were to include an interactive something (case study, whatever) into the book:

i) It would have to live and be maintained in this repo. We wouldn't be able to pull in something from, say, a Carpentries lesson that lives in the Carpentries Incubator. Effectively limits the ability to reuse existing materials beyond a link out.
ii) We would have to maintain all the dependencies for each interactive piece in a single environment file. I can personally tell you from the kitchen-sink type environments I've come into contact with at 2i2c that this can get very unwieldy very quickly, especially if someone wrote one thing using version X of a package, and someone else contributes another thing using version Y of the same package. @acocac probably has similar experience managing a single "mega environment" for the environmental data science book.

Conclusion

We can do one of two things:

1. Actively work with the Jupyter Book community to outline the need for the feature that was deprecated, or;
2. Document what it means to have interactive content in the Turing Way, both from a technical maintenance aspect and how contributors should go about designing/adding their interactive pieces. Again @acocac will probably have some really useful experience around this.

I think either way, the path forward we choose should be tracked in a new issue.

[sgibson91]

I guess there is also option 3: do not accept interactive content (we don't yet have any... seems a shame though)

[dingaaling]

@sgibson91 thank you for following up on this and continuing the conversation. Agree that given that the need persists, we should continue exploring what the right option should be.

Would it make sense to revive this conversation during the upcoming Nov 2022 Book Dash or perhaps another Core TTW meeting to help decide which of the 3 options are best and create a new issue to track the chosen path?

[sgibson91]

I think either of those options makes sense!

[dingaaling]

Following a quick chat with @aleesteele seems like the plan is to add this to the agenda for the next Core TTW meeting and to action next steps in 2023!

[sgibson91]

I think the most modern docs on Jupyter Book's interactivity live here: https://jupyterbook.org/en/stable/interactive/launchbuttons.html

[acocac]

2. Document what it means to have interactive content in the Turing Way, both from a technical maintenance aspect and how contributors should go about designing/adding their interactive pieces. Again @acocac will probably have some really useful experience around this.

Glad to see the discussion of interactive case studies in TTW. Sarah is right a mega single environment in a Jupyter Book is quite paintful and challenging to maintain, mainly due to conflicts between dependencies. Also, Binder is usually slower to build and launch.

What about creating a sort of TTW gallery organisation for interactive use cases? Let me explain how it might work according to the experience of the EDS book.

@annefou and I have decided to create separate github repositories per interactive use case which are all part of the EDS book organisation. Then we call the rendered version as a submodule in the jupyter book (see here). We haven't worked in a smart and automated way to maintain and check updates in submodules so most of the process is still manual. We provide a template for contributors which facilitates them to upload their notebook and customised environment. For the Jupyter book, we also removed the launch on Binder button. Instead each rendered notebook contains a launch Binder badge which calls and launches the environmental within its corresponding notebook repository in the EDS organisation.

Note we build upon the Pangeo Gallery which has a more elegant manner to maintain submodules. However we've found very complex the workflow files so we worked in our own one using the mini-conda action from GitHub Marketplace. Thanks to Sarah, we recently tested repo2docker action which supports different programming languages and not only python. It works well, however the mini-conda action is easier and handy to maintain if use-cases are only in python.

Finally, how data- and compute-intensive will TTW interactive use cases be? There's a very interesting statement of need in the Pangeo discourse about integrating JupyterBook and JupyterHubs via CI. It aims to alleviate issues which expensive notebooks which don't run in the free-of-charge VMs provide by GitHub. Is the statement of need something TTW is interested?

Happy to chat further in the coming TTW core meeting or book dash.

[sgibson91]

We haven't worked in a smart and automated way to maintain and check updates in submodules so most of the process is still manual.

(Off topic: You can use Dependabot to keep submodules up-to-date using config like this)

[acocac]

(Off topic: You can use Dependabot to keep submodules up-to-date using config like this)

@sgibson91 cool! - I wasn't aware about dependabot, we just implemented a basic action to check submodules updates triggered at any push to the main branch. dependabot looks a better option ^^

[aleesteele]

Hi folks, just flagging that I've linked this issue in the agenda progressively being built for the core team meeting in December, thanks @dingaaling for flagging: https://hackmd.io/r0bZWWqkRZqvp4FFGR95vQ

Also adding that these kinds of issues are very topical for the upcoming infrastructure working group (#2690) as it emerges! 🚀