Turing Way/REG collaboration on UX/UI project update

[malvikasharan]

Summary

• This is a checklist for the collaborators to help navigate the scoping, development and execution of this project

What needs to be done?

• Create a dedicated directory for the UX/UI, add the proposal
• Provide a roadmap (this should be done by the REG members working on the project)
• Decide on the process and provide a prototype by November
• Finalise the presentable UX/UI plan for the Turing Way members to take forward before 3rd week of December

Who the stakeholders are?

• The Turing Way team
• REG members: Lydia France (0.5 FTE), May Yong (0.4 FTE), Iain Stanson (0.3 FTE 1 month)
• Jupyter Book/Executable books project members

Timeline

Phase 1: October

• Members working on this project will create a roadmap with the following details
  • Vision
  • Mission
  • Goals: 3 Key Performance Indicator
  • Milestones

Phase 2: November

• Create a prototype and demo it to The Turing Way team and community
• Community engagement and feedback

Phase 3: December

• Deploy the prototype for one user group (DSG) in the Turing Way repo and book (GitHub, Jupyter Book and Netlify)
• Provide a guidance document for The Turing Way to deploy for different user groups

Main tasks beyond as described in the timeline

• coordinate with the REG team and allocate member(s) in 1 FTE capacity for 3 full months
• Bring an executable books team member on board if needed
• Other Jupyter Book users who can suggest updates on UX/UI and design wishes for accessible navigation of the book #2034 <-- engage users

---

Updates

[malvikasharan]

Here is a related issue: executablebooks/jupyter-book#1389

[malvikasharan]

Moving discussion with @choldgraf here:

Malvika: Chris, can you connect us to someone in Jupyter who might be thinking about it and will be able to share what can or cannot be done?
Chris: Could you clarify a bit where you'd like the UX/UI improvements to be made? Do you mean for Jupyter Book, or for one of the Jupyter interfaces like JupyterLab? (I ask because it's two totally different subsets of people that work on these 🙂 ).

With @LydiaFrance and @myyong, we are interested in improving user experience in The Turing Way.
With more than 200 pages, our materials are not easy to explore for people with different user profiles. New data scientists or researchers at different stages of their work wouldn't know what is relevant for them unless they have been told by someone - which is not ideal from reusability perspective.

Over the next 2 months, we will explore different ways to improve findability. Some ideas include:

• improve search option by adding tag/metadata to each file (for instance what you have already raised here: Generate lists of pages based on tag metadata executablebooks/jupyter-book#1389)
• providing different entry points for people by curating materials, tagging them by persona/research-stage, and presenting them in the front page of the book
• or entirely different solution that we haven't yet thought of!

This work may affect JupyterBook codebase downstream in The Turing Way repo, or dependencies of JupyterBook.
This is something we would love to discuss beforehand to make sure that we are aligned with JupyterBook developers in case they would like this contribution to be added into the book.

I am talking about it from my perspective and I am sure both Lydia and May might be able to add more notes and query that could be useful for moving this project forward.

[BrainonSilicon]

Going to jump into this thread to get notifications!! I would love to help 🚀

[malvikasharan]

• Project duration: Oct-Dec.
• Members: Lydia France (0.5 FTE), May Yong (0.4 FTE), Iain Stanson (0.3 FTE 1 month)

Update: 29 Oct

• Project members have explored how the Jupyter Book works (builds)
• Have been exploring ways to improve searchability
  • Have checked Jupyter Book, can't figure where the JupyterBook page tags will go
  • There are tag option for Sphinx, but they think making changes to Sphinx currently doesn't feel like a productive way to go forward
• We need to have a chat with with the Jupyter Book developers to learn what they have tried and what they consider a good investment from our side (pinging @choldgraf)

[sgibson91]

I'm also going to ping @damianavila who has been working in the EBP project 🙂

[choldgraf]

It sounds like most of this work will be at the level of html-based UI. In the Jupyter Book world I've been the one doing much of the work on the book theme (which is what defines the default UI of Jupyter Book). I'm happy to give you pointers and brainstorm (though only have "nights and weekends" time for dedicated development). @damianavila is planning to get more acquainted with the sphinx builds and html stuff soon, so perhaps he can help as well. I also wonder if this issue is something that @mmcky has run into with the quantecon books, which I believe is pretty big.

One thing to check out for search is algolia - I've seen other sphinx sites use algolia for their search and I think it is much more powerful than the default sphinx search. Have you looked into that?

For multi-usecase books, one question I have is whether you've explore just having different books in general, or if you have a preference for one book with lots of different ways to slice and dice the chapters?

[myyong]

@choldgraf,

For multi-usecase books, one question I have is whether you've explore just having different books in general, or if you have a preference for one book with lots of different ways to slice and dice the chapters?

I think, if I've understood you correctly we're looking at the second option. We are aiming for a book (let's call it the-turing-way), which contains the superset of all chapters. We will create multiple books (eg. the-turing-collaborator, the-turing-version-controller) based on this superset.

We are looking at a profile.yaml file to allows users to edit reader-profiles, to contain a subset of pages relevant to that reader-profile. The idea is that we use the profile.yaml to auto-generate multiple tailored table of contents and folders based on superset content and superset toc.

Each time someone makes a change to the-turing-way superset, the multiple books will be regenerated and redeployed.

[malvikasharan]

Meeting: 19 Nov 2021

• Onboarding Iain Stenson and Alden to the project
• Iain is already working with May and Lydia on the project

We have 3 KPIs that we agreed upon:

1. Two design alternatives for achieving the goals set out by project (eg. to be able to generate different books targeted to different audiences/users), evaluate and document design decisions. This KPI will also engage users to capture their experience and collect feedback before finalising the prototype.
2. A prototype for deploying books for up to 3 user groups based on design chosen in 1. This KPI will also involve community engagement to assess the impact of suggested deployment process.
3. Documentation and deployment of prototype (with feedback taken into account) as deliverable if accepted. This KPI will enable future users and contributors to tailor and deploy books developed in this project for different user groups.

[choldgraf]

A quick though on implementations here. One thing you could try is using a little bit of Javascript to hide different chapters in the left sidebar. E.g., you could use a little dictionary like:

{
  "persona_a": ["page1", "page2", etc],
  "persona_b": ["page1", "page4", etc]
}

And then a few buttons in your sidebar or topbar where each button maps on to a user persona / use-case. Clicking a button would trigger some javascript that looks up the list of pages associated with a persona, and then hides all of the other pages.

It might get complex if you wanted a ton of specificity here, but maybe it'd be enough to hide just the top-level sections in the table of contents? Or maybe 1-2 levels deep?

[malvikasharan]

Feedback needed: 30 Nov 2021

We need some help here. May Yong, Lydia France and Iain Stenson have created a prototype here for a student group called ‘Enrichment’ which is a Turing scheme for PhD (still need to be populated with curated chapters): https://the-turing-way-customise.netlify.app/editions/enrichment/welcome.html (notice the -customise.netlify and /editions/ in the link)

:baby_yoda_soup: Can you please provide some suggestions to replace ‘customise’ and ‘edition’ with something like ‘persona’ or ‘profile’ - can you reply below with some ideas?
🎨 Based on the prototype, do you have some design wishes that we can integrate?

(once a few prototypes are in place, we will be working on providing entry way by redesigning the welcome page - we will come back for feedback on that in a week or so).

[sgibson91]

Porting my suggestion from Slack:

• edition -> pathway: Conceptually closer to what we're trying to achieve. Providing a specific persona a pathway through the resource
• the-turing-way-customise.netlify.app -> the-turing-way-choose-your-own-adventure.netlify.app: Slightly silly suggestion 😆 https://en.wikipedia.org/wiki/Choose_Your_Own_Adventure

[malvikasharan]

I like the use of pathways for each persona. Can we instead think of combining “persona and pathway”: https://the-turing-way-persona.netlify.app/pathway/enrichment? or makes more sense when reversed https://the-turing-way-pathway.netlify.app/persona/enrichment

[sgibson91]

Is the idea that the-turing-way-pathway.netlify.app will provide options for multiple pathways? If yes, my only nitpick (and it is a nitpick!) would be to pluralise pathways in the URL, and then the persona is singular. Otherwise, +1

[malvikasharan]

Plural (pathways) does fit better. @myyong suggested plural as well.
For the second part, we can go with either 'persona' or one the synonyms that is more appropriate such as profile, group, class, user, etc.

[choldgraf]

just a note that the netlify link above doesn't work for me! When I go here:

https://the-turing-way-customise.netlify.app/editions/enrichment/welcome.html

I get:

Not Found - Request ID: 01FNS31SZ663TT9G741Z8AQPR5

[malvikasharan]

Apology for not updating @choldgraf, the link has moved to the test link (which will update again soon to some semi-permanent location): https://the-turing-way-choose-your-own-adventure.netlify.app/welcome.html.

[choldgraf]

ah interesting - so you're building a new book for each pathway?

[malvikasharan]

At the moment that seemed like one of the solutions for us. I don't see the repo that @Iain-S @LydiaFrance @myyong are working on (can folks working on it make it public: https://github.com/alan-turing-institute/bio-Turing-Way/?) - but they have gone ahead with the plan to create a separate yaml file that has list of curated chapters for each persona which then builds separate books which can be accessed via the main book (as suggested in the test page).

Do you have some feedback on this?

This solution can likely be useful for the translated versions as well (pinging @camachoreina, @acocac, @BatoolMM, @AndreaSanchezTapia).

[acocac]

wowww great solution for improving users experience! also very useful for the translated versions (: @AndreaSanchezTapia @crangelsmith and I were indeed discussing today the most optimal path to deploy the translated versions in Netlify... a pending discussion with @sgibson91

[sgibson91]

I had recommended branch subdomains to @BatoolMM https://docs.netlify.com/domains-https/custom-domains/multiple-domains/#branch-subdomains, which would look something like:

• French version: fr.the-turing-way...
• German version: de.the-turing-way...
• etc...

where each translated version was hosted on a branch of the GitHub repo named with the same subdomain name (i.e., fr for French, de for German, etc).

But doing a little further reading, you would need to configure a custom domain name for the site, i.e., this wouldn't work with the-turing-way.netlify.app and I don't know how attached The Turing Way is to using the default domain name vs buying a domain outright.

@acocac you asked why we hadn't gone with a custom domain name in Slack, so if your plan is to do that for the Environmental AI book (sorry, I've forgotten the new name!) then I think the above link would work for you :)

[acocac]

thanks @sgibson91 for the guidance (: we'll get back to you to double check about the custom domains once we all have a decent knowledge of Crowdin, the localisation platform for translation. I'll also have a look at the link to the Environmental DS book 📚 .

[myyong]

@choldgraf (adding @Iain-S)

ah interesting - so you're building a new book for each pathway?

We are experimenting with different solutions. In the current version, we have a single book with pathways branching off that book. So the dir structure to the phd-student pathway would look like _build/html/pathway/phd-student/welcome.html.

There are pros and cons, but the blocker at the moment is that we end up with one ginormous build (~20 mins build time). So yes, we are now considering the implications of having different sites/build/book for each pathway. We are checking to see:

1. The links won't break
2. What the user experience might be like, to navigate from one book to another
3. How easy it would be to update across the different books when a change is made
4. How easy it would be to add a new pathway

The current design with the huge build checks all the above boxes, but we think perhaps might be less scalable if Netlify has a limit on each build time, resulting in timeout errors.

[sgibson91]

IIRC @ltetrel is using multiple Jupyter Books for the NeuroLibre paper submission platform (https://www.neurolibre.org/ https://docs.neurolibre.org/). Do you have any insight into the above questions?

[ltetrel]

@myyong In our case we have one jupyter-book that takes 8 hours to build (!!). We ended up using jupyter-book caching to cache the re-execution of notebooks for our build: neurolibre/neurolibre-reviews#6
https://github.com/NBCLab/nimare-paper/blob/master/content/_config.yml
This requires user to execute and push their cache to github, so may not be ideal for scalability in your case (one repo, multiple authors), maybe you could datalad each phd-student build folder?

[myyong]

@ltetrel

I wasn't sure if I wanted to laugh or cry when I saw this:

@myyong In our case we have one jupyter-book that takes 8 hours to build (!!).

Does the timeout error affect your build?

I had to google Datalad, thank you. I'll have a look.

[ltetrel]

I wasn't sure if I wanted to laugh or cry when I saw this:

Hahaha, we are actually debating if this submission fits our requirements.

Does the timeout error affect your build?

So in our case, build time was never an issue, but rather notebook execution (which is really fast now with caching).

I had to google Datalad, thank you. I'll have a look.

Basically it will help you decompose your jupyter-book build into multiple independent pieces (one piece being phd-student build):

for each phd-student:
    datalad get phd-student-i
    jupyter-book build
    datalad drop phd-student-i
merge all build folder

[myyong]

@ltetrel

That's exactly what our code does:

for each phd-student:
    datalad get phd-student-i
    jupyter-book build
    datalad drop phd-student-i
merge all build folder

But the final build folder is large. For each piece (phd-student-i in your example) it adds about 6 mins to the build. If we reach up to i = 30, we would be looking at 180 mins for each build. And each time a contributor makes a change, we'd be rebuilding. I am wondering if we should deploy a new site for each phd-student-i, but really, the total build time is still the same. But at least each of those builds won't be exiting with a timeout. But there are many advantages in having a single build. goes round in circles

Updated 0800:

Netlify has a build cutout time of 15 mins. My impression from here is that Netlify would bump it to 30 mins if it was a Pro account with an associated payment card. I think for us, this means splitting each pathway into a separate deployment.

We could create a separate index page for each pathway, and maintain a single content but I see some user experience issues there. Also, this approach wouldn't solve the problem when we have necessarily a different content (eg. different language).

Updated 0840:

The different language books would have to be deployed separately anyway.

[KirstieJane]

Rocking up a bit late but I like persona and pathways as discussed above ✔️

+1 on please make https://github.com/alan-turing-institute/bio-Turing-Way public.

[malvikasharan]

Tagging @myyong @LydiaFrance and @Iain-S for update from our last meeting as a few sentence in the comment here. 🙌

[malvikasharan]

Process for handover and conlusion of this project

1. Set a handover and debrief meeting: @myyong and @malvikasharan

Actions on REG members, @myyong, @LydiaFrance and @Iain-S, working on this project
2. [ ] Update this issue #2071 with comments summarising the progress and updates from the project for community record
3. [ ] Prepare ready to review PR (under progress #2246 ) that can be merged to The Turing Way book with at least one profile to allow community members to see and feedback on the developed UX
4. [ ] Create a new issue (connect with this issue 2071) sharing the test book which was used for creating and demo-ing the prototype
5. [ ] Provide manual/documentation for JupyterBook users who would like to use this feature in and outside The Turing Way

Actions on me
3. [ ] Design a GSoC project that builds on this project allowing us to properly integrate the developed feature to The Turing Way and conduct community feedback
4. [ ] Create a document to share with the funders to report on the project (progress, challenges, next steps), this document should be shared on The Turing Way and JupyterBook repository by capturing all the possible solutions that we tried (with pros and cons) and reason for the chosen option in The Turing Way (this should be co-developed with REG members, especially @myyong)

[myyong]

The deployed version on this project Turing-Way/Pathways can be found here. The repo where this project is developed is here. The pull request to merge this work into The Turing Way is here.

The deployed version contains three pathways:

1. DSG
2. Enrichment
3. Group Leaders

The documentation contains information about:

1. How to set up a notebook based on The Turing Way as example
2. How to add a new pathway (eg. PhD s)
3. How to change the position of cards on the page
4. How to run the tests.

A summary is as follows:

• Each pathway is represented as a card as seen in the Different Profiles section.
• Each card leads to a separate table of contents which contains links to pages marked for that pathway,
• Each page which has been selected for a pathway is marked with a coloured badge and label.