Support collapsible hierarchy of sections

[scottdraves]

Allow users to toggle open/close sections by clicking on some kind of UI element.
This helps with navigating and organizing large notebooks.

[sccolbert]

Related issues:
#1742
#713

[scottdraves]

This might mean more integrated support for sections. Right now, afaik the only way to make them is to create a markdown cell and then type "#". This maybe hard to discover, and also makes them hard to build GUI around them (since they are in the middle of cells). Should sections be their own cell type?

[ellisonbg]

The original version of the ipython notebook did have heading cells. However, over time we found that users were just typing headings using markdown. Because of that, we changed the notebook format to remove the heading cells. After seeing how it worked both ways, I think it was a good decision. However, our current approach is completely compatible with offering sectioning. Here is the rough idea: * A separate plugin that would watch a notebook model. * On change, it would scan the cells, and find markdown cells that begin with a MD heading (one or more `#`). * It would then infer the "virtual" sections by looking at those headings. * We can build a side panel UI that presents those virtual sections in a useful manner. It shouldn't be too difficult to build that...

…

On Fri, May 26, 2017 at 11:10 AM, Scott Draves ***@***.***> wrote: This might mean more integrated support for sections. Right now, afaik the only way to make them is to create a markdown cell and then type "#". This maybe hard to discover, and also makes them hard to build GUI around them (since they are in the middle of cells). Should sections be their own cell type? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2275 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABr0IuudAQL86qcj9HsWlKEy1mJC3ATks5r9xWpgaJpZM4Nmtms> .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[scottdraves]

totally agree on keeping the data model the same and building improved UI around it (the virtual sections).

[scottdraves]

Is there an issue already filed for making the markdown editor more GUI, either with a toolbar like github's, or a full on WYSIWYG editor? Isn't there an effort to make this pluggable and Maybe that's where this gui lives?

[ellisonbg]

Ran into a need for this working with some large notebooks recently. Would really help a lot.

[aquirdTurtle]

This is a really important feature for me. In my lab, we use a single notebook to organize all the data & notes that we collect over the course of a day, and by the end of each day the notebook is quite large. Notebook organization is important enough that I probably need to stick with the plain notebook + the nb_extensions collapsible headers extension until collapsible headers or a TOC is supported in Jupyter lab. Really excited for Jupyter lab though!

[pluviosilla]

I find the NB Extensions feature that allows a TOC on the lefthand side invaluable with large notebooks, and all my notebooks tend to get large, because of the way I use this marvelous tool. Unfortunately, after the classic notebook reaches a certain size it starts to time out when loading the Javascript for NB Extensions and MathJax. I am extremely eager to find out whether JupyterLab will offer the same functionality and fix the timing out problem.

This image of the TOC supplied by NB Extensions shows how it makes a complex document much more accessible. My fear is that Jupyterlab developers will be too focused on producing an IDE with lots of new features and that you might sacrifice the qualities of Jupyter that make it suitable for turning programs into documents. I'd rather you focus on getting current features to work better, especially document publishing features that tend to falter as documents grow.

[jsmm]

There is no way I can orderly write nor browse large notebooks without table of contents. I am also waiting for TOC2 to be ported before moving from Jupyter notebook to lab.

[jasongrout]

My fear is that Jupyterlab developers will be too focused on producing an IDE with lots of new features and that you might sacrifice the qualities of Jupyter that make it suitable for turning programs into documents. I'd rather you focus on getting current features to work better, especially document publishing features that tend to falter as documents grow.

I think that the largest part of our time is spent on making a rock-solid extensible underlying framework so that people can explore this and many other ideas and workflows with extensions.

For example, @ian-r-rose is experimenting right now with building a TOC extension: #3949

[ellisonbg]

We would be foolish to try to compete with traditional IDEs - we remain focused on interactive computing and computational narratives.

…

On Sat, Feb 24, 2018 at 9:10 AM, Jason Grout ***@***.***> wrote: My fear is that Jupyterlab developers will be too focused on producing an IDE with lots of new features and that you might sacrifice the qualities of Jupyter that make it suitable for turning programs into documents. I'd rather you focus on getting current features to work better, especially document publishing features that tend to falter as documents grow. I think that the largest part of our time by far is spent on making a rock-solid extensible underlying framework so that people can explore this and many other ideas and workflows with extensions. For example, @ian-r-rose <https://github.com/ian-r-rose> is experimenting right now with building a TOC extension: #3949 <#3949> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2275 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABr0CWCbFU19tWlcc8G7g1z6k65nepIks5tYEKdgaJpZM4Nmtms> .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[shgidi]

@cdagnino This issue is about collapsible headers, not about table of contents.

[riemannzetagambit]

Are there any branches that are working on this? I'd like to contribute if I can-- I know a number of folks who haven't transitioned over from plain notebooks because of the lack of this feature. I love the other features of jupyterlab too much now to not use it but it's a real pain on any notebook of substance to navigate without collapsible headings.

[ellisonbg]

We are working on a new table of contents extension that will have that feature here: https://github.com/jupyterlab/jupyterlab-toc We are not quite there yet, but things have been moving quickly ;-) on this extension

…

On Fri, Aug 24, 2018 at 1:48 PM David Stone ***@***.***> wrote: Are there any branches that are working on this? I'd like to contribute if I can-- I know a number of folks who haven't transitioned over from plain notebooks because of the lack of this feature. I love the other features of jupyterlab too much now to not use it but it's a real pain on any notebook of substance to navigate without collapsible headings. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2275 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABr0KTnnCR-USZ1_3mJ552kfBsedgsaks5uUDx0gaJpZM4Nmtms> .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[marius311]

@ellisonbg Thanks for you guys' work on that, I'm very happy to hear collapsable headers are on the way via that extension, since at the moment its the main drawback of Jupyterlab for me.

As a general point though, as stated above, the question of whether sections/headings within the notebook are collapsible is separate from where the TOC knows about it or can trigger it, right? One could imagine the notebook itself allows collapsible sections independent of TOC behavior, such as how the collapsible_headings plugin works for Jupyter notebook.

[ellisonbg]

Yep, very true that the notebook can have collapsible sections without the TOC - although right now the TOC extension is where all the logic for computing the inferred hierarchy of nested sections is located. One reason for that is that we are building out the TOC to work with lots of different files types. But this will likely evolve to enabling this type of collapsing in the documents themselves in some manner.

…

On Tue, Aug 28, 2018 at 10:50 AM Marius Millea ***@***.***> wrote: @ellisonbg <https://github.com/ellisonbg> Thanks for you guys' work on that, I'm very happy to hear collapsable headers are on the way via that extension, since at the moment its the main drawback of Jupyterlab for me. As a general point though, as stated above, the question of whether sections/headings within the notebook are collapsible is separate from where the TOC knows about it or can trigger it, right? One could imagine the notebook itself allows collapsible sections independent of TOC behavior, such as how the collapsible_headings plugin works for Jupyter notebook. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2275 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABr0E316AcRlXzbtXNPA6_dSOnF5KNSks5uVVjTgaJpZM4Nmtms> .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[pluviosilla]

I’d like to chime in with another blue skies scenario feature wish list with regard to TOCs. J When the dust settles on TOC development and the collapsible hierarchy in particular, I sincerely hope that its default behavior will **not** be for the entire Tree View to expand every single time one does an edit, the way it does in Jupyter notebooks. I find that behavior to be extremely annoying. From: Brian E. Granger [mailto:notifications@github.com] Sent: Wednesday, August 29, 2018 2:16 PM To: jupyterlab/jupyterlab Cc: John Strong; Comment Subject: Re: [jupyterlab/jupyterlab] Support collapsible hierarchy of sections (#2275) Yep, very true that the notebook can have collapsible sections without the TOC - although right now the TOC extension is where all the logic for computing the inferred hierarchy of nested sections is located. One reason for that is that we are building out the TOC to work with lots of different files types. But this will likely evolve to enabling this type of collapsing in the documents themselves in some manner.

…

On Tue, Aug 28, 2018 at 10:50 AM Marius Millea ***@***.***> wrote: @ellisonbg <https://github.com/ellisonbg> Thanks for you guys' work on that, I'm very happy to hear collapsable headers are on the way via that extension, since at the moment its the main drawback of Jupyterlab for me. As a general point though, as stated above, the question of whether sections/headings within the notebook are collapsible is separate from where the TOC knows about it or can trigger it, right? One could imagine the notebook itself allows collapsible sections independent of TOC behavior, such as how the collapsible_headings plugin works for Jupyter notebook. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2275 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABr0E316AcRlXzbtXNPA6_dSOnF5KNSks5uVVjTgaJpZM4Nmtms> .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2275 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AASVbHdlMpXHL_2TnDhm0ZaQH64uaYbIks5uVuhmgaJpZM4Nmtms> . <https://github.com/notifications/beacon/AASVbN9ybDzAvD6GxSlTaSMKzX3XemQaks5uVuhmgaJpZM4Nmtms.gif>

[watkinrt]

Like a couple other comments, the lack of collapsible headings has prevented me from fully switching over to JupyterLab; it is just too hard to manage large documents (even with the TOC extension). For that reason, I started developing a collapsible heading extension based on original notebook extension. I don't want to replicate work if the developers of the TOC extension are going to implement the same feature (especially since they area to be much more sophisticated developers than me), but with no roadmap or timeline, I also don't want to wait around for its development.

I posted a comment on the JupyterLab discussion board regarding development suggestions for this feature, but it looks like this might be a better place to get feedback. As I stated earlier, I'm just trying to transition the original notebook extension to JupyterLab's framework; however, it isn’t always clear to me how to handle this transition. I’ve been able to implement methods to identify header cells and to manipulate metadata, but actually implementing the collapse philosophy is a little lost on me. The original extension directly manipulates the javascript to incorporate event handling, css modifications, etc, which seems to be what JupyterLab is trying to get away from (although I can't say I fully understand the JupyterLab framework). I took a look into the input/output cell collapse functionality build into JupyterLab, but haven’t figured out how to translate that into manipulating groups of cells denoted by headings.

Anyone have any thoughts on the best way to approach this?

[aquirdTurtle]

I got a little tired of waiting, so I took the weekend to take a shot at implementing my own collapsible headings extension. The very early version of this is located here. I think the basic collapse functionality is working well, but the UI still leaves a lot to be desired. I haven't figured out how to properly add some sort of clickable widget next to the markdown headers like in the original, so right now you can only collapse the currently selected header with a hotkey (ctrl Q). Any advice on how to add such a ui element to the cells in addition would be very useful.

This is simultaneously my first attempt at a Jupyterlab extension, as well as my first foray into anything java/typescript or any sort of web-application, so I'm sure the implementation could be better as well. Feedback is very welcome.

[marctorsoc]

We are working on a new table of contents extension that will have that feature here: https://github.com/jupyterlab/jupyterlab-toc We are not quite there yet, but things have been moving quickly ;-) on this extension
…
On Fri, Aug 24, 2018 at 1:48 PM David Stone @.***> wrote: Are there any branches that are working on this? I'd like to contribute if I can-- I know a number of folks who haven't transitioned over from plain notebooks because of the lack of this feature. I love the other features of jupyterlab too much now to not use it but it's a real pain on any notebook of substance to navigate without collapsible headings. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2275 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AABr0KTnnCR-USZ1_3mJ552kfBsedgsaks5uUDx0gaJpZM4Nmtms .
-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

you're my heroes!

[kgryte]

For those wanting to collapse notebook sections, I submitted a PR to the JupyterLab ToC extension to provide such functionality. Feel free to experiment in the following Binder.

[VPetukhov]

Thanks so much @kgryte !
So it's merged, and it works perfectly! Should the issue be closed?

[kgryte]

On my end, yes, this issue can be closed due to the ToC extension; however, not clear whether desire exists to build this directly into core.

[layne-sadler]

This item ranked 8th most important of 32 items in survey.

[jasongrout]

I think the fundamental issue making this hard is that a notebook has two notions of structure that conflict with each other:

• The "header hierarchy" is a purely visual construct from rendered markdown header lines, giving the user a sense of structure purely from formatting, but this visual structure is not really backed up by any actual structure in the document.
• The notebook file format, visual appearance, and UX has a flat structure of a list of cells which are all peers to each other. These cells are the atomic units of structure in the notebook.

This tension creates some very odd and confusing behaviors, for example:

• A header can start what appears to be a subsection in middle of a markdown cell, but that visual subsection extends beyond the end of the cell to the following code cells (which breaks the notion that cells are the atomic units)
• When a user navigates through a notebook, they do it cell by cell, completely ignoring any header hierarchy
• When a user wants to edit or change a subsection, they can only edit the markdown of the entire cell containing that subsection, which breaks the illusion that the subsection is actually a structural part of the document.
• Collapsing a section may hide a part of a markdown cell, multiple code cells, and part of another cell, since the header hierarchy can span all or parts of following cells.

Changing the UX to understand the header hierarchy also leads to some code smell in the implementation. For example, the DOM structure of the notebook at the highest level is a list of cells, but then this means that changes in one cell spill over to effects in adjacent cells. Also, the header hierarchy cannot be understood until you parse all of the markdown cells in a document. For example, it's impossible to ask "what section does this code cell belong to" until you parse backwards all of the markdown cells above it until you find a header.

Things were much clearer when we had notebook header cells as part of the notebook structure - then the header hierarchy structure was a first-class citizen of the notebook format and model. As Brian mentions above, in practice users preferred the easier in-text headers that visually were the same, but structurally were very different.

One of the proposals above attempted to bridge this gap by saying that the first header in a markdown cell text effectively determined what kind of header cell it was, essentially giving us back header cells. I thought that was a nice compromise given the state of the problem, but it still leads to some confusion about why other headers in a single markdown cell are ignored.

[aiqc]

@jasongrout 'header cell' comes to mind.

[jasongrout]

And just to be clear, I would love to have a way to group cells and have a hierarchy in a notebook document, but it's a challenge since the notebook format doubled down years ago on headers that were purely visual instead of actually structural.

[aquirdTurtle]

I'm mildly confused by some of this discussion considering my extension which, while far from perfect, basically implements the request. (It's also much better now than the early screen-recording above). I guess users in the above mentioned survey either don't know about the extension or would prefer the functionality be integrated and shipped by default.

One of the proposals above attempted to bridge this gap by saying...

I'm not sure if you're referring to my extension but this is indeed what my extension does. It doesn't even put a collapse button near other markdown headers or collapse any markdown code within the header cell, so I think practically speaking it's pretty clear to the user how the extension is working and encourages people to make header-only cells. Maybe the issue is rather when using the extension to view a notebook which was not designed with the extension in mind.

[jasongrout]

@aquirdTurtle - thank you very much for replying here, and my apologies for not investigating the current state of your extension that you mentioned in the comments above.

I'm not sure if you're referring to my extension but this is indeed what my extension does.

I think your extension behavior is the best compromise I've seen to give some nesting structure to a notebook, but still have cells be the atomic units. If you wanted to submit a PR to core with that behavior, I think that would be awesome and a great step towards supporting section hierarchies.

[ellisonbg]

@aquirdTurtle Thanks for sharing this, I was also unaware of your extension. I too would love to see this in JupyterLab itself :-)

[marthacryan]

@aquirdTurtle Would you like any help making a PR for that? I was planning to spend some time working on this so I'd be happy to help