Allow sections to be collapsed and expanded in the notebook

[lumberbot-app]

I haven't seen this discussed or planned so I thought I should bring it to the table. By cell grouping I mean exactly what Mathematica has: the possibility to group your cells in sections, subsections, etc, which let you easily move, evaluate and eliminate them.

I think one of the most important usability problems with the Sage notebook is the lack of this capability. That's why I don't use anymore but I've also seen on the Sage lists that a lot of people request it.

The only python notebook I know of that has this feature is codenode, now a defunct project by the same guy who developed the Sage notebook. Maybe you can use some of its ideas to implement it here, since it is BSD licensed.

---

Originally opened as ipython/ipython#805 by @ccordoba12, migration requested by @Carreau

[lumberbot-app]

@minrk commented: We have definitely discussed this in our many design conversations (Skype or IRC, so not published), and do plan to do it.

We haven't been creating Issues for most such things, as Issues are too heavyweight for the very large number of improvements/little fixes we have planned. The current notebook is very far from where we see it going. We do need a lighter weight palace to put these, which probably means the wiki.

Though Issues probably do make sense for the slightly larger-scale things like this one, such as integrating with git, better printing/publishing, etc.

[lumberbot-app]

@ellisonbg commented: There are two different issues here: one is the idea of grouping, the other the idea of nesting. Fernando and I have talked extensively about the fact that the way Mathematica handles grouping through nesting is quite annoying if you use it for anything non-trivial. Our idea is to implement grouping, but not nesting. How would this work?

We would keep the notebook as a linear sequence of cells. We would introduce different levels of heading cells, but these would be flat/inline (like latex section/subsection/subsubsection) rather than nested like the Mathematica notebook. We feel this style of document will be easier to import/export to different formats and is much simpler for users to manage and for us to develop a UI for. To introduce grouping like functionality, we thought about using the locations of different heading levels to identify logical groups that could be collapsed, moved and treated as a unit. We feel this would give the desired benefits of grouping without the complexity of nesting. The first step along with path is to create the heading cells, them we can start to think about a UI for using them as logical groups.

[lumberbot-app]

@minrk commented: It seems to me that explicit nesting is less complex than implicit nesting based on the level and location of header cells. Isn't it easier to flatten a nested structure than to nest a flat structure? I imagine we will need both notions internally, so should we not pick the one from which is is most sensible to make the other?

I also think performing actions on a group is more straightforward with nested structures (e.g. increase/decrease level, relocation, collapse/expand) because you invariably have the whole group as a single element in the structure, no matter what level. When the list is flat, you have to reconstruct the group every time you perform group actions.

[lumberbot-app]

@ellisonbg commented: On Fri, Jan 6, 2012 at 2:26 AM, Min RK
reply@reply.github.com
wrote:

It seems to me that explicit nesting is less complex than implicit nesting based on the level and location of header cells.  Isn't it easier to flatten a nested structure than to nest a flat structure?  I imagine we will need both notions internally, so should we not pick the one from which is is most sensible to make the other?

From the computer science/data structure perspective, I completely
agree that the theoretically "correct" data structure for the notebook
is a tree. But there are important reasons why this is not the right
model for the notebook.

• When it comes to authoring documents of almost any kind (pen/paper,
  Word, Google Docs, Latex) the model that users are used to is a linear
  sequence of text and headings with any hierarchy being only implicitly
  defined by how heading levels are used. Presenting an actual
  hierarchy to users requires them to track that information in their
  head increasing the cognitive load. They have to translate thoughts
  like "I want to move section 1.B after section 1.C and rename it
  section 2" to their nested equivalent.
• When I compare my experience in authoring documents in Mathematica
  (explicit nesting) with LyX (flat, implicit nesting) there is a huge
  difference. In LyX, I just set the heading level of a section and
  move on. Once those headings are set, I can just type content without
  any thought as to what level of nesting it appears at. With
  Mathematica, representing sections using explicit nesting is horribly
  painful - so much so that I avoid using it if at all possible. When
  you edit nested documents there are some operations (insertion/paste
  is the worst, but also with indent/dedent) that are ambiguous.
  Mathematica tries to guess how you want to handle these operations,
  but inevitably, you end up fighting with it. This is not a problem
  with Mathematica per-se, but with the underlying model which itself
  has these ambiguities.
• UI complexity. If you have a deeply nested, real hierarchy, you
  have to reflect that in the UI. Mathematica uses a sequence of
  vertical lines on the R side of the document that span the
  section/cells they group together. For one level of nesting this
  isn't bad. But for multiple levels of nesting, you end up with a rats
  nest of vertical lines and it become extremely difficult to figure out
  which one you want to click to manage things.
• Implementation complexity. The linear sequence of cells is simple
  to implement in both the notebook format and the UI. I am
  particularly concerned about the UI becoming too complex that we can't
  implement it in a clean robust manner in a reasonable amount of time.

I also think performing actions on a group is more straightforward with nested structures (e.g. increase/decrease level, relocation, collapse/expand) because you invariably have the whole group as a single element in the structure, no matter what level.  When the list is flat, you have to reconstruct the group every time you perform group actions.

While agree at the theoretical level, I have never found this to be
the case in authoring documents.

[lumberbot-app]

@minrk commented: Sounds reasonable. I suppose the bulk actions for which the nested structure makes things simpler don't actually come up that often.

[lumberbot-app]

@ellisonbg commented: The master now has the notion of heading cells. We should not be able to create a UI that will collapse and expand sections based on their headings.

[lumberbot-app]

@ccordoba12 commented: I've been following the notebook development quite closely and I think @ellisonbg is doing a terrific job! I just have a couple of thoughts to add to this discussion:

1. My original request was not directed exactly towards collapse/expand but more broadly to cell grouping. Collapse/expand is one of the things you can do with grouping and it's very important. I think it's more or less the equivalent of the "include document" functionality in Lyx: it lets you hide some parts of the document to focus your attention on others.
2. However, there are two more things you can do with grouping (as far as I can tell):
   • Delete/move a group. In Mathematica this is as easy as: select the group bracket (the blue vertical line, I mean), hit Ctrl+X and then Ctrl+V in another part of the notebook. I read in the last @fperez blog post that you plan to introduce an outline explorer similar to the Lyx one. Since this one also has the ability to move sections, I hope you could reproduce this functionality in IPython (although that seems a bit hard to do).
   • Evaluate a particular group. Perhaps this is not well known but, again in Mathematica, if you select a group bracket and hit Shift+Enter, you can evaluate just that group. I find this quite neat and useful but I don't know if you could implement it with current flat model.

As a side note: @ellisonbg, I haven't experienced in Mathematica the indent/dedent problem you described: to indent you just have to introduce a new heading level with a lower precedence from the previous one and Mathematica will create the group automatically until the next higher heading it finds. To dedent you change the type of your group heading to a higher precedence one.

To conclude: I'm not saying that Mathematica's way of doing things is the only possible one but that IMHO the cases I described are worth considering when you're taking the next design decisions about grouping.

Just my two cents,
Carlos

[lumberbot-app]

@ellisonbg commented: We do support cell groupings, but we are choosing to use Heading levels to define those cell groupings. You are right that Mathematica allows other types of cell grouping, but we feel that this more constrained model is cleaner.

There is no reason at all why we can't add other group level actions such as move, delete, evaluate, etc. We have to develop the UI for this, but I think the it is clear at the conceptual level. The core underlying logic we will need is the following: Given a Heading cell, we need to be able to walk the following cell and figure out which of them are "under" that heading. I think the basic logic is that you keep going until you encounter another heading that has the same level or less as the starting one. Once you have the list of contained cells, you can start to apply actions to the group of them. The challenging part will be designing a clean and functional UI for all of this.

[lumberbot-app]

@fperez commented: There's something I want to add here that I've been cooking in my head but so far have only mentioned to @stefanv at a whiteboard... a different kind of 'cell groups'.

The one thing where even the notebook doesn't work very well if you want to mix text-based explanations with code, is the definition of complex objects. The reason is a limitation imposed by Python itself: Python does not allow you to declare a class in multiple execution chunks (well, you can, but only using dynamic hacks that aren't very readable and which I will henceforth ignore).

This means that if we start using the notebook for declaring more complex classes, right now they have to live in a single, long cell. This prevents you for example from inserting a text paragraph with a mathematical explanation before continuing with the next method.

It occurred to me that if we had a concept of a 'cell group' where all the cells in the group would be unconditionally executed always together, all of them, then we'd solve this problem. Consider creating a cell group as stated above, then when you hit Shift-Enter from any cell in the group, we actually send for execution the union of all the code cells in the group ignoring non-code ones. Then, Python would always see them as a single cell, while the human could break up the visual layout between code and text as best fit the problem.

There are some UI challenges in how to best do this, but the more I think about it the more I like this idea. It would make the notebook a very nice environment for literate programming in a way that no existing python tool allows. Thoughts?

[lumberbot-app]

@minrk commented: Right now, Cell means two things:

A. atom of execution
B. editable area

And @fperez' proposal is to have some situation where one A-Cell refers to multiple B-Cells. Another phrasing of the same thing is essentially to have rich text comments without having to split a code block into separate executions.

With the implicit cell groups given by headings, we can actually make this work quite easily with optional behavior in 'run group', where it is not run as separate completely independent executions, but concatenated prior to execution. This is not difficult, but would lack the enforcement of a special notion of multi-textarea cell, and running the cells individually would be permitted, if not necessarily valid.

[lumberbot-app]

@ellisonbg commented: I will have to think more about this, but it definitely takes the
notebook in a literate programming direction. I almost think the best
way of handling this would be some sort of syntax within the cells,
maybe that taps into the multiline magic ideas:

%mycell
class Foo(object):

...a Markdown cell or so...

%mycell
def init(self, a):
self.a = a

It would be trivial to piece together and execute the union of these.
It also keeps the actual UI complete clean and simple. We could also
use this to introduce other grouping symantics. The downside is that
it clutters up the code cells themselves.

I do like Min's idea of having a "Run Union of Cell Group" action that
is based on the heading grouping. In some ways that is the simplest
way of going and would require no additional UI of cell clutter.

I think this would make the notebook extremely literate - more so than
any environment I have seen before.

On Wed, Feb 1, 2012 at 7:53 PM, Fernando Perez
reply@reply.github.com
wrote:

There's something I want to add here that I've been cooking in my head but so far have only mentioned to @stefanv at a whiteboard...  a different kind of 'cell groups'.

The one thing where even the notebook doesn't work very well if you want to mix text-based explanations with code, is the definition of complex objects.  The reason is a limitation imposed by Python itself: Python does not allow you to declare a class in multiple execution chunks (well, you can, but only using dynamic hacks that aren't very readable and which I will henceforth ignore).

This means that if we start using the notebook for declaring more complex classes, right now they have to live in a single, long cell.  This prevents you for example from inserting a text paragraph with a mathematical explanation before continuing with the next method.

It occurred to me that if we had a concept of a 'cell group' where all the cells in the group would be unconditionally executed always together, all of them, then we'd solve this problem.  Consider creating a cell group as stated above, then when you hit Shift-Enter from any cell in the group, we actually send for execution the union of all the code cells in the group ignoring non-code ones.  Then, Python would always see them as a single cell, while the human could break up the visual layout between code and text as best fit the problem.

There are some UI challenges in how to best do this, but the more I think about it the more I like this idea.  It would make the notebook a very nice environment for literate programming in a way that no existing python tool allows.  Thoughts?

---

Reply to this email directly or view it on GitHub:
ipython/ipython#805 (comment)

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

[lumberbot-app]

@tmbdev commented: I like the fact that headings now seem to define a hierarchy in the next notebook release.

I think it would be great if there was an outline window in which this hierarchy was explicitly visible, and that could be used to navigate, reorder, and rename entire subtrees.

Note that a tree structure or hierarchy need not be imposed; rather, it can be inferred on the fly from the content, like many LaTeX editors do.

[lumberbot-app]

@janschulz commented: from ipython/ipython#2748:
It would be nice to include https://github.com/danielstocks/jQuery-Collapse/ and letting it collapse everything between (manually included) headings.

[lumberbot-app]

@Carreau commented: You can include any library you wish in your custom.js via a simple $.getScript(url)

[lumberbot-app]

@hadim commented: Sorry to "re open" this issue. But do you have any news about it ? Are you planning to implement such feature or do we need to use custome script with the help of https://github.com/danielstocks/jQuery-Collapse/.

I am convinced this would be a great UI notebook improvement.

PS: I am not talking here about output cells hiding (which is already in ipython 0.13) but the ability to hide or show any part of notebook as well as creating group of cells to be able to hide and show severals cells at the same time.

[lumberbot-app]

@janschulz commented: Seems that this is implemented in ipython-contrib/jupyter_contrib_nbextensions#24

[lumberbot-app]

@chiwhalee commented: I think a bookmark navigation panel is better than collapse/folding idea. Folding is more suitable for pure source code, while navigation panel more suitable for a document or a notebook. Just see what MS word, acrabat pdf do; they all use bookmarks.
Navigation panel should also be much easier to implement.

[lumberbot-app]

@nthiery commented: I have been using folding in Maple a loong time ago, and have seen colleagues use it very effectively, in particular to maintain a "laboratory notebook" of their experiments. Maybe I am biased by being a coder, but I find it more practical and flexible for my use cases than a navigation panel.

In any case, I believe folding is a key feature; I have seen many people ask for it; personnally that would make me switch to using notebooks rather than terminal sessions (inside emacs).

Kuddos in advance to whoever puts energy into it!

[lumberbot-app]

@chiwhalee commented: Both navigation panel and section folding have been implemented. check out https://github.com/ipython-contrib/IPython-notebook-extensions. It is full featured and awsome, satisfy all my need. Thank those ipython-contrib guys!

[lumberbot-app]

@nthiery commented: Cool, thanks for the pointer! It's indeed a good step forward. Kudos to the ipython-contrib guys!

Things I believe would remain to do:

• Advertise hierarchical_collapse.js in the ipython-contrib wiki
• Activate this by default in IPython (extensions are great for power users, but folding is a nice basic feature for newcomers, and at this point installing an extension is not a "one-click" thing)
• Add keybord shortcuts (Alt-F as in the codefolding extension?)
• Use some standard visual scheme (e.g. a little button at the left of the section header, as in the codefolding extension). A button in the title bar is not so practical as it requires moving a lot the pointer back and forth.
• A cycle through mechanism? (show subsection titles, show all the section, collapose all the section)

Thanks!

[lumberbot-app]

@lrq3000 commented: Please set them as default, or at least advertise the ipython-contrib because I'm using ipython since v0.10 and I didn't know those features were available. I am excited to try the codefolding and the navigation panel extensions!

[lumberbot-app]

@drorata commented: Is there any update on this front?

[lumberbot-app]

@mhooreman commented: Hello,

I would find it very interesting. Is there any update on that?

Thanks a lot.

[lumberbot-app]

@pjc42 commented: in case you find this before you find the extension, this is now implemented at nbextensions - collapsible_headings. seems to work great, many thanks to the contributors...

[lumberbot-app]

I was not able to apply the following label(s): notebook

[Carreau]

I see that some usernames where not but in backticks when doing the migration. Apologies if you were mentioned and subscribed.

[rainwoodman]

How is this going?
I just worked with a new Jupyter notebook user who complained more than 5 times about not being able to fold chunks of the file like Mathematica in 30 minutes.

The vertical visible area of a notebook is only about half (between half and 2/3) of screen on her moderate mac book (the top bar, the browser bars, the tab bar, the scroll bar, then the Jupyter banner)

So usually a cell and the figure for that cell occupies more than the full available screen estate, hiding all context information for locating the cell in the full document.

Thus it is crucial to have some navigate aids, be it folding or a live TOC. TOC may be more efficient because it makes use of the extra horizontal space on wide monitors, without cluttering the existing notebook cells.

We may actually want a horizontal cell input/output layout too.

[gnestor]

There is a notebook extension for this: https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/nbextensions/collapsible_headings

I'm not sure about its current state or compatibility, but if nothing else it's a reference implementation.

[rainwoodman]

Thanks for the link.

The particular case was affecting new users who likely have no idea how to install an extension. Therefore I definitely would recommend including the feature as a builtin.

[wernight]

Colab has a rather good implementation (visually).

[gnestor]

Jupyter Lab also has a pretty good one 👍

[wernight]

But that's only a since cell, no a section with many possible cells.

[gnestor]

Ok, good point! Collapsible sections make more sense than individual cells in my opinion. Let's move this discussion over to jupyterlab/jupyterlab#2636 (comment) because new features like this will be implemented in JupyterLab vs. Jupyter Notebook.