Avoiding large output in notebooks

[astrofrog]

I am using the IPython notebook as a teaching tool (as do many others I imagine), and as part of this, students have to write simple programs using loops. Of course, as would be expected, students end up writing infinite loops, which if they contain a print statement will basically crash the web browser, and it's hard to explain to them how to kill the browser process, restart it, etc.

Would it be possible to have an option (or even have it by default) for the notebook that a single cell cannot print out more than a given number of lines or characters, to ensure that the notebook is more fool-proof against infinite loops? Of course, when not using print statements, students can simply interrupt the kernel, but if values are printed, then this can cause severe crashes.

I'd be fine with this being optional, and I could tell my students to enable it at the start of the course or notebook. Are there already ways to do this?

[Carreau]

Probably not at baked in IPython,

Try the following in custom.js, which will drop (roughly) anything after 500 char/output.
You could of course package it and refine. Would it suit you ?

require(["notebook/js/outputarea", "base/js/dialog","notebook/js/codecell"], function(oa, dialog, cc){
    console.log(oa.OutputArea, oa.OutputArea.prototype)
    oa.OutputArea.prototype._handle_output = oa.OutputArea.prototype.handle_output
    console.log('[extension]: will drop ifmore than 10 messages/cell')
    oa.OutputArea.prototype.handle_output = function (msg){
        if(!this.count){this.count=0}
        this.count = this.count+String(msg.content.data).length;
        if(this.count>500){
            if(!this.drop){
            dialog.modal({
                    title : 'Infinite loop ? ',
                    body : "Oops, I got too many things from kernel ! I'll just drop them !",
                    buttons : {'OK' : { 'class' : 'btn-primary' },
                    }
                }
                )
            }
            this.drop=true;
            return 
        }
        return this._handle_output(msg);
    }

    cc.CodeCell.prototype._execute = cc.CodeCell.prototype.execute;
    cc.CodeCell.prototype.execute = function(){
        // reset counter on execution.
        this.output_area.count = 0;
        this.output_area.drop  = false;
        return this._execute();
    }

});

[astrofrog]

@Carreau - thanks for the js code! I will try it out. However, this maybe be something to consider putting as a default menu option (like many terminals that have 'limit output to x lines'), since I'm not sure if I can easily get my students to install custom javascript if they are having issues with infinite loops ;) (and I suspect this is a common issue).

[keflavich]

👍 to @astrofrog's suggestion - when I was teaching with the notebook, the entire computing node sometimes froze on costly infinite loops, which froze everyone at the same time. It required me calling in the IT folks with superuser access to kill the offending Firefox instances.

[ccordoba12]

I also think this belongs to IPython proper, instead of leaving it to users to deal with it. Qtconsole already has support to print up to a certain number of lines of output, which prevents apps like Spyder to freeze with infinite loops.

[Carreau]

One of the issues we have right now with theses things is that we have no way to persist the configuration except custom.js. And until this is fixed, the answer to any of these question where the response might depend on use case will be no. So even if it makes sens to have a "reasonable" limit, I think we will have to postpone the implementation of this feature. It should push anyone interested in the project (us included) in implementing the config persistence. Then things like limiting output get easy.

In the meantime, if it is on nodes/clusters, as you (or your IT) is responsible for deployment, it should be relatively easy to add the custom extensions to all students. That's what @dsblank is doing for a course and IIRC they have highly customized notebook.

The multi-user notebook is taking shape so the need to ask student to install notebook get smaller.
You can send profile in the folder that have material for the course and ask to launch notebook with the flag that user profile in CWD.

[astrofrog]

@Carreau - thanks - what is the preferred approach for adding a custom extension for all students if I have access to the IPython installation? (but not their .ipython directories)

[dsblank]

We are using fairly straightforward notebooks, with an external kernel (using Scheme). But, we have provided some instruction for students to customize their notebooks [1]. Most of the customization I have needed for class, I just do in the kernel, and update in the installed codebase on the server. We are running the jupyterhub on our own servers [2]. This is really a nice way to go, once Jupyter is stable. Only one install... once it works, it mostly works for everyone.

I do think that IPython should not allow a user to crash the browser. Just like Python's max recursion limit, one shouldn't be allowed to output too much text in any one cell. I have had similar problems, but hadn't thought that it may have been caused by sheer volume of output.

If I wanted to install a particular nbextension, I could just copy it to their custom.js, as all of the accounts are on one computer. Or I could supply a very simple set of directions for them to copy paste, ala [1] instructions on installing the Spell Check (eg, combinations of !ipython install-nbextension and %%file.

Hope that helps! If not, I'd be glad to say more about our setup.

[1] - http://jupyter.cs.brynmawr.edu/hub/dblank/public/Jupyter%20Help.ipynb

[2] - http://jupyter.cs.brynmawr.edu/

[astrofrog]

Just a ping on this - I've had more students crash their notebooks this morning while learning loops (it's not easy for me to deploy a custom JS file to all of them)

[jhamrick]

Yeah, this really should be addressed.

@Carreau I am of the opinion that this should be in the notebook proper as well, and that there should be some high default limit -- I don't know, like 10000 lines of output -- that gets truncated past that. I think if anybody really wants that to be bigger than that, they are probably a power user, and should be able to go edit custom.js themselves. Students or anyone else just learning how to use the notebook are the most likely to make the mistake of creating an infinite for loop with output, which then crashes their browser, which is a horrible experience for them. Even worse, I've had students for whom this happened, and they had to restart their kernel, but in the process accidentally saved the notebook and then couldn't reopen it because there was too much output saved into it. That's somewhat of a separate issue, but still related -- it really shouldn't end up being the case that someone inadvertently saves the notebook while it's crashing and then can't open it again without manually going in a clearing the outputs.