move utils.fixConsole calls closer to display, so they don't clobber the actual data stored in the notebook.
also store nonexistent input_prompt_number as null, rather than , which is what should be drawn, not the actual value.
We need to be careful with this one, because currently we do write HTML-escaped text to the notebook (see #1205), so this will change the output. So if you open a notebook written by 0.12, plaintext output will be double-escaped until you run it again. The fixed output seems to display correctly in the old/bad code.
Let's not merge until we check if there's an easy way to avoid escaping to HTML a second time.
don't preserve fixConsole output in json
store nonexistent prompt number as null
Just kidding - I wasn't diligent with my browser-refresh when switching branches. Bad notebooks (written by 0.12) loaded in this branch will see the HTML markup as plaintext, because the content will be double-escaped on load. Good notebooks (written by this branch) will be loaded unescaped ([0;31m in tact, output uncolored) in 0.12.
Of course, everything should be safe, so the only thing needed when moving from one to the other is to re-run the notebook to generate the outputs with/without the bug so it matches the running frontend.
I don't know if this is fixable, because the escaped HTML output is always perfectly valid as plaintext, so it's impossible to be certain about whether an extra escape should be made.
Note that this is not related to the notebook format, but a bug in the notebook frontend, which puts the wrong information in plaintext fields. However, one way to 'fix' the problem is to define this bug into the current notebook format, and increment the format with no changes. We then edit v2, and patch it to perform the escape/unescape around the files themselves. I do not think this is worth it, but it is an option.
Pinging @ellisonbg as notebook format coder-in-chief.
Would it be a possible third option to increment the nb format without adding the conversion code for now? Then, if someone decides that it is worth the extra code (and possibly volunteers to contribute it), one could already know which notebooks need which treatment.
incrementing the nbformat without adding the conversion code would mean that all your notebooks would immediately be considered unreadable, because they are the old format.
I suppose that's imprecise - If we define the conversion as a no-op initially, then they would be readable and the old notebooks would be identifiable as you describe. New notebooks would be rejected as unreadable by IPython 0.12, and there's no way that updating the nbformat version would not cause that to happen.
I would consider this the highest priority bug found in 0.12 so far, and the principal motivation for an 0.12.1 bugfix release, depending on which approach we chose.
I will try to have a look at this one.
I also agree that we shouldn't do a format number change unless it's really necessary, and in this case it isn't. Changing the nb format number has a pretty high cost, and we're starting to have a lot of users in the field, so we shouldn't make things hard for them without careful consideration of the benefit.
It's a bummer that this one is going to produce some glitches on load across this merge point, but there doesn't seem to be a way to avoid that...
@ellisonbg @fperez any further thoughts on this one? I think it's the biggest bug in 0.12, and the fix has been outstanding for a over a month.
@fperez - there is a way to avoid the glitches - defining the bug into the notebook format, and incrementing the version, but I would put the costs of that (unreadability of 0.12.1 notebooks in 0.12) as much higher than the glitches themselves, which are trivially resolved by rerunning the notebook (works both directions).
I would vote for fixing the bug in the frontend (as done here), and pushing out 0.12.1 as a critical bugfix ASAP.
Should we apply this to master and cherry-pick it onto 0.12.1? I don't like cutting 0.12.1 in the middle of a bunch of other work that's being done, the notebook UI has changed significantly, etc. So I think our options are:
I actually think I'd prefer the latter. The notebook is so much nicer with the menus and the codemirror fixes we just merged, that I think it more than warrants being called 0.13. We could consider making a release in a 2-3 week timeframe, which would also be ideal for PyCon. Thoughts?
Great idea. I also think that the new notebook is much nicer (polished) than the 0.12 one, and that releasing it soon and as 0.13 would be warranted.
BTW: In case someone thinks that the polished appearance would be diminished by doubly-escaped HTML, how about detecting this and adding a notification (similar to modern browser’s "do you want to save this password?" bars at the top) suggesting to the user to re-evaluate all cells? Again, I am not suggesting that it’s worth it, but just wanted to throw in the thought.
If we are doing 0.13 by March, maybe it doesn't make sense anymore to do 0.12.1, which we probably should have done a month ago, closer to when the issue came up.
regenerate example notebooks to remove transformed output
Reviewed on IRC by @ellisonbg, and merging now.