Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Handle carriage return characters ("\r") in HTML notebook output. #1659

Merged
merged 3 commits into from

4 participants

@mdboom

It does not currently save/restore notebooks correctly. Any pointers on that?

@mdboom

To reproduce:

import time
for i in range(10):
    print ("\r" + "#" * i),
    time.sleep(0.1)
print "Step 2"
for i in range(10):
    print ("\r" + "#" * i),
    time.sleep(0.1)

should produce:

#########
Step 2
#########
IPython/frontend/html/notebook/static/js/utils.js
@@ -74,6 +73,16 @@ IPython.utils = (function (IPython) {
return txt;
}
+ // Remove chunks that should be overridden by the effect carriage

"effect of" or "effect from" probably, right?

@mdboom
mdboom added a note

Indeed. "effect of" is what I intended.

@minrk Owner
minrk added a note

Did you want to make this change?

@mdboom
mdboom added a note

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jasongrout

Thanks for bringing this up; we have the same problem on aleph.sagemath.org with our use of the IPython messaging protocol.

@minrk
Owner

I had a look at the save/restore, and it's a bit problematic.

When we write the ipynb to JSON, we use splitlines() for the text. So roundtrip to files, '\r#' becomes \n#. The change needed for this is to use splitlines(True), and ''.join(lines) instead of splitlines() and '\n'.join(lines). I believe I have a safe fix that will behave properly for notebooks written before/after the fix, but I will have to test carefully to make sure it doesn't cause any problems.

I do want to make a point of discouraging actually using terminal-centric behaviors in the notebook (\r, and ANSI colors, etc.). It's important that people don't think the notebook is a terminal emulator, and it never will be, nor should it be. But \r is simple enough.

IPython/frontend/html/notebook/static/js/codecell.js
@@ -623,7 +623,8 @@ var IPython = (function (IPython) {
if (json.stream == undefined){
json.stream = 'stdout';
}
- if (!utils.fixConsole(json.text)){
+ var text = utils.fixConsole(json.text);
+ if (!text){
@minrk Owner
minrk added a note

This check depended on fixConsole stripping \r. You will have to make a different fix that prevents creation of the HTML element if there is no text to display.

@mdboom
mdboom added a note

The problem I ran into is that you have to do something now if text == '\r', is the last line of the pre-existing content needs to be erased, so you can't just return anymore in that case. The line removal could be done later, I suppose, if all we had was '\r', but care would need to be taken to not throw the '\r' away.

@minrk Owner
minrk added a note

right - the check needs to be later, then.

Note that there is only one problem case:

Creating a new div, with no content, so that's the situation to fix. If it enters the 'append to latest ouput' block, the problem is already avoided.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@minrk minrk referenced this pull request
Merged

Keep line-endings in ipynb #1663

@minrk
Owner

After PR #1663, this should load from saves properly without further changes.

@mdboom mdboom Fix the "test for nothing was streamed" so it doesn't add empty eleme…
…nts -- but only when there wasn't already something there.
13d934d
@mdboom

I think I've corrected the test in which to not add a div when the content is empty. This merged with #1663 also fixes the saving/loading issue.

Thanks for helping with this!

@minrk minrk merged commit cb4d9d8 into ipython:master
@minrk
Owner

thanks, merged.

@minrk minrk referenced this pull request from a commit
@minrk minrk revert PR #1659
caused critical problems with subprocess output.

See `!ls` for an example.
e921622
@minrk
Owner

Crap - I merged this prematurely, and it caused a pretty critical failure (try !ls, and you will see no output at all). I reverted the original patch, so can you submit a new PR after testing the output of !ls?

@takluyver
Owner

Hang on, I've just understood this issue. So if there's a percentage counter that gets overwritten 100 times, we store every percentage, separated by \r characters? That seems unnecessary. Shouldn't we handle \r for display, but then just save the final state, as the user sees it when they hit save?

@minrk
Owner

Correct, the notebook structure is a transcript of what actually happened, excluding any UI-specific interpretation. \r is a frontend-specific special character, which is interpreted (in this PR) in the notebook as clearing the preceding line, which is distinct from the terminal, where it simply moves the cursor, and discards no information until overwrites start happening.

This is a difference between \r and clear_output, and further reason why real notebook operations are better than faked terminal-behavior in the notebook. This is acceptable to me, because I consider \r support like this decidedly second-class, and have no problem with it suffering such a disadvantage.

We could store the transformed output, but we would have to be careful that we do not store the escaped HTML output (we had this bug before). If we make that change, then we must do fixCR always on the raw text and before fixConsole, because we would be saving the results of fixCR, and must never save the results of fixConsole.

@takluyver
Owner

I'd side with removing overwritten content at some point before saving. Even if we'd rather they didn't, people are going to use \r for progress counters, because it works and library authors shouldn't have to detect and handle the notebook. If it's left in, it's extra noise for version control. Also, if one day another application can load ipynb files, but can't handle \r, its users won't appreciate seeing a load of messy progress information.

@minrk
Owner

Do you want to add that as a criterion for PR #1674 (note that this one is already closed)?

@minrk
Owner

I think it does make sense to do, my main point is that I don't think it's worth the effort. If someone does feel like doing it, they are welcome, but must be very careful with the points I mentioned.

@jasongrout jasongrout referenced this pull request in sagemath/sagecell
Closed

Handle carriage returns correctly #296

@mattvonrocketstein mattvonrocketstein referenced this pull request from a commit in mattvonrocketstein/ipython
@minrk minrk revert PR #1659
caused critical problems with subprocess output.

See `!ls` for an example.
06158dd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 25, 2012
  1. @mdboom
Commits on Apr 26, 2012
  1. @mdboom

    Fix the "test for nothing was streamed" so it doesn't add empty eleme…

    mdboom authored
    …nts -- but only when there wasn't already something there.
Commits on Apr 27, 2012
  1. @mdboom

    Fix typo in comment

    mdboom authored
This page is out of date. Refresh to see the latest.
View
23 IPython/frontend/html/notebook/static/js/codecell.js
@@ -623,11 +623,8 @@ var IPython = (function (IPython) {
if (json.stream == undefined){
json.stream = 'stdout';
}
- if (!utils.fixConsole(json.text)){
- // fixConsole gives nothing (empty string, \r, etc.)
- // so don't append any elements, which might add undesirable space
- return;
- }
+
+ var text = utils.fixConsole(json.text);
var subclass = "output_"+json.stream;
if (this.outputs.length > 0){
// have at least one output to consider
@@ -636,15 +633,22 @@ var IPython = (function (IPython) {
// latest output was in the same stream,
// so append directly into its pre tag
// escape ANSI & HTML specials:
- var text = utils.fixConsole(json.text);
- this.element.find('div.'+subclass).last().find('pre').append(text);
+ pre = this.element.find('div.'+subclass).last().find('pre');
+ text = utils.fixCarriageReturn(pre.text() + text);
+ pre.text(text);
return;
}
}
-
+
+ if (!text.replace("\r", "")) {
+ // text is nothing (empty string, \r, etc.)
+ // so don't append any elements, which might add undesirable space
+ return;
+ }
+
// If we got here, attach a new div
var toinsert = this.create_output_area();
- this.append_text(json.text, toinsert, "output_stream "+subclass);
+ this.append_text(text, toinsert, "output_stream "+subclass);
this.element.find('div.output').append(toinsert);
};
@@ -702,6 +706,7 @@ var IPython = (function (IPython) {
var toinsert = $("<div/>").addClass("box_flex1 output_subarea output_text");
// escape ANSI & HTML specials in plaintext:
data = utils.fixConsole(data);
+ data = utils.fixCarriageReturn(data);
if (extra_class){
toinsert.addClass(extra_class);
}
View
4 IPython/frontend/html/notebook/static/js/pager.js
@@ -89,9 +89,9 @@ var IPython = (function (IPython) {
Pager.prototype.append_text = function (text) {
var toinsert = $("<div/>").addClass("output_area output_stream");
- toinsert.append($('<pre/>').html(utils.fixConsole(text)));
+ toinsert.append($('<pre/>').html(utils.fixCarriageReturn(utils.fixConsole(text))));
this.pager_element.append(toinsert);
- };
+ };
IPython.Pager = Pager;
View
17 IPython/frontend/html/notebook/static/js/utils.js
@@ -47,7 +47,7 @@ IPython.utils = (function (IPython) {
"37":"ansigrey", "01":"ansibold"
};
- // Transform ANI color escape codes into HTML <span> tags with css
+ // Transform ANSI color escape codes into HTML <span> tags with css
// classes listed in the above ansi_colormap object. The actual color used
// are set in the css file.
function fixConsole(txt) {
@@ -57,8 +57,6 @@ IPython.utils = (function (IPython) {
var cmds = [];
var opener = "";
var closer = "";
- // \r does nothing, so shouldn't be included
- txt = txt.replace('\r', '');
while (re.test(txt)) {
var cmds = txt.match(re)[1].split(";");
closer = opened?"</span>":"";
@@ -74,6 +72,16 @@ IPython.utils = (function (IPython) {
return txt;
}
+ // Remove chunks that should be overridden by the effect of
+ // carriage return characters
+ function fixCarriageReturn(txt) {
+ tmp = txt;
+ do {
+ txt = tmp;
+ tmp = txt.replace(/^.*\r/gm, '');
+ } while (tmp.length < txt.length);
+ return txt;
+ }
grow = function(element) {
// Grow the cell by hand. This is used upon reloading from JSON, when the
@@ -95,7 +103,8 @@ IPython.utils = (function (IPython) {
return {
uuid : uuid,
fixConsole : fixConsole,
- grow : grow
+ grow : grow,
+ fixCarriageReturn : fixCarriageReturn
};
}(IPython));
Something went wrong with that request. Please try again.