semicolon doen't suppress output with ast_node_interactivity="all"

[liushapku]

In an ipython terminal with c.InteractiveShell.ast_node_interactivity="all", the following code

1;
2;
if 0:
  print('3')

will output

1
2

• Without the if-block, the output is suppressed.
• With c.InteractiveShell.ast_node_interactivity="last_expr", the output is suppressed.

[pmav99]

I did have a look at this and I think that output suppression is "broken" in general when multiline input is involved and not just in ast_node_interactivity = 'all'. I put "broken" in quotes because according to this comment output suppression only works for the last top-level expression. Unfortunately, even though @Carreau's comment is accurate, this is not documented so I am not sure it is considered a bug or not.

Issues #2493 and #6130 are probably related. Until this is fixed, a fix to the docs would not be bad idea.

Rationale

IMHO ast_node_interactivity = "all" + output suppression is especially useful in multiline cells. E.g. you can check the value of a variable inside a loop without using a print() and then you just add a semicolon to suppress it (but hey, that's just me :) ).

Breakdown

The relevant function for output suppression is quiet(). To help debug this issue, I added a print(token) in the for loop (line 98). In the following output, I will omit the debug output when it is not relevant.

Disclaimer: The "expected outputs" of the following test cases showcase how I would expect this to work; you don't necessarily have to agree with this.

1st test case

a = 1
b = 2
a    # out1
a;   # out2
b    # out3

Expected output:

• last_expr -> out3
• all -> out1 & out3

2nd test case

a = 1
b = 2
a    # out1
a    # out2
b;   # out3

Expected output:

• last_expr -> out2
• all -> out2 & out3

last_expr

[I] ➜ ipython
Python 3.7.1 (default, Oct 22 2018, 10:41:28) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help.

[ins] In [1]: from IPython.core.interactiveshell import InteractiveShell                                                                                                                      

[ins] In [2]: InteractiveShell.ast_node_interactivity.default_value
Out[2]: 'ast_node_interactivity = last_expr'

1st test case

[ins] In [6]: a = 1 
         ...: b = 2 
         ...: a    # out1 
         ...: a;   # out2 
         ...: b    # out3                                                                                                                                                                     
TokenInfo(type=0 (ENDMARKER), string='', start=(6, 0), end=(6, 0), line='')
TokenInfo(type=4 (NEWLINE), string='', start=(5, 11), end=(5, 12), line='')
TokenInfo(type=55 (COMMENT), string='# out3', start=(5, 5), end=(5, 11), line='b    # out3')
TokenInfo(type=1 (NAME), string='b', start=(5, 0), end=(5, 1), line='b    # out3')
Out[6]: 2

The behaviour is correct, i.e. only out3 is being shown.

2nd test case

[nav] In [7]: a = 1 
         ...: b = 2 
         ...: a    # out1 
         ...: a    # out2 
         ...: b;   # out3                                                                                                                                                                     
TokenInfo(type=0 (ENDMARKER), string='', start=(6, 0), end=(6, 0), line='')
TokenInfo(type=4 (NEWLINE), string='', start=(5, 11), end=(5, 12), line='')
TokenInfo(type=55 (COMMENT), string='# out3', start=(5, 5), end=(5, 11), line='b;   # out3')
TokenInfo(type=53 (OP), string=';', start=(5, 1), end=(5, 2), line='b;   # out3')

In this case, the behaviour is wrong, i.e. we get no output instead of out2.

ast_node_interactivity = all

[ins] In [10]: InteractiveShell.ast_node_interactivity = "all"                                                                                                                                

1st test case

[ins] In [11]: a = 1 
          ...: b = 2 
          ...: a    # out1 
          ...: a;   # out2 
          ...: b    # out3                                                                                                                                                                    
TokenInfo(type=0 (ENDMARKER), string='', start=(6, 0), end=(6, 0), line='')
TokenInfo(type=4 (NEWLINE), string='', start=(5, 11), end=(5, 12), line='')
TokenInfo(type=55 (COMMENT), string='# out3', start=(5, 5), end=(5, 11), line='b    # out3')
TokenInfo(type=1 (NAME), string='b', start=(5, 0), end=(5, 1), line='b    # out3')
Out[11]: 1
TokenInfo(type=0 (ENDMARKER), string='', start=(6, 0), end=(6, 0), line='')
TokenInfo(type=4 (NEWLINE), string='', start=(5, 11), end=(5, 12), line='')
TokenInfo(type=55 (COMMENT), string='# out3', start=(5, 5), end=(5, 11), line='b    # out3')
TokenInfo(type=1 (NAME), string='b', start=(5, 0), end=(5, 1), line='b    # out3')
Out[11]: 1
TokenInfo(type=0 (ENDMARKER), string='', start=(6, 0), end=(6, 0), line='')
TokenInfo(type=4 (NEWLINE), string='', start=(5, 11), end=(5, 12), line='')
TokenInfo(type=55 (COMMENT), string='# out3', start=(5, 5), end=(5, 11), line='b    # out3')
TokenInfo(type=1 (NAME), string='b', start=(5, 0), end=(5, 1), line='b    # out3')
Out[11]: 2

The behaviour is wrong, i.e. we should only get out1 and out3 but we get out2 too.

The interesting thing here is that the cell is (correctly) being tokenized 3 times, but each time the tokenization starts from the bottom of the cell (i.e. b). As a result, the output is not suppressed.

2nd test case

[ins] In [12]: a = 1 
          ...: b = 2 
          ...: a    # out1 
          ...: a    # out2 
          ...: b;   # out3                                                                                                                                                                    
TokenInfo(type=0 (ENDMARKER), string='', start=(6, 0), end=(6, 0), line='')
TokenInfo(type=4 (NEWLINE), string='', start=(5, 11), end=(5, 12), line='')
TokenInfo(type=55 (COMMENT), string='# out3', start=(5, 5), end=(5, 11), line='b;   # out3')
TokenInfo(type=53 (OP), string=';', start=(5, 1), end=(5, 2), line='b;   # out3')
TokenInfo(type=0 (ENDMARKER), string='', start=(6, 0), end=(6, 0), line='')
TokenInfo(type=4 (NEWLINE), string='', start=(5, 11), end=(5, 12), line='')
TokenInfo(type=55 (COMMENT), string='# out3', start=(5, 5), end=(5, 11), line='b;   # out3')
TokenInfo(type=53 (OP), string=';', start=(5, 1), end=(5, 2), line='b;   # out3')
TokenInfo(type=0 (ENDMARKER), string='', start=(6, 0), end=(6, 0), line='')
TokenInfo(type=4 (NEWLINE), string='', start=(5, 11), end=(5, 12), line='')
TokenInfo(type=55 (COMMENT), string='# out3', start=(5, 5), end=(5, 11), line='b;   # out3')
TokenInfo(type=53 (OP), string=';', start=(5, 1), end=(5, 2), line='b;   # out3')

The behaviour is again wrong, i.e. instead of only suppressing out3, we get no output.

In this case the cell is also being tokenized 3 times, but each time the tokenization starts from the bottom of the cell (i.e. b). As a result, the output out1 and out2 are suppressed, even thought they shouldn't.

Conclusion(?)

If I understand this correctly, the "problem" is that the tokenization of the cell always starts from the bottom node and not from the interactive node (not sure if the terms are correct). If someone provides some guidance on how to tackle this, I could have a look at this.

[achampion]

It would be useful to get clarity on what the expected behaviour is. My preference would be, especially with ast_node_interactivity="all", to suppress output from any line with a ; in the multi-line block. One of the arguments I've also seen mentioned is that this also doesn't create a reference to it in the output history allowing the output to be garbage collected sooner.

[Carreau]

I think the behavior is "best effort", if we find a better implementation that we'll be happy to get it.
The semi-colon is mostly use to suppress the last output of cells, there is no particular expected behavior beyond this I believe.