IDLE: Document how Shell displays user code output

[AlSweigart]

BPO	23220
Nosy	@terryjreedy, @taleinat, @ned-deily, @4kir4, @vadmium, @serhiy-storchaka, @willingc, @miss-islington
PRs	bpo-23220: Explain how IDLE's Shell displays output #10356 [3.7] bpo-23220: Explain how IDLE's Shell displays output (GH-10356) #10365 [3.6] bpo-23220: Explain how IDLE's Shell displays output (GH-10356) #10366 [3.7] bpo-23220: Explain how IDLE's Shell displays output (GH-10356) #10368 [3.6] bpo-23220: Explain how IDLE's Shell displays output (GH-10356) #10369
Dependencies	bpo-33000: IDLE Doc: Text consumes unlimited RAM, consoles likely not

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2018-11-10.23:34:26.296>
created_at = <Date 2015-01-11.00:54:47.152>
labels = ['3.8', 'expert-IDLE', 'type-feature', '3.7', 'docs']
title = 'IDLE: Document how Shell displays user code output'
updated_at = <Date 2019-08-12.07:00:43.640>
user = 'https://bugs.python.org/AlSweigart'

bugs.python.org fields:

activity = <Date 2019-08-12.07:00:43.640>
actor = 'taleinat'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2018-11-10.23:34:26.296>
closer = 'terry.reedy'
components = ['Documentation', 'IDLE']
creation = <Date 2015-01-11.00:54:47.152>
creator = 'Al.Sweigart'
dependencies = ['33000']
files = []
hgrepos = []
issue_num = 23220
keywords = ['patch']
message_count = 28.0
messages = ['233828', '233830', '233831', '233832', '233838', '233862', '234162', '246596', '246602', '246614', '246626', '246629', '246633', '246673', '251825', '261528', '261529', '261531', '261738', '328856', '329370', '329373', '329374', '329391', '329650', '349392', '349438', '349442']
nosy_count = 11.0
nosy_names = ['terry.reedy', 'taleinat', 'ned.deily', 'THRlWiTi', 'docs@python', 'akira', 'martin.panter', 'Al.Sweigart', 'serhiy.storchaka', 'willingc', 'miss-islington']
pr_nums = ['10356', '10365', '10366', '10368', '10369']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23220'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[AlSweigart]

IDLE cannot display the \b backspace character correctly. From IDLE's interactive shell:

>>> print('hello\b\b\b\b\bHELLO')
hello�������◙◙◙◙◙HELLO

Whereas from cmd:

>>> print('hello\b\b\b\b\bHELLO')
HELLO

[willingc]

FWIW: On Mac OS X 10.9.5 using Python 3.4.2, IDLE's interactive shell (started from the IDLE.app icon):

>>print('hello\b\b\b\b\bHELLO')
helloHELLO

From the command line using python3 interactive shell:

>>print('hello\b\b\b\b\bHELLO')
HELLO

Both return a <class 'str'> for type('hello\b\b\b\b\bHELLO')
Interestingly, both behave the same when executing:

>>'hello\b\b\b\b\bHELLO'
'hello\x08\x08\x08\x08\x08HELLO'

I'm not sure that IDLE is used much on OS X since the Terminal is easily available. Since K12 education may use it, it would be nice to have consistency across the OSes.

[vadmium]

As far as I understand, Idle doesn’t interpret any terminal control codes apart from a plain \n for a new line. I know it doesn’t do a carriage return for \r either.

[AlSweigart]

For clarification, this happens on Windows 7.

[ned-deily]

It's helpful to keep in mind that IDLE is a Tk application written in Python and, as such, depends on Tk to do nearly everything associated with writing to displays and reading from keyboards and pointing devices. If you try outputting the same string using the Tk shell, wish, you'll see exactly the same behavior as you do in IDLE:

$ wish
% .text insert 1.0 "hello\b\b\b\b\bHELLO"
% pack .text

And you'll see the same Tk platform differences. With the native OS X Tk's, the backspace characters aren't displayed; with a Linux or OS X X11 Tk, empty box characters are displayed. That may also depend on which font is in use. In any case, the Tk text widget does not behave the same way in this regard as most terminal emulator windows do would do with backspace characters. So it's up to any Tk app to figure out how it wants to deal with them.

This issue has come up before for Tkinter in general: back in 2001, effbot suggested some code to search for and edit backspace runs in Tk text. Presumably something similar could be added to IDLE if there was general agreement that this was desirable (after verifying that it works on all of the Tk platforms that IDLE supports). See https://mail.python.org/pipermail/python-list/2001-December/085908.html

[willingc]

Ned, Thanks for the detailed example and confirming my gut instinct that Tk was the root cause of the differences seen between the IDLE's Python interactive shell (https://docs.python.org/3.4/library/idle.html) and the interactive interpreter invoked from the command line (https://docs.python.org/3.4/tutorial/interpreter.html#tut-invoking).

As an end user learning Python (such as the elementary education market), the current Standard Library documentation on IDLE guides me to the incorrect conclusion that the "Python shell window (aka interactive interpreter)" in IDLE would behave the same as invoking the interactive interpreter from the command line.

It seems reasonable to explicitly state in the Standard Library doc that:
"In rare cases, such as text handling with certain special characters (i.e. '\b' in a string), the IDLE's interactive Python shell may return a different response than the Python interactive interpreter invoked from the command line. This is due to IDLE's low level dependence on Tk (Tk itself is not part of Python; it is maintained at ActiveState. reference: first paragraph of https://docs.python.org/3.4/library/tkinter.html#module-tkinter)."

[terryjreedy]

Going back to msdos, there are graphic chars for all 256 bytes, including may single and double line box-drawing chars. Many In Idle, I see 5 solid white circles. In FireFox, there are 5 empty circles (on dark background, which are chr(9689). When I copy from FF back to Idle (3.4.2, Win7), there are 5 of each. I have no idea if the 9689s are on the site or added by FF.

Here is another difference.

>>> print('\x03')  # console
♥  # heart

>>> print('\x03')  # idle
�  # lower left single line corner in Idle, box on FF

Trying to match console-Idle(tk) print output for control chars even on Windows would be tough.
---

I have been planning to add a subsection of the doc that mentions known differences between console interpreter and Idle shell. The result of print() is one of them. Another print difference is that Idle displays many unicode chars that Windows replaces with boxes or ?s, depending on the codepage.

[terryjreedy]

I closed bpo-24572 as a duplicate of this. It is the same issue except for printing \r instead of \b. These issues are not about responses to keyboard actions when entering text into an Idle editor window. During entry, just about any cntl/alt/function/shift/key combination can be intercepted to and translated to arbitrary action. They are also not about REPL echo. For 3.4+ Win7, both console Python and Shell echo '\b\t\x08\x09' as '\x08\t\x08\t'. Carol's report suggest that the same is true on Mac also.

Both issues *are* about print(s, file=outfile), where s is the result of processing all args except 'file' and outfile defaults to sys.stdout. The print call is the same as outfile.write(s), so outfile (and sys.stdout) could be any object with a write method.

>>> print('hello\b\b\b\b\bHELLO')
hello�����HELLO
>>> import sys; sys.stdout.write('hello\b\b\b\b\bHELLO'+'\n')
hello�����HELLO
(I actually see the same circles as Al, but copy-paste does not seem to work as well for me.)

So both issues are about the effect of writing 'control chars', in particular \b and \r, to a file. Well, that depends on the file. Some possibilities are copy as is (StringIO), encode and copy (disk file, socket), ignore, display as one glyph, display as multiple chars, non-destructively backspace (like backspace on typewriters and printing terminals and left-arrow elsewhere), or destructively backspace (like backspace on as most (all?) screen keyboards). After non-destructive movement of the 'cursor' position, the possibilities for following graphical chars are overwrite (like typewriters), replace, and insert (the modes sometimes selected by the Insert key). Non-destructive backspace followed by overwrite (meaning 'HELLO' printed on top of 'hello') is the original meaning of 'backspace'.

Having said all this, I am sympathetic to the idea that there should be an option to have 'print(ascii_string)' in user code give the same result in the console and Idle. I think this would best be accomplished by a least-common-denominator SimpleTerm subclass of tkinter.Text put somewhere in the tkinter package. (This would be a new issue that should start on python-ideas list.) However, I would consider something Idle specific.

Does the following work the same on *nix and Mac as Windows?

>>> print('hello\rHELLO')
HELLO  # helloHELLO with Win7 Idle

Are there any control-chars (other than \n) that work might work in the consoles on all three systems and that should be included?

Carol, another difference between the Windows console and Idle is that tk and hence Idle support the entire BMP subset of unicode. This should also be mentioned in the doc.

[serhiy-storchaka]

Control characters are named control characters because they are control the output device. Different devices have different capabilities and reacts different on the same codes. Windows console, different sorts of Linux terminals and Tk text widget are different devices. Some prints funny characters, others not, some beeps, others not, some interprets particular flavor of ESC sequences, others not, some allows color and positioning, others not. Python can't unify the behavior of these devices without lost most of functionality as it can't unify the behavior of black-white matrix printer, graphical plotter and 24-bit color LCD monitor.

I would close this issue as not related to Python.

[terryjreedy]

Serhiy, thanks for the paraphrase of what I tried to say. This issue has already been changed to a doc issue to explain the input/output effects of Idle's way of running user code. I should have retitled before. Control codes are just one of the effects that have come up on the tracker, python-list, or stackoverflow. Misunderstanding had lead to user puzzlement and even undeserved insults directed as Python.

I propose to something like the following to the introductory section of the Idle doc, https://docs.python.org/3/library/idle.html#index-0, after the feature list.

---
When Idle runs user code from either the shell or an editor window, the result is nearly always be the same as running the same code directly with python in either interactive or batch mode. However, environmental differences can have visible effects. For instance, Idle import statements add numerous entries to sys.modules and a few module attributes to modules such as tkinter. The line `1import tkinter; tkinter.colorchoosernormally raises AttributeError but works when run with Idle because Idle has already imported tkinter.colorchooser. To detect whether code is running under Idle, runimport idlelib; idlelib.run`` within a try-except statement.

More important are the input/output differences. Python normally runs in a text console process with direct access to keyboard and screen. For code run with Idle, the keyboard and screen are controlled by the tk graphics subsystem and sys.stdin, sys.stdout, and sys.stderr are bound to objects that connect to the gui. (Note that print calls sys.stdout.write.) Here are some of the effects of keyboard and screen access being indirect.

• Operating system (OS) specific functions that directly access the keyboard may not work.
• The Idle shell works with complete statements rather than individual lines of code. One can edit and retrieve complete multiline statements instead of single lines.
• User code gets colorized. Normal output and error output to the shell get their own colors.
• Tk supports the Basic Multilingual Plane (BMP) subset of Unicode characters. This is worse than consoles that support supplementary planes (with appropriate fonts in use), but better than the Windows console, which only supports restricted subsets of the BMP, depending on the code page in use.
• Tk handling of ascii control chars depends on the OS and is usually different from the text console.
  ---

[vadmium]

“run import idlelib; idlelib.run within a try-except statement”: It might be nice to say what exceptions are expected. My guess is ImportError if Idle or TK is not available, or AttributeError if it is but Idle is not running.

“Tk handling of ascii control chars”: I presume you mean stdout and stderr handling, which this bug was originally about (or also input, source code, etc as well?). It might be good to say that output of Unix newlines (\n) is guaranteed to be supported. Also might be worth explicitly pointing out that output of CRLFs is not supported, even if os.linesep is "\r\n". In my experiments between Linux and Wine, this does not appear to depend on the OS.

[terryjreedy]

I was thinking AttributeError, as mentioned in the previous sentence. But you are correct that ImportError is possible too. Maybe I should just give the code.

try:
import idlelib
idlelib.run
running_idle = True
except (ImportError, AttributeError):
running_idle = False

tk does not 'handle' stdout. Idle does, by inserting strings into a tk text widget. tk does not care where inserted chars come from. tk \b behavior is OS dependent. tk may always ignore \r, but this is different from (at least some) consoles. Attempt 2, with and added paragraph.

...

• Except for newline ('\n'), tk handling of ascii control chars may depend on the OS and may by different from text consoles. Both are true for backspace ('\b') and the latter for return ('\r'), which tk ignores.

These differences noted above are not bugs. If an application is going to be run repeatedly after being developed with Idle, it should usually be run directly, without Idle. (An exception would be non-gui Windows apps that need tk's better unicode support.) That means testing at least once without Idle.
---

Thanks for the comments.

[vadmium]

I wouldn’t say TK ignores carriage returns, though I agree it would be better if Idle stripped them out. Currently I get a glyph displayed for them, similarly to \b. They wouldn’t copy to my clipboard, so I fudged them after pasting here:

>>> _ = stdout.write("CRLF\r\n")  # Linux: box-drawing corner piece
CRLF┌
>>> _ = stdout.write("CRLF\r\n")  # Wine: euro sign
CRLF€

[terryjreedy]

OK. "Both are true for backspace ('\b') and return ('\r')."

[terryjreedy]

Issue bpo-21995 discussed same issue from cause side, as opposed to result. The new section is partly based on what I wrote above. I am not satisfied with it yet.

New changeset ac6ade0c5927 by Terry Jan Reedy in branch '2.7':
bpo-21995: Explain some differences between IDLE and console Python.
https://hg.python.org/cpython/rev/ac6ade0c5927

New changeset ca6c9cc77c20 by Terry Jan Reedy in branch '3.4':
bpo-21995: Explain some differences between IDLE and console Python.
https://hg.python.org/cpython/rev/ca6c9cc77c20

[terryjreedy]

Different \r behavior is the gist of https://stackoverflow.com/questions/35895864/what-is-the-difference-between-cmd-and-idle-when-using-tqdm. The new section should have more on the different effect of control characters on different display devices and windows. It should also better differentiate between Python result, which is the string written to stdout, and which is nearly always the same in IDLE and console, and visual display result, which is outside of Python's control. This separation is the point of Serhiy's msg246602.

[terryjreedy]

Part of this issue is the difference between typing a character at the keyboard, which produces a key event whose effect is determined by key bindings (some default, some added) and inserting a character into a Text widget, which does not produce an event (except, in some sense, \t and \n). Whether or not a key event results in character insertion depends on key bindings. The visual effect of inserted chars other than \t and \n is determined by the font. (I believe I got the above right.) I believe some terminals treat chars coming over the wire the same as those typed. I am not sure what the Win10 console is doing with control-x keystrokes.

>>> import sys; out=sys.stdout.write
# use direct write to avoid str(), repr() conversion of control chars
>>> out('\x03')
�1  # The first char is an empty thin box in my IDLE.
>>> # type ^C
KeyboardInterrupt

[serhiy-storchaka]

As for StackOverflow question, there is tkinter.ttk.Progressbar.

[4kir4]

IDLE can implement functionality similar to what colorama [1] module does on Windows: translate ANSI escape character sequences into corresponding GUI method calls.

For example, \b might be implemented using a .delete() call, \r using .mark_set(), etc.

[1] https://pypi.python.org/pypi/colorama

[terryjreedy]

While editing 'IDLE-console differences' (added in the patch above) for bpo-35099, I decided that it should be renamed (to, say, 'Executing user code') and limited to the effect of executing user code in IDLE's execution process instead of a standard python process. Except for a intentional change to make developing tkinter code easier, such effects are unintended and possibly bad for developers using IDLE.

This issue is about one aspect of how IDLE's Shell, in the GUI process, displays strings and bytes sent from the execution process via sys.stdout and sys.stderr. My previous revision of the title was wrong and my previous patch not directly relevant.

I will make a new patch to add a new section ('Displaying user code output'?) describing various aspects of how Shell displays std text output, starting with this one. Most of the differences from various consoles and terminal are intentional and considered to be net positives.

While the handling of control characters is inherited from tk, Serhiy and I have claimed on other issues that displaying one glyph per character has advantages for development as well as disadvantages. (Being about to display all but only the BMP subset of unicode is also inherited from tcl/tk.)

One of my unposted ideas is to add an option to the run menu to run edited code in the system console/terminal, detached from IDLE, the same as if the file were run from a command line or directory listing. This would bypass both IDLE's execution and display effects.

[terryjreedy]

New changeset 75d9d59 by Terry Jan Reedy in branch 'master':
bpo-23220: Explain how IDLE's Shell displays output (GH-10356)
75d9d59

[miss-islington]

New changeset 34fcee9 by Miss Islington (bot) in branch '3.7':
bpo-23220: Explain how IDLE's Shell displays output (GH-10356)
34fcee9

[terryjreedy]

New changeset 7476fef by Terry Jan Reedy (Miss Islington (bot)) in branch '3.6':
bpo-23220: Explain how IDLE's Shell displays output (GH-10356) (bpo-10369)
7476fef

[terryjreedy]

I merged a first edition of the new section. It does not include Mac behavior, so will need revision. But I want to do some experiments with tk/inter on various systems before doing so.

[terryjreedy]

In the patch for bpo-33000, to cover MacOS behavior, I changed the comment about control chars from 'replaced' to 'replaced or deleted' Users can fill in the details by comparing IDLE on their system to a particular console or terminal.

[taleinat]

See initial working implementation of handling \r and \b characters is the shell, PR #59416.

[terryjreedy]

The conclusion of this discussion was that we should insert control chars, in particular \b and \r, as are into tk.Text and document the (variable) result. Please describe your proposed new behavior, how it differs from the current behavior, and why you think it better. I think that this should be a new issue.

[taleinat]

Ah, sorry for missing that, Terry. See new issue, bpo-37827, for the proposed change.