readline 8.1 enables the bracketed paste mode by default

[dtrodrigues]

BPO	42819
Nosy	@Yhg1s, @gpshead, @vstinner, @encukou, @hroncok, @miss-islington, @asmeurer, @fxcoudert, @dtrodrigues
PRs	bpo-42819: disable Readline bracketed paste #24108 [3.9] bpo-42819, readline: Disable bracketed paste (GH-24108) #24545 [3.8] bpo-42819, readline: Disable bracketed paste (GH-24108) #24546

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-02-16.00:19:11.508>
created_at = <Date 2021-01-03.20:52:35.880>
labels = ['type-bug', '3.8', '3.9', '3.10']
title = 'readline 8.1 enables the bracketed paste mode by default'
updated_at = <Date 2021-02-16.00:19:11.500>
user = 'https://github.com/dtrodrigues'

bugs.python.org fields:

activity = <Date 2021-02-16.00:19:11.500>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-02-16.00:19:11.508>
closer = 'vstinner'
components = []
creation = <Date 2021-01-03.20:52:35.880>
creator = 'dtrodrigues'
dependencies = []
files = []
hgrepos = []
issue_num = 42819
keywords = ['patch']
message_count = 19.0
messages = ['384297', '384330', '385245', '385855', '386929', '386981', '386982', '386987', '386988', '386989', '386993', '386994', '386995', '387061', '387062', '387071', '387072', '387077', '387078']
nosy_count = 9.0
nosy_names = ['twouters', 'gregory.p.smith', 'vstinner', 'petr.viktorin', 'hroncok', 'miss-islington', 'asmeurer', 'fxcoudert', 'dtrodrigues']
pr_nums = ['24108', '24545', '24546']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42819'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[dtrodrigues]

Readline 8.1 enables bracketed paste by default. Package managers like Homebrew for macOS and distributions like Arch Linux which use the latest version of readline (released December 2020) now have new behavior when pasting multiline strings into the python REPL. Disabling bracketed paste on 8.1 reverts to the expected behavior, and enabling bracketed paste on 8.0 also reproduces the behavior. Further information in Homebrew/homebrew-core#68193

Example with bracketed paste on:
$ cat in
1+2
3+4
5+6

$ pbcopy < in
$ /usr/local/Cellar/python\@3.9/3.9.1_3/bin/python3
Python 3.9.1 (default, Dec 28 2020, 11:22:14) 
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+2
3+4
5+6

File "<stdin>", line 1
1+2
3+4
5+6

   ^

SyntaxError: multiple statements found while compiling a single statement

>>

Example with it off:
$ /usr/local/bin/python3
Python 3.9.1 (v3.9.1:1e5d33e9b9, Dec  7 2020, 12:10:52) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+2
3
>>> 3+4
7
>>> 5+6
11
>>>

[fxcoudert]

This bug is present in all versions since the 3.6 branch at least, but was only triggered when users' readline settings explicitly enabled bracketed-paste. Previous reports: https://bugs.python.org/issue32019 and https://bugs.python.org/issue39820 Longer discussion (on a more extensive feature request) at https://bugs.python.org/issue38747

The new factor is that readline 8.1 now enables bracketed-paste by default. So the broken behaviour is seen for Python built with the latest readline, with no specific setting.

Python should probably in the long term be improved to handle bracketed-paste, which will make a nicer user experience. But in the short term, it should communication with readline to disable bracketed-paste at runtime, and restore the previous behaviour.

[dtrodrigues]

The linked PR disables bracketed paste regardless of if the user has it as a configuration option or if Python was configured with a version of readline which defaults to on for bracketed paste. Is this a viable approach?

[dtrodrigues]

Bumping this issue because it's a bug affecting all users who build or use a Python built with the current version of readline, to include all macOS HomeBrew and Arch Linux users who use the python from the respective package managers.

[hroncok]

This also affects Fedora 34+

[vstinner]

Readline 8.1 enables bracketed paste by default.

Can you try to find why it changed the default? Why should Python change the default readline default behavior?

[hroncok]

https://lists.gnu.org/archive/html/bug-readline/2020-11/msg00010.html
https://lists.gnu.org/archive/html/bug-bash/2020-10/msg00048.html
https://lists.gnu.org/archive/html/bug-bash/2020-10/msg00087.html

[vstinner]

I didn't know "bracketed paste mode". It seems like a protocol between a terminal and the user to add markers before/after a pasted text.
https://cirw.in/blog/bracketed-paste

"""
In summary:

1. Enable bracketed paste: printf "\e[?2004h"
2. Wait for paste to start: you’ll see \e[200~ on STDIN.
3. Wait for paste to stop: you’ll see \e[201~ on STDIN.
4. Disable bracketed paste: printf "\e[?2004l"
   """

I understand that this mode prevents to run arbitrary command when pasting blindly an untrusted command copied from a webpage, like this example:
https://thejh.net/misc/website-terminal-copy-paste

It can be enabled explicitly by adding "set enable-bracketed-paste" on to your ~/.inputrc configuration file.

[vstinner]

On my Fedora 33 (readline-8.0-5.fc33.x86_64), I enabled the bracketed paste mode by adding "set enable-bracketed-paste" on to my ~/.inputrc config file.

When I test https://thejh.net/misc/website-terminal-copy-paste : the evil command is no long executed.

If I copy/paste "1+1" and "2+2" commands (two lines pasted at once) manually in the Python REPL, they are no longer executed immediately, Python waits for me pressing ENTER to execute, *as expected*:

----

$ ./python 
Python 3.10.0a5+ (heads/master:fcbe0cb04d, Feb 15 2021, 10:30:10) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+1
2+2

---

But using "./python -i < commands", each line is executed immediately and Python exits after displaying the results:

---

$ cat commands 
1+1
2+2

$ ./python -i < commands
Python 3.10.0a5+ (heads/master:fcbe0cb04d, Feb 15 2021, 10:30:10) 
>>> 2
>>> 4
>>> 

---

If I disable again bracketed mode (ex: remove ~/.inputrc file in my case), when I copy/paste manually the two commands at once, they are executed again, as expected:
---

$ ./python
>>> 1+1
2
>>> 2+2
4

---

The bracketed paste mode can be enabled explicitly in Python with this patch to simulate readline 8.1 default behavior:

diff --git a/Modules/readline.c b/Modules/readline.c
index c900e07954..ce8662c000 100644
--- a/Modules/readline.c
+++ b/Modules/readline.c
@@ -217,6 +217,7 @@ readline_read_init_file_impl(PyObject *module, PyObject *filename_obj)
         errno = rl_read_init_file(NULL);
     if (errno)
         return PyErr_SetFromErrno(PyExc_OSError);
+    rl_variable_bind ("enable-bracketed-paste", "on");
     Py_RETURN_NONE;
 }
 

All of these behavior look consistent, correct and expected.

Note: When I redirect Python output into a file, I cannot see "\e[?2004h" (likely because stdout is not a TTY if it's redirected into a file), so I'm not sure how to check if this escape sequence is injected in Python stdout or not.

[vstinner]

Oh, I forgot to mention that I ran my tests in Gnome Terminal 3.38.1-2.fc33.

[vstinner]

Related issue: test_pdb_interaction_doctest test of pytest fails on Fedora Rawhide because of a "\x1b[?2004h" string:

• The PDB tests fail with Bash in bracketed-paste mode: Unexpected escape sequences in stdout pytest-dev/pytest#8256
• https://bugzilla.redhat.com/show_bug.cgi?id=1892612

It fails with readline 8.1 which enables bracketed mode by default, but it fails with any readline version if ~/.inputrc contains "set enable-bracketed-paste on".

[vstinner]

https://lists.gnu.org/archive/html/bug-readline/2020-11/msg00010.html
says that bracketed mode is turned off if the terminal type is "dumb".

Setting TERM env var to dumb gives me a surprising behavior when I paste "1+1\n2+2" in Python REPL:
---

$ TERM=dumb python3
Python 3.9.1 (default, Jan 20 2021, 00:00:00) 
>>> ^J1+1^J2+2^J

---

If I press ENTER, I get:
---

>>> ^J1+1^J2+2^J
  File "<stdin>", line 1
    
1+1
2+2

^

SyntaxError: multiple statements found while compiling a single statement

>>
---

Note: if I redirect Python output into a file ("python3 > output"), I get a different behavior since stdout is no longer a TTY.

[vstinner]

There are different things:

• Bracketed paste mode prevents to execute malicious command copied from evil internet web pages
• Python REPL is not really convenient in the bracked paste mode: bpo-39820
• Users are not used to the bracketed mode which gives a surprising behavior in Python REPL
• Copy/paste in a shell running in a graphical terminal is fine in bracketed mode, since users running a shell are used to modify a command before running and to run manually a command by pressing ENTER
• The bracketed paste mode causes test failures in applications (like pytest test suite) which doesn't support it.
• It is not easy/convenient to opt-out for the bracketed paste mode.

For all these reasons, it sounds reasonable to disable the readline bracketed paste mode by default in Python, even if it's enabled explicitly in ~/.inputrc.

If an user opts in for the bracketed paste mode, it is more likely to prevent running malicious commands in a shell, rather than not executing immediately commands executed in Python.

Once the bpo-39820 will be fixed, we can reconsider to leave the bracketed paste mode default unchanged (no longer disable it explicitly).