shlex doesn't differentiate escaped characters in output

[MatthewGamble]

BPO	36897
Nosy	@ericvsmith

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-05-13.00:59:51.237>
labels = ['3.7', 'type-bug', 'library']
title = "shlex doesn't differentiate escaped characters in output"
updated_at = <Date 2019-05-14.01:12:01.944>
user = 'https://bugs.python.org/MatthewGamble'

bugs.python.org fields:

activity = <Date 2019-05-14.01:12:01.944>
actor = 'Matthew Gamble'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-05-13.00:59:51.237>
creator = 'Matthew Gamble'
dependencies = []
files = []
hgrepos = []
issue_num = 36897
keywords = []
message_count = 5.0
messages = ['342276', '342383', '342399', '342406', '342410']
nosy_count = 2.0
nosy_names = ['eric.smith', 'Matthew Gamble']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36897'
versions = ['Python 3.6', 'Python 3.7']

[MatthewGamble]

The output of the following invocations are exactly the same:

list(shlex.shlex('a ; b', posix=True, punctuation_chars=True))

list(shlex.shlex('a \; b', posix=True, punctuation_chars=True))

They both output the following:

['a', ';', 'b']

This makes it impossible to determine when the user wanted to escape the semi-colon for some reason, such as if they were using find's -exec argument.

[ericvsmith]

The goal is to match posix shell semantics. Can you provide a concrete example where shlex.shlex does something different from a posix-compliant shell? With all the escaping, it's going to be tough.

Note also that your code raises a DeprecationWarning in 3.7, at least, and will be an error in the future. You should probably use r-strings in your examples.

[MatthewGamble]

The point is that it's not possible to use the output of shlex.shlex to try to match the behaviour of a POSIX-compliant shell by reliably splitting up a user's input into multiple commands. In the first case I presented (no escape character), the user entered two commands. In the second case, the user entered a single command with two arguments. However, there's no way to differentiate the two situations based on the output of shlex.

It's also worth noting that the output is the same with this too:

list(shlex.shlex('a \\; b', posix=True, punctuation_chars=True))

I tested this code on python 3.6.7 and 3.7.2, and didn't see any deprecation warnings at all. I also checked the history of shlex.py:

https://github.com/python/cpython/commits/master/Lib/shlex.py

The last commit was from 2017, and I don't see any usages of DeprecationWarning inside that file. I'm also not sure how r-strings are relevant, as I don't see any regular expressions used inside of the shlex class.

[ericvsmith]

Run 3.7 with -Wd:

$ python3 -Wd
Python 3.7.3 (default, Mar 29 2019, 13:03:53)
[GCC 7.4.0] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 'a \; b'
<stdin>:1: DeprecationWarning: invalid escape sequence \;
'a \\; b'
>>>

The deprecation is in relation to invalid escape sequences, not shlex.

My point is just that you should use r'a \; b' or 'a \\;b', and not rely on invalid escape sequences. For one reason, I can never remember how they're interpreted, and had to look it up. r-strings don't have anything to do with regular expressions per-se, they're a way of changing how python interprets stings, no matter what they're used for.

The point is that it's not possible to use the output of shlex.shlex to try to match the behaviour of a POSIX-compliant shell by reliably splitting up a user's input into multiple commands. In the first case I presented (no escape character), the user entered two commands. In the second case, the user entered a single command with two arguments. However, there's no way to differentiate the two situations based on the output of shlex.

My question is: can a posix-compliant shell tell the difference? I don't know, it's an honest question. Can you show some shell code where it can tell the difference?

[MatthewGamble]

My apologies, I didn't realise you were talking about the invalid escape sequence. Thanks for letting me know about the fact that it's deprecated, I'll definitely be keeping that in mind going forward.

In a bash shell with the find command available, run the following command:

find . -type f -exec ls {} \;

You should see a list of files.

If you run this:

find . -type f -exec ls {} ;

You should see an error message from find:

"find: missing argument to `-exec'"

If I pass the first example in this message to shlex, I get no indication that the user attempted escaped the semi-colon in their input.