shlex with string ending in space gives "ValueError: No closing quotation"

[ekorn]

BPO	13543
Nosy	@bitdancer, @meadori

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-12-09.21:46:15.445>
created_at = <Date 2011-12-06.22:33:53.188>
labels = ['invalid', 'type-bug', 'library']
title = 'shlex with string ending in space gives "ValueError: No closing quotation"'
updated_at = <Date 2011-12-09.22:06:07.933>
user = 'https://bugs.python.org/ekorn'

bugs.python.org fields:

activity = <Date 2011-12-09.22:06:07.933>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2011-12-09.21:46:15.445>
closer = 'ekorn'
components = ['Library (Lib)']
creation = <Date 2011-12-06.22:33:53.188>
creator = 'ekorn'
dependencies = []
files = []
hgrepos = []
issue_num = 13543
keywords = []
message_count = 5.0
messages = ['148941', '148942', '149061', '149129', '149131']
nosy_count = 3.0
nosy_names = ['r.david.murray', 'ekorn', 'meador.inge']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13543'
versions = ['Python 2.7']

[ekorn]

It seems shlex fails on processing strings ending in space, causing this bug that I reported for IPython:
ipython/ipython#1109

Fernando Perez made a minimal example of the problem, quoted below. As he points out, it would be best to fix this at the source, namely the shlex module.

Python 2.7.2 |EPD 7.1-1 (32-bit)| (default, Jul  3 2011, 15:13:59) [MSC v.1500 32 bit (Intel)] on win32
>>> import shlex
>>> lex = shlex.shlex(' ("a ")', posix=False)
>>> lex.whitespace_split = True
>>> list(lex)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\shlex.py", line 269, in next
    token = self.get_token()
  File "C:\Python27\lib\shlex.py", line 96, in get_token
    raw = self.read_token()
  File "C:\Python27\lib\shlex.py", line 172, in read_token
    raise ValueError, "No closing quotation"
ValueError: No closing quotation

[bitdancer]

Why do you consider this to be a bug? You set posix=False and whitespace_split=True, so it seems to me that according to the documented rules the " inside the word ("a is as documented not recognized as a quote character. If you use either posix=True or whitespace_split=False the string parses without error. (Since I have no idea what standard non-posix mode is based on, I have no idea whether or not this is in fact correct behavior, though.)

[ekorn]

FYI, Min RK commented on the IPython issue:
ipython/ipython#1109 (comment)

In short,
shlex.shlex('blob f(" ")', posix=False)
fails, whereas
shlex.shlex('blob f( " ")', posix=False)

"The problem appears to be that Python source obviously doesn't sit well with non-posix whitespace-split shlex. [...] if you lead all your open-quotes with whitespace, you should be fine (works for all given examples, at least). [...] I have no idea whether this is a Python bug or not, since I don't know what the reference standard is, but this is definitely an IPython bug. We should not be trying to use shlex to parse Python code as if it were command-line arguments."

[ekorn]

ipython/ipython#1109 (comment)
It seems this was an IPython bug due to slight abuse of the shlex module. Closing as invalid; thanks for your help.

[bitdancer]

And just for your information, as far as I know *no one* knows what standard (or model) non-posix mode shlex is based on.