argparse wrapping fails with metavar="" (no metavar)

[sjfranklin]

BPO	37910
Nosy	@rhettinger, @shihai1991, @sjfranklin, @iritkatriel
PRs	bpo-37910: argparse usage wrapping should allow whitespace differences caused by metavar #15372

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-08-21.22:08:19.919>
labels = ['type-bug', 'library', '3.10', '3.11']
title = 'argparse wrapping fails with metavar="" (no metavar)'
updated_at = <Date 2021-10-19.09:28:30.797>
user = 'https://github.com/sjfranklin'

bugs.python.org fields:

activity = <Date 2021-10-19.09:28:30.797>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-08-21.22:08:19.919>
creator = 'sjfranklin'
dependencies = []
files = []
hgrepos = []
issue_num = 37910
keywords = ['patch']
message_count = 6.0
messages = ['350117', '350169', '354647', '371415', '371446', '404280']
nosy_count = 5.0
nosy_names = ['rhettinger', 'paul.j3', 'shihai1991', 'sjfranklin', 'iritkatriel']
pr_nums = ['15372']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37910'
versions = ['Python 3.10', 'Python 3.11']

[sjfranklin]

When argparse wraps the usage text, it can fail its assertion tests with whitespace differences. This can occur when metavar="", needed if a user wishes to avoid having a metavar print. It also could occur if a user specifies any other whitespace.
Here's a minimum example (depending on $COLUMNS):

import argparse
# based on Vajrasky Kok's script in https://bugs.python.org/issue11874
parser = argparse.ArgumentParser(prog='PROG')
parser.add_argument('--nil', metavar='', required=True)
parser.add_argument('--a', metavar='a' * 165)
parser.parse_args()

This produces the AssertionError at the bottom of this comment.

A solution is to have the two asserts ignore whitespace. I'll submit a pull request very shortly for this. (First time so happy for any comments or critiques!)

A more extensive example:
import argparse
# based on Vajrasky Kok's script in https://bugs.python.org/issue11874
parser = argparse.ArgumentParser(prog='PROG')

parser.add_argument('--nil', metavar='', required=True)
parser.add_argument('--Line-Feed', metavar='\n', required=True)
parser.add_argument('--Tab', metavar='\t', required=True)
parser.add_argument('--Carriage-Return', metavar='\r', required=True)
parser.add_argument('--Carriage-Return-and-Line-Feed',
                    metavar='\r\n', required=True)
parser.add_argument('--vLine-Tabulation', metavar='\v', required=True)
parser.add_argument('--x0bLine-Tabulation', metavar='\x0b', required=True)
parser.add_argument('--fForm-Feed', metavar='\f', required=True)
parser.add_argument('--x0cForm-Feed', metavar='\x0c', required=True)
parser.add_argument('--File-Separator', metavar='\x1c', required=True)
parser.add_argument('--Group-Separator', metavar='\x1d', required=True)
parser.add_argument('--Record-Separator', metavar='\x1e', required=True)
parser.add_argument('--C1-Control-Code', metavar='\x85', required=True)
parser.add_argument('--Line-Separator', metavar='\u2028', required=True)
parser.add_argument('--Paragraph-Separator', metavar='\u2029', required=True)
parser.add_argument('--a', metavar='a' * 165)
parser.parse_args()

This is related to https://bugs.python.org/issue17890 and https://bugs.python.org/issue32867.

File "/minimum_argparse_bug.py", line 7, in <module>
parser.parse_args()
File "/path/to/cpython/Lib/argparse.py", line 1758, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/path/to/cpython/Lib/argparse.py", line 1790, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/path/to/cpython/Lib/argparse.py", line 1996, in _parse_known_args
start_index = consume_optional(start_index)
File "/path/to/cpython/Lib/argparse.py", line 1936, in consume_optional
take_action(action, args, option_string)
File "/path/to/cpython/Lib/argparse.py", line 1864, in take_action
action(self, namespace, argument_values, option_string)
File "/path/to/cpython/Lib/argparse.py", line 1037, in __call__
parser.print_help()
File "/path/to/cpython/Lib/argparse.py", line 2483, in print_help
self._print_message(self.format_help(), file)
File "/path/to/cpython/Lib/argparse.py", line 2467, in format_help
return formatter.format_help()
File "/path/to/cpython/Lib/argparse.py", line 281, in format_help
help = self._root_section.format_help()
File "/path/to/cpython/Lib/argparse.py", line 212, in format_help
item_help = join([func(*args) for func, args in self.items])
File "/path/to/cpython/Lib/argparse.py", line 212, in <listcomp>
item_help = join([func(*args) for func, args in self.items])
File "/path/to/cpython/Lib/argparse.py", line 336, in _format_usage
assert ' '.join(opt_parts) == opt_usage
AssertionError

[paulj3]

That usage formatting is extremely brittle. It's not just "" metavar that can mess it up. Other 'usual' characters can mess it in the same way.

The underlying problem is that it formats the whole usage, and if it is too long tries to split it into pieces, and then reassemble it in wrapped lines. The assertion tries to verify that the split was accurate.

Usage really needs to be rewritten in a way that keeps the individual Action pieces separate until it is ready to assemble them into final lines. Anything else is just bandaids.

[sjfranklin]

Paul, very true. If I find time I may take a look at rewriting it as you suggest.
For the moment, though, there's a relatively simple change that opens up a range of allowable whitespace characters. It's still a bandage, but it covers decent-sized area and should be easily backported. What do you think of the pull request? It's passed a first review.

[rhettinger]

Paul, what do you think about the PR?

[shihai1991]

LGTM. It's a lightweight patch :)

[iritkatriel]

Reproduced on 3.11.

[encukou]

Fixed in #105039:

import argparse
# based on Vajrasky Kok's script in https://bugs.python.org/issue11874
parser = argparse.ArgumentParser(prog='PROG')
parser.add_argument('--nil', metavar='', required=True)
parser.add_argument('--a', metavar='a' * 165)
parser.parse_args()

usage: PROG [-h] --nil 
            [--a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa]
PROG: error: the following arguments are required: --nil

---

import argparse
# based on Vajrasky Kok's script in https://bugs.python.org/issue11874
parser = argparse.ArgumentParser(prog='PROG')
parser.add_argument('--nil', metavar='', required=True)
parser.add_argument('--Line-Feed', metavar='\n', required=True)
parser.add_argument('--Tab', metavar='\t', required=True)
parser.add_argument('--Carriage-Return', metavar='\r', required=True)
parser.add_argument('--Carriage-Return-and-Line-Feed',
                    metavar='\r\n', required=True)
parser.add_argument('--vLine-Tabulation', metavar='\v', required=True)
parser.add_argument('--x0bLine-Tabulation', metavar='\x0b', required=True)
parser.add_argument('--fForm-Feed', metavar='\f', required=True)
parser.add_argument('--x0cForm-Feed', metavar='\x0c', required=True)
parser.add_argument('--File-Separator', metavar='\x1c', required=True)
parser.add_argument('--Group-Separator', metavar='\x1d', required=True)
parser.add_argument('--Record-Separator', metavar='\x1e', required=True)
parser.add_argument('--C1-Control-Code', metavar='\x85', required=True)
parser.add_argument('--Line-Separator', metavar='\u2028', required=True)
parser.add_argument('--Paragraph-Separator', metavar='\u2029', required=True)
parser.add_argument('--a', metavar='a' * 165)
parser.parse_args()

usage: PROG [-h] --nil  --Line-Feed 
 --Tab 	 --Carriage-Return 
            --Carriage-Return-and-Line-Feed 
 --vLine-Tabulation 
                     --x0bLine-Tabulation 

            --fForm-Feed 
                          --x0cForm-Feed 
                                          --File-Separator  --Group-Separator 
            --Record-Separator  --C1-Control-Code 
 --Line-Separator 

            --Paragraph-Separator 

            [--a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa]
PROG: error: the following arguments are required: --nil, --Line-Feed, --Tab, --Carriage-Return, --Carriage-Return-and-Line-Feed, --vLine-Tabulation, --x0bLine-Tabulation, --fForm-Feed, --x0cForm-Feed, --File-Separator, --Group-Separator, --Record-Separator, --C1-Control-Code, --Line-Separator, --Paragraph-Separator