argparse does not preserve namespace with subparser defaults #89398
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
assignee = None closed_at = None created_at = <Date 2021-09-17.14:58:54.103> labels = ['type-bug', '3.8', '3.9', '3.10', '3.11', '3.7', 'library'] title = 'argparse does not preserve namespace with subparser defaults' updated_at = <Date 2021-12-22.19:20:37.779> user = 'https://github.com/ALSchwalm'
activity = <Date 2021-12-22.19:20:37.779> actor = 'gvanrossum' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = <Date 2021-09-17.14:58:54.103> creator = 'ALSchwalm' dependencies =  files =  hgrepos =  issue_num = 45235 keywords = ['patch'] message_count = 23.0 messages = ['402060', '402120', '402121', '402122', '404990', '404991', '405007', '405129', '406052', '406079', '406121', '406123', '406128', '406171', '406172', '406175', '406186', '406231', '406232', '406233', '406571', '406814', '408991'] nosy_count = 10.0 nosy_names = ['rhettinger', 'lukasz.langa', 'paul.j3', 'pablogsal', 'miss-islington', 'Frost Ming', 'RhinosF1', 'ALSchwalm', 'dgw', 'lwalejko'] pr_nums = ['28420', '28442', '28443', '29525', '29530', '29531', '29574', '30219'] priority = 'normal' resolution = None stage = 'patch review' status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue45235' versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']
The text was updated successfully, but these errors were encountered:
The following snippet demonstrates the problem. If a subparser flag has a default set, argparse will override the existing value in the provided 'namespace' if the flag does not appear (e.g., if the default is used):
import argparse parser = argparse.ArgumentParser() sub = parser.add_subparsers() example_subparser = sub.add_parser("example") example_subparser.add_argument("--flag", default=10) print(parser.parse_args(["example"], argparse.Namespace(flag=20)))
This should return 'Namespace(flag=20)' because 'flag' already exists in the namespace, but instead it returns 'Namespace(flag=10)'. This intended behavior is described and demonstrated in the second example here: https://docs.python.org/3/library/argparse.html#default
Lib's behavior is correct for the non-subparser cause.
I haven't studied or tested this change, but it looks like a partial retraction of
Originally the main namespace was passed to the subparser. Steven Bethard changed it so that the subparser got a fresh namespace, and all values were copied to the main namespace.
I and others raised the question of how it affected user provided values
Looks like this patch tries to solve some problems by moving the self._defaults step to the end of parser_know_args. I view that change with some trepidation. Handling defaults is tricky enough, with setting them at the start, but then only passing them through 'type' at the end if they still match the original strings.
Mostly I've been telling StackOverflow questioners that it best not to use the same argument 'dest' in both the main and subparsers. Flags can be the same, but the 'dest' should be different to avoid conflicts over which default has priority.
Again, I haven't tested this change, but I have a gut feeling it could have backward compatibility issues.
I just downloaded this
This change makes it impossible to use a subparser argument if it is defined in the user provided namespace, or by the main parser. It blocks not only subparser default, but also user input.
It has reverted the 9351 patch which dates to 2014.
parser = argparse.ArgumentParser() sub = parser.add_subparsers() example_subparser = sub.add_parser("example") example_subparser.add_argument("--flag", default=10) print(parser.parse_args(["example","--flag=15"], argparse.Namespace(flag=20)))
still returns flag=20
User input should override values set by the provided Namespace.
I should study previous posts in more detail, but here are some thoughts on correctly handling user namespace.
At the start of
if namespace is None: namespace = Namespace()
We need to hang on to a copy of this namespace, e.g. call it
import copy orig_namespace = copy.copy(namespace)
In _SubParsersAction.__call__, pass this copy to the subparser (instead of None):
Prior to 9351, the main namespace was passed to the subparser
The trick is to get orig_namespace from the main parse_known_args to SubParsersAction.__call__ method.
in a 9351 post I explore the idea of allowing the user to specify a 'sub_namespace' for the subparser.
In any case, this is a complicated issue that needs careful thought and more extensive testing. I didn't entirely like the original 9351 change, but since that's been part of argparse for many years, we need to very careful about clobbering it.
Can confirm that this patch DOES cause backward compatibility issues, as paul.j3's gut feeling said it could. One of my projects, testing against 3.6-3.9, now fails its test suite on Python 3.9.8, which includes this change. Arguments passed to a subparser are indeed ignored in lieu of default values. We are tracking the problem in our own issue, sopel-irc/sopel#2210
I, for one, am not amused that 7-year-old behavior was "clobbered" (as previously described in this thread) in a patch release.
I can also confirm that there is an regression in Python 3.9.8 regarding argparse
For example, using watchdog 2.1.6 package
Python 3.9.7 (correct behaviour)
Python 3.9.8 (incorrect behaviour)
On python 3.8.10 (and 3.10), subparsers are not updated with defaults. I suspect this is related to . Fix this by explicitly updating subparsers with settings.  python/cpython#89398 Fixes: 3145b63 ("patman: Update defaults in subparsers") Signed-off-by: Sean Anderson <firstname.lastname@example.org> Reviewed-by: Alper Nebi Yasak <email@example.com> Tested-by: Alper Nebi Yasak <firstname.lastname@example.org>