Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

argparse required arguments displayed under "optional arguments" #53903

Closed
benschmaus mannequin opened this issue Aug 26, 2010 · 50 comments
Closed

argparse required arguments displayed under "optional arguments" #53903

benschmaus mannequin opened this issue Aug 26, 2010 · 50 comments
Assignees
Labels
3.10 stdlib type-bug

Comments

@benschmaus
Copy link
Mannequin

@benschmaus benschmaus mannequin commented Aug 26, 2010

BPO 9694
Nosy @rhettinger, @terryjreedy, @ericvsmith, @merwok, @bitdancer, @vadmium, @mgrandi, @hroncok, @maggyero, @jacktose
PRs
  • #23858
  • Files
  • argparse-help-says-required-args-are-optional.py
  • alt_grouping.py
  • helpgroups.diff
  • alt_grouping2.py: Script demonstrating helpgroups.diff patch
  • parrot.py: Script to test alternate wording
  • argparse_option.patch
  • option-internal.patch
  • option-internal.patch
  • argparse_option.v2.patch
  • argparse_optional.txt: Survey of practices
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/rhettinger'
    closed_at = <Date 2020-12-23.17:41:40.125>
    created_at = <Date 2010-08-26.18:19:17.172>
    labels = ['type-bug', 'library', '3.10']
    title = 'argparse required arguments displayed under "optional arguments"'
    updated_at = <Date 2021-01-08.23:51:14.321>
    user = 'https://bugs.python.org/benschmaus'

    bugs.python.org fields:

    activity = <Date 2021-01-08.23:51:14.321>
    actor = 'hroncok'
    assignee = 'rhettinger'
    closed = True
    closed_date = <Date 2020-12-23.17:41:40.125>
    closer = 'rhettinger'
    components = ['Library (Lib)']
    creation = <Date 2010-08-26.18:19:17.172>
    creator = 'benschmaus'
    dependencies = []
    files = ['18654', '34065', '34073', '34074', '36528', '37200', '37461', '37505', '42370', '49692']
    hgrepos = []
    issue_num = 9694
    keywords = ['patch']
    message_count = 50.0
    messages = ['115017', '115019', '115021', '115023', '115032', '115037', '115038', '115045', '115048', '115058', '115059', '115069', '115109', '115117', '115148', '132327', '166183', '208659', '210808', '211121', '211125', '211132', '211205', '226279', '226286', '226290', '231189', '232715', '232928', '243290', '262273', '262297', '262315', '262323', '262553', '262893', '327376', '345542', '345545', '357750', '373997', '383399', '383402', '383404', '383617', '383627', '383650', '384677', '384684', '384700']
    nosy_count = 22.0
    nosy_names = ['rhettinger', 'terry.reedy', 'bethard', 'eric.smith', 'eric.araujo', 'r.david.murray', 'docs@python', 'benschmaus', 'tshepang', 'martin.panter', 'paul.j3', 'markgrandi', "Martin.d'Anjou", 'Oliver.Smith', 'rhartkopf', 'Albert White', 'tonygaetani', 'shaharg', 'hroncok', 'maggyero', 'Jacktose', 'kkarbowiak']
    pr_nums = ['23858']
    priority = 'low'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue9694'
    versions = ['Python 3.10']

    @benschmaus
    Copy link
    Mannequin Author

    @benschmaus benschmaus mannequin commented Aug 26, 2010

    The argparse module lists required args as optional in the default help message.

    If you run the following program (also attached) you'll get the output listed below.

    #!/usr/bin/env python

    import argparse
    
    parser = argparse.ArgumentParser(
        description = 'Do something'
    )
    parser.add_argument('--reqarg', '-r', help = 'This is required', required = True)
    parser.add_argument('--optarg','-o', help = "This is optional", required = False)
    args = parser.parse_args()
    $ python argparse-help-says-required-args-are-optional.py -h
    usage: argparse-help-says-required-args-are-optional.py [-h] --reqarg REQARG
                                                            [--optarg OPTARG]

    Do something

    optional arguments:
    -h, --help show this help message and exit
    --reqarg REQARG, -r REQARG
    This is required
    --optarg OPTARG, -o OPTARG
    This is optional
    $

    @benschmaus benschmaus mannequin added type-bug stdlib labels Aug 26, 2010
    @ericvsmith
    Copy link
    Member

    @ericvsmith ericvsmith commented Aug 26, 2010

    It looks to me like reqarg is marked as required, since it's not in brackets. Or am I missing something?

    @bitdancer
    Copy link
    Member

    @bitdancer bitdancer commented Aug 26, 2010

    Yeah, the fact that it is listed under the heading "optional arguments:" :) Guess we need a new section?

    @ericvsmith
    Copy link
    Member

    @ericvsmith ericvsmith commented Aug 26, 2010

    Duh. Sorry about that.

    Also applies to 3.2.

    @bethard
    Copy link
    Mannequin

    @bethard bethard mannequin commented Aug 26, 2010

    Yeah, I guess the optional vs. positional isn't the best terminology now that you can have required flag-based arguments. Did you have a word other than "optional" that you'd prefer?

    @bitdancer
    Copy link
    Member

    @bitdancer bitdancer commented Aug 26, 2010

    Perhaps you could just label them 'options:'? After all, even if you have several options you may be required to pick at least one :)

    @ericvsmith
    Copy link
    Member

    @ericvsmith ericvsmith commented Aug 26, 2010

    Or "parameters:"?

    @benschmaus
    Copy link
    Mannequin Author

    @benschmaus benschmaus mannequin commented Aug 26, 2010

    FWIW, I like the idea of just using the label "options".

    @ericvsmith
    Copy link
    Member

    @ericvsmith ericvsmith commented Aug 27, 2010

    If you add a positional parameter by adding:
    parser.add_argument('foo')
    then the output becomes:

    $ python argparse-help-says-required-args-are-optional.py -h
    usage: issue9649.py [-h] --reqarg REQARG [--optarg OPTARG] foo

    Do something

    positional arguments:
    foo

    optional arguments:
    -h, --help show this help message and exit
    --reqarg REQARG, -r REQARG
    This is required
    --optarg OPTARG, -o OPTARG
    This is optional

    $

    So whatever replaces "optional arguments:" needs to read well with "positional arguments:". Maybe just plain "options:" is good enough, but I think a word to replace "optional" (leaving "arguments:") would be better. I just don't have any useful suggestion :)

    @bethard
    Copy link
    Mannequin

    @bethard bethard mannequin commented Aug 27, 2010

    I guess one possibility might be "flag arguments". It's not great, but I guess it's more accurate.

    @bethard
    Copy link
    Mannequin

    @bethard bethard mannequin commented Aug 27, 2010

    And I guess the bigger issue to think about is how to add this in a backwards compatible way. I guess we could just add methods like "set_positionals_group_name(name)" and then fiddle with "self._positionals.title" in there. Not sure that's a great solution though - it seems like adding one method to change just this single attribute is overkill and not very general.

    In the meantime, here's a workaround:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('--foo', required=True)
    >>> parser._optionals.title = "flag arguments"
    >>> parser.print_help()
    usage: PROG [-h] --foo FOO

    flag arguments:
    -h, --help show this help message and exit
    --foo FOO

    I can't promise this will continue to work, since it uses the undocumented _optionals attribute, but at least it's a way of getting something like what you want now.

    @bitdancer
    Copy link
    Member

    @bitdancer bitdancer commented Aug 27, 2010

    Well, there's also bpo-9652, which speaks to having a more general facility, I suppose. Maybe an exposed dictionary attribute containing the constant strings?

    @terryjreedy
    Copy link
    Member

    @terryjreedy terryjreedy commented Aug 27, 2010

    Is this really a behavior bug or doc bug?
    Or a feature request for better message customization?

    @bitdancer
    Copy link
    Member

    @bitdancer bitdancer commented Aug 27, 2010

    *This* bug is a behavior bug (required flags are mis-labelled as being optional in the help text). The referenced bug is a feature request, but it may make sense to consider it while fixing this one.

    @bethard
    Copy link
    Mannequin

    @bethard bethard mannequin commented Aug 28, 2010

    I think this is still really a feature request. We can't just change the text from "optional" - that would silently change a large number of help messages without any warning. So to fix this "bug", we're going to have to add an API to explicitly set the group names - which can only be done as a new feature. People using 2.7 will have to use the workaround using parser._optionals I posted here.

    @bethard bethard mannequin added type-feature and removed type-bug labels Aug 28, 2010
    @bethard
    Copy link
    Mannequin

    @bethard bethard mannequin commented Mar 27, 2011

    So it strikes me that there already exists an officially supported way to rename your option groups. Just only create your own option groups (never use the default ones) and only put arguments there, e.g.:

    ------------------------- temp.py --------------------------

    parser = argparse.ArgumentParser(description = 'Do something', add_help=False)
    flags = parser.add_argument_group('flag arguments')
    flags.add_argument('-h', '--help', action='help')
    flags.add_argument('--reqarg', '-r', help='This is required', required=True)
    flags.add_argument('--optarg','-o', help="This is optional", required=False)
    args = parser.parse_args()

    $ python temp.py --help
    usage: temp.py [-h] --reqarg REQARG [--optarg OPTARG]

    Do something

    flag arguments:
    -h, --help
    --reqarg REQARG, -r REQARG
    This is required
    --optarg OPTARG, -o OPTARG
    This is optional
    ------------------------------------------------------------

    The documentation for action='help' needs to be added, as pointed out in Issue# 10772.

    So basically, the API for customizing group names is already there. So I'm changing this to a documentation request - there should be an example in the docs showing how to change the default group names as above.

    @bethard bethard mannequin added docs and removed stdlib labels Mar 27, 2011
    @bethard bethard mannequin assigned docspython Mar 27, 2011
    @bethard
    Copy link
    Mannequin

    @bethard bethard mannequin commented Jul 22, 2012

    I'm changing the title because I keep seeing duplicates.

    Documentation patches still welcome!

    @bethard bethard mannequin changed the title argparse: Default Help Message Lists Required Args As Optional argparse required arguments displayed under "optional arguments" Jul 22, 2012
    @MartindAnjou
    Copy link
    Mannequin

    @MartindAnjou MartindAnjou mannequin commented Jan 21, 2014

    How about calling required arguments "required arguments"?

    required arguments:
    --reqarg REQARG, -r REQARG
    This is required
    optional arguments:
    -h, --help show this help message and exit
    --optarg OPTARG, -o OPTARG
    This is optional

    Clear and unambiguous. With this approach the user does not have to bloat the help to state "This is required".

    We're having the same discussion over at github regarding argparse4j:
    argparse4j/argparse4j#26 (comment)

    @vadmium
    Copy link
    Member

    @vadmium vadmium commented Feb 10, 2014

    A new “required arguments” section seems too arbitrary to me. It would clash with the “positional arguments” heading, since those are also required by default.

    I would go with the heading “options”, as a noun. That term seems to be well used, at least on Linux and Wikipedia (see https://en.wikipedia.org/wiki/Command-line_option). Other terms are “flag” and “switch”. In this thread I see two arguments against this:

    1. Eric Smith prefers to retain the noun “arguments”. How about something like “non-positional arguments” then?

    2. Steven Bethard is worried about backwards compatibility. I thought the Python people were happy to make these sort of changes each minor release (e.g. 3.4 to 3.5).

    The module’s source code uses the term “optionals” a lot more than this one heading. It would be clearer if this term were dropped, or only used for things that are truly optional. So even if you can’t fix the help output until Python 4, please fix the documentation and the rest of the source code :)

    @rhettinger
    Copy link
    Contributor

    @rhettinger rhettinger commented Feb 13, 2014

    How about calling required arguments "required arguments"?
    ...
    Clear and unambiguous. With this approach the user does
    not have to bloat the help to state "This is required".

    +1 This is straight-forward, logical, and easy-to-read.

    @paulj3
    Copy link
    Mannequin

    @paulj3 paulj3 mannequin commented Feb 13, 2014

    As Steven pointed out, the existing add_argument_group mechanism can be used to group required arguments. For example

    ------------------------- temp.py \--------------------------
    
        parser = argparse.ArgumentParser(description = 'Do something')
        group1 = parser.add_argument_group('required arguments')
        group1.add_argument('--reqarg', '-r', required=True)
        parser.add_argument('--optarg','-o')
        parser.add_argument('foo')
        parser.print_help()
        

    usage: ipython [-h] --reqarg REQARG [--optarg OPTARG] foo
    Do something
    positional arguments:
      foo
    optional arguments:
      -h, --help            show this help message and exit
      --optarg OPTARG, -o OPTARG
    required arguments:
      --reqarg REQARG, -r REQARG
    

    Positional 'foo' can also be put in the 'required' group:

    group1.add_argument('foo')
    
    required arguments:
      --reqarg REQARG, -r REQARG
      foo
    

    The distinction between 'positionals' and 'optionals' (or flagged) is essential to the parsing, but it is not necessary for Help Formatting.

    I can imagine grouping arguments by 'required/not-required' properties. It might be worth constructing an alternative HelpFormatter class that regroups the arguments in this way. Subclassing the HelpFormatter is the established way of adding features to the help display.

    The existing HelpFormatter flags 'required' arguments in the usage line with '[]'. There it is has the added task of flagging Mutually Exclusive Groups in the same way.

    It's worth keeping in mind that whether an argument is 'required' or not is determined in 2 different ways. There is an optional 'required' flag (default False). But this flag is not allowed for 'positionals'. Instead with those 'argparse' looks at 'nargs' ('?*' are not required).

    The 'required' attribute of an argument (Action) is ignored during 'parse_args' until the end. At that time it makes an inventory of 'required' arguments that have not been seen, and potentially raises an error. That testing was changed in a relatively recent patch, and produced an unintended change in whether 'subparsers' were required or not. (I could look up those issues in needed).

    I'll think about creating the alternative HelpFormatter.

    @paulj3
    Copy link
    Mannequin

    @paulj3 paulj3 mannequin commented Feb 13, 2014

    The attached file shows how the default argument groups could be redefined, using 'required' as the criteria.

    I've implemented it as a method that is added to a subclass of ArgumentParser. This method is invoked after arguments are defined, prior to generating the help.

    The help looks something like this:

    usage: alt_grouping.py [-h] [-f FOO] -g GOO pos [baz]
    
    required:
      pos                required positional
      -g GOO, --goo GOO  required optional
    
    optional:
      -h, --help         show this help message and exit
      -f FOO, --foo FOO  optional
      baz                optional positional
    

    I was thinking of implementing this as a formatter subclass, but given the way the help is assembled, invoking this method from the parser is simpler.

    @vadmium
    Copy link
    Member

    @vadmium vadmium commented Mar 24, 2016

    Still applicable to Python 3 AFAIK

    @paulj3
    Copy link
    Mannequin

    @paulj3 paulj3 mannequin commented Mar 24, 2016

    I can see changing the group title from 'optional arguments' to 'options' or 'optionals'

    parser._optionals.title
    'optional arguments'
    

    But I don't think there's a need to change references in the code or its comments from 'optionals' to 'options'. I like the parallelism between 'optionals' and 'positionals'. The terms are well defined in the code. During parsing, the 'required' attribute is only used at the end to check for missing arguments.

    In Stackoverflow questions I'm tended to talk about 'flagged arguments'.

    I still favor encouraging users to define their argument group(s), and making it easier to modify the titles of the two predefined groups. I don't see enough of a consensus on alternative titles to make more sweeping changes.

    @vadmium
    Copy link
    Member

    @vadmium vadmium commented Mar 28, 2016

    I am willing to drop most of the code and comment changes if that moves us closer to consensus. Paul: would you accept changing the heading “optional arguments” to “options” in 3.6?

    I thought we were close to consensus to use “options”, but maybe that is my own bias showing. Re-reading through this thread, here is a summary of the people that seem to prefer renaming the “optional arguments” heading:

    Steven: “flag arguments”, but concerned about affecting existing help messages
    David: options, switches
    Eric: replace adjective “optional” but leave “. . . arguments”
    Ben (OP): options
    Martin Panter: options, flags, switches, or non-positional arguments
    Oliver: switches
    Terry: keyword arguments
    Ryan: options, or other suggestions

    Martin d'Anjou and Raymond seem to prefer splitting out a third group, “required arguments”, although it is not clear how that would tie in with the “positional arguments” group.

    I think making it easier to modify the headings is worthwhile only if we deprecated the old heading in favour of using a new default name in the future. To me the problem is the default behaviour, not that argparse is inflexible.

    Another option might be to fix bpo-10529 (explain using gettext with argparse), and then use that to translate the "optional arguments" string. But I don’t know how to do this.

    Another thought is if we added a notice to Porting to Python 3.6 in What’s New, that might ease concerns about changing the heading.

    @vadmium
    Copy link
    Member

    @vadmium vadmium commented Apr 5, 2016

    Posting argparse_option.v2.patch, which is minimally complete version of Ryan’s patch. I have dropped all the nonessential code and documentation tweaks. I also added a What’s New entry. I’d like to know if people think this is the right direction to move in.

    @jacktose
    Copy link
    Mannequin

    @jacktose jacktose mannequin commented Oct 8, 2018

    I'd like to note that this also happens with a required mutually exclusive group:

    group = parser.add_mutually_exclusive_group(required=True)

    The arguments in the group are listed under “optional arguments:”.

    I'm guessing the mechanism is the same.

    @mgrandi
    Copy link
    Mannequin

    @mgrandi mgrandi mannequin commented Jun 14, 2019

    Is there anything that can be done to help this issue move along? I just ran into it just now

    @paulj3
    Copy link
    Mannequin

    @paulj3 paulj3 mannequin commented Jun 14, 2019

    Mark,

    Have you tried defining your own Argument Group? If that didn't work, what fix do you want? Why?

    @maggyero
    Copy link
    Mannequin

    @maggyero maggyero mannequin commented Dec 3, 2019

    I have just run into the same issue here: https://bugs.python.org/issue38950

    • I prefer Terry J. Reedy's "keyword arguments" as it is clear and consistent with "positional arguments".
    • But Steven Bethard 's "flag arguments" looks fine since it is well known to shell users.
    • Martin Panter's "options" looks okay since it is the standard name in GNU Coreutils (https://www.gnu.org/software/coreutils/manual/coreutils.html#Common-options). However I don't like it very much as it is still ambiguous: "options", like "optional arguments", still suggests something that is non required. And "options" is less consistent with "positional arguments" (nobody seems to have suggested "option arguments").
    • Oliver Smith's "switches" does not look okay because it is not general enough since it is commonly restricted to Boolean arguments.

    Anyway, the first 3 solutions are better than the current "optional arguments". What is blocking the approval of Martin Panter's PR?

    @kkarbowiak
    Copy link
    Mannequin

    @kkarbowiak kkarbowiak mannequin commented Jul 20, 2020

    It seems the discussion has so far revolved around 'optional' arguments with required=True.

    What about the other way around?

    While trying to set required=False for a positional argument raises an exception, it is still possible to make the positional argument effectively optional by nargs='?' (it is then printed in brackets in usage message).

    @rhettinger
    Copy link
    Contributor

    @rhettinger rhettinger commented Dec 19, 2020

    Based on the attached survey of practices, I propose a minimal edit to the help display. Instead of "optional arguments:", we say "the options are as follows:".

    The use of the word "option" is dominant is in the CLI world, followed by "action" and "switch". The noun form "option" doesn't seem to cause the same confusion that arises in the adjective form "optional arguments" which strongly implies "not required".

    For the documentation, I suggest adding a sentence or two in the introduction to explain the terminology used throughout the rest of the argparse docs.

    @rhettinger rhettinger added docs type-bug and removed type-feature labels Dec 19, 2020
    @rhettinger
    Copy link
    Contributor

    @rhettinger rhettinger commented Dec 19, 2020

    Since any chance to the help output will likely break tests, marking this as 3.10 only.

    @rhettinger rhettinger added stdlib and removed docs 3.9 labels Dec 19, 2020
    @rhettinger
    Copy link
    Contributor

    @rhettinger rhettinger commented Dec 19, 2020

    I also like Eric'c suggestion of just using "options:" instead of "optional arguments".

    @rhettinger
    Copy link
    Contributor

    @rhettinger rhettinger commented Dec 22, 2020

    Since this change will break tests that rely matching help output exactly, I would like to hear if there are any objections to replacing "optional arguments" with "options".

    The words "switch" or "flag" don't work as well because they imply on/off and don't encompass option that take arguments.

    @ericvsmith
    Copy link
    Member

    @ericvsmith ericvsmith commented Dec 23, 2020

    I wouldn't let breaking these tests deter you from improving the output. I think using "options" is an improvement.

    @rhettinger
    Copy link
    Contributor

    @rhettinger rhettinger commented Dec 23, 2020

    New changeset 41b223d by Raymond Hettinger in branch 'master':
    bpo-9694: Fix misleading phrase "optional arguments" (GH-23858)
    41b223d

    @hroncok
    Copy link
    Mannequin

    @hroncok hroncok mannequin commented Jan 8, 2021

    Coudl this please be mentioned on https://docs.python.org/3.10/whatsnew/3.10.html ?

    At least two packages fail tests because of the change (ipython and sphinxcontrib-autoprogram).

    @paulj3
    Copy link
    Mannequin

    @paulj3 paulj3 mannequin commented Jan 8, 2021

    Since this issue is closed it might be a good idea to open a new one with this problem. And if possible identify the failed tests.

    We forgot to allow for the fact that working code/tests might be checking for specific help messages, checks the will fail when this group label is changed.

    @hroncok
    Copy link
    Mannequin

    @hroncok hroncok mannequin commented Jan 8, 2021

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 stdlib type-bug
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants