Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

argparse required arguments displayed under "optional arguments" #53903

Closed
benschmaus mannequin opened this issue Aug 26, 2010 · 50 comments
Closed

argparse required arguments displayed under "optional arguments" #53903

benschmaus mannequin opened this issue Aug 26, 2010 · 50 comments
Assignees
Labels
3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@benschmaus
Copy link
Mannequin

benschmaus mannequin commented Aug 26, 2010

BPO 9694
Nosy @rhettinger, @terryjreedy, @ericvsmith, @merwok, @bitdancer, @vadmium, @mgrandi, @hroncok, @maggyero, @jacktose
PRs
  • bpo-9694: Fix misleading phrase "optional arguments" #23858
  • Files
  • argparse-help-says-required-args-are-optional.py
  • alt_grouping.py
  • helpgroups.diff
  • alt_grouping2.py: Script demonstrating helpgroups.diff patch
  • parrot.py: Script to test alternate wording
  • argparse_option.patch
  • option-internal.patch
  • option-internal.patch
  • argparse_option.v2.patch
  • argparse_optional.txt: Survey of practices
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/rhettinger'
    closed_at = <Date 2020-12-23.17:41:40.125>
    created_at = <Date 2010-08-26.18:19:17.172>
    labels = ['type-bug', 'library', '3.10']
    title = 'argparse required arguments displayed under "optional arguments"'
    updated_at = <Date 2021-01-08.23:51:14.321>
    user = 'https://bugs.python.org/benschmaus'

    bugs.python.org fields:

    activity = <Date 2021-01-08.23:51:14.321>
    actor = 'hroncok'
    assignee = 'rhettinger'
    closed = True
    closed_date = <Date 2020-12-23.17:41:40.125>
    closer = 'rhettinger'
    components = ['Library (Lib)']
    creation = <Date 2010-08-26.18:19:17.172>
    creator = 'benschmaus'
    dependencies = []
    files = ['18654', '34065', '34073', '34074', '36528', '37200', '37461', '37505', '42370', '49692']
    hgrepos = []
    issue_num = 9694
    keywords = ['patch']
    message_count = 50.0
    messages = ['115017', '115019', '115021', '115023', '115032', '115037', '115038', '115045', '115048', '115058', '115059', '115069', '115109', '115117', '115148', '132327', '166183', '208659', '210808', '211121', '211125', '211132', '211205', '226279', '226286', '226290', '231189', '232715', '232928', '243290', '262273', '262297', '262315', '262323', '262553', '262893', '327376', '345542', '345545', '357750', '373997', '383399', '383402', '383404', '383617', '383627', '383650', '384677', '384684', '384700']
    nosy_count = 22.0
    nosy_names = ['rhettinger', 'terry.reedy', 'bethard', 'eric.smith', 'eric.araujo', 'r.david.murray', 'docs@python', 'benschmaus', 'tshepang', 'martin.panter', 'paul.j3', 'markgrandi', "Martin.d'Anjou", 'Oliver.Smith', 'rhartkopf', 'Albert White', 'tonygaetani', 'shaharg', 'hroncok', 'maggyero', 'Jacktose', 'kkarbowiak']
    pr_nums = ['23858']
    priority = 'low'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue9694'
    versions = ['Python 3.10']

    @benschmaus
    Copy link
    Mannequin Author

    benschmaus mannequin commented Aug 26, 2010

    The argparse module lists required args as optional in the default help message.

    If you run the following program (also attached) you'll get the output listed below.

    #!/usr/bin/env python

    import argparse
    
    parser = argparse.ArgumentParser(
        description = 'Do something'
    )
    parser.add_argument('--reqarg', '-r', help = 'This is required', required = True)
    parser.add_argument('--optarg','-o', help = "This is optional", required = False)
    args = parser.parse_args()
    $ python argparse-help-says-required-args-are-optional.py -h
    usage: argparse-help-says-required-args-are-optional.py [-h] --reqarg REQARG
                                                            [--optarg OPTARG]

    Do something

    optional arguments:
    -h, --help show this help message and exit
    --reqarg REQARG, -r REQARG
    This is required
    --optarg OPTARG, -o OPTARG
    This is optional
    $

    @benschmaus benschmaus mannequin added type-bug An unexpected behavior, bug, or error stdlib Python modules in the Lib dir labels Aug 26, 2010
    @ericvsmith
    Copy link
    Member

    It looks to me like reqarg is marked as required, since it's not in brackets. Or am I missing something?

    @bitdancer
    Copy link
    Member

    Yeah, the fact that it is listed under the heading "optional arguments:" :) Guess we need a new section?

    @ericvsmith
    Copy link
    Member

    Duh. Sorry about that.

    Also applies to 3.2.

    @bethard
    Copy link
    Mannequin

    bethard mannequin commented Aug 26, 2010

    Yeah, I guess the optional vs. positional isn't the best terminology now that you can have required flag-based arguments. Did you have a word other than "optional" that you'd prefer?

    @bitdancer
    Copy link
    Member

    Perhaps you could just label them 'options:'? After all, even if you have several options you may be required to pick at least one :)

    @ericvsmith
    Copy link
    Member

    Or "parameters:"?

    @benschmaus
    Copy link
    Mannequin Author

    benschmaus mannequin commented Aug 26, 2010

    FWIW, I like the idea of just using the label "options".

    @ericvsmith
    Copy link
    Member

    If you add a positional parameter by adding:
    parser.add_argument('foo')
    then the output becomes:

    $ python argparse-help-says-required-args-are-optional.py -h
    usage: issue9649.py [-h] --reqarg REQARG [--optarg OPTARG] foo

    Do something

    positional arguments:
    foo

    optional arguments:
    -h, --help show this help message and exit
    --reqarg REQARG, -r REQARG
    This is required
    --optarg OPTARG, -o OPTARG
    This is optional

    $

    So whatever replaces "optional arguments:" needs to read well with "positional arguments:". Maybe just plain "options:" is good enough, but I think a word to replace "optional" (leaving "arguments:") would be better. I just don't have any useful suggestion :)

    @bethard
    Copy link
    Mannequin

    bethard mannequin commented Aug 27, 2010

    I guess one possibility might be "flag arguments". It's not great, but I guess it's more accurate.

    @bethard
    Copy link
    Mannequin

    bethard mannequin commented Aug 27, 2010

    And I guess the bigger issue to think about is how to add this in a backwards compatible way. I guess we could just add methods like "set_positionals_group_name(name)" and then fiddle with "self._positionals.title" in there. Not sure that's a great solution though - it seems like adding one method to change just this single attribute is overkill and not very general.

    In the meantime, here's a workaround:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('--foo', required=True)
    >>> parser._optionals.title = "flag arguments"
    >>> parser.print_help()
    usage: PROG [-h] --foo FOO

    flag arguments:
    -h, --help show this help message and exit
    --foo FOO

    I can't promise this will continue to work, since it uses the undocumented _optionals attribute, but at least it's a way of getting something like what you want now.

    @bitdancer
    Copy link
    Member

    Well, there's also bpo-9652, which speaks to having a more general facility, I suppose. Maybe an exposed dictionary attribute containing the constant strings?

    @terryjreedy
    Copy link
    Member

    Is this really a behavior bug or doc bug?
    Or a feature request for better message customization?

    @bitdancer
    Copy link
    Member

    *This* bug is a behavior bug (required flags are mis-labelled as being optional in the help text). The referenced bug is a feature request, but it may make sense to consider it while fixing this one.

    @bethard
    Copy link
    Mannequin

    bethard mannequin commented Aug 28, 2010

    I think this is still really a feature request. We can't just change the text from "optional" - that would silently change a large number of help messages without any warning. So to fix this "bug", we're going to have to add an API to explicitly set the group names - which can only be done as a new feature. People using 2.7 will have to use the workaround using parser._optionals I posted here.

    @bethard bethard mannequin added type-feature A feature request or enhancement and removed type-bug An unexpected behavior, bug, or error labels Aug 28, 2010
    @bethard
    Copy link
    Mannequin

    bethard mannequin commented Mar 27, 2011

    So it strikes me that there already exists an officially supported way to rename your option groups. Just only create your own option groups (never use the default ones) and only put arguments there, e.g.:

    ------------------------- temp.py --------------------------

    parser = argparse.ArgumentParser(description = 'Do something', add_help=False)
    flags = parser.add_argument_group('flag arguments')
    flags.add_argument('-h', '--help', action='help')
    flags.add_argument('--reqarg', '-r', help='This is required', required=True)
    flags.add_argument('--optarg','-o', help="This is optional", required=False)
    args = parser.parse_args()

    $ python temp.py --help
    usage: temp.py [-h] --reqarg REQARG [--optarg OPTARG]

    Do something

    flag arguments:
    -h, --help
    --reqarg REQARG, -r REQARG
    This is required
    --optarg OPTARG, -o OPTARG
    This is optional
    ------------------------------------------------------------

    The documentation for action='help' needs to be added, as pointed out in Issue# 10772.

    So basically, the API for customizing group names is already there. So I'm changing this to a documentation request - there should be an example in the docs showing how to change the default group names as above.

    @bethard bethard mannequin added docs Documentation in the Doc dir and removed stdlib Python modules in the Lib dir labels Mar 27, 2011
    @bethard bethard mannequin assigned docspython Mar 27, 2011
    @bethard
    Copy link
    Mannequin

    bethard mannequin commented Jul 22, 2012

    I'm changing the title because I keep seeing duplicates.

    Documentation patches still welcome!

    @bethard bethard mannequin changed the title argparse: Default Help Message Lists Required Args As Optional argparse required arguments displayed under "optional arguments" Jul 22, 2012
    @MartindAnjou
    Copy link
    Mannequin

    MartindAnjou mannequin commented Jan 21, 2014

    How about calling required arguments "required arguments"?

    required arguments:
    --reqarg REQARG, -r REQARG
    This is required
    optional arguments:
    -h, --help show this help message and exit
    --optarg OPTARG, -o OPTARG
    This is optional

    Clear and unambiguous. With this approach the user does not have to bloat the help to state "This is required".

    We're having the same discussion over at github regarding argparse4j:
    argparse4j/argparse4j#26 (comment)

    @vadmium
    Copy link
    Member

    vadmium commented Feb 10, 2014

    A new “required arguments” section seems too arbitrary to me. It would clash with the “positional arguments” heading, since those are also required by default.

    I would go with the heading “options”, as a noun. That term seems to be well used, at least on Linux and Wikipedia (see https://en.wikipedia.org/wiki/Command-line_option). Other terms are “flag” and “switch”. In this thread I see two arguments against this:

    1. Eric Smith prefers to retain the noun “arguments”. How about something like “non-positional arguments” then?

    2. Steven Bethard is worried about backwards compatibility. I thought the Python people were happy to make these sort of changes each minor release (e.g. 3.4 to 3.5).

    The module’s source code uses the term “optionals” a lot more than this one heading. It would be clearer if this term were dropped, or only used for things that are truly optional. So even if you can’t fix the help output until Python 4, please fix the documentation and the rest of the source code :)

    @rhettinger
    Copy link
    Contributor

    How about calling required arguments "required arguments"?
    ...
    Clear and unambiguous. With this approach the user does
    not have to bloat the help to state "This is required".

    +1 This is straight-forward, logical, and easy-to-read.

    @paulj3
    Copy link
    Mannequin

    paulj3 mannequin commented Feb 13, 2014

    As Steven pointed out, the existing add_argument_group mechanism can be used to group required arguments. For example

    ------------------------- temp.py \--------------------------
    
        parser = argparse.ArgumentParser(description = 'Do something')
        group1 = parser.add_argument_group('required arguments')
        group1.add_argument('--reqarg', '-r', required=True)
        parser.add_argument('--optarg','-o')
        parser.add_argument('foo')
        parser.print_help()
        

    usage: ipython [-h] --reqarg REQARG [--optarg OPTARG] foo
    Do something
    positional arguments:
      foo
    optional arguments:
      -h, --help            show this help message and exit
      --optarg OPTARG, -o OPTARG
    required arguments:
      --reqarg REQARG, -r REQARG
    

    Positional 'foo' can also be put in the 'required' group:

    group1.add_argument('foo')
    
    required arguments:
      --reqarg REQARG, -r REQARG
      foo
    

    The distinction between 'positionals' and 'optionals' (or flagged) is essential to the parsing, but it is not necessary for Help Formatting.

    I can imagine grouping arguments by 'required/not-required' properties. It might be worth constructing an alternative HelpFormatter class that regroups the arguments in this way. Subclassing the HelpFormatter is the established way of adding features to the help display.

    The existing HelpFormatter flags 'required' arguments in the usage line with '[]'. There it is has the added task of flagging Mutually Exclusive Groups in the same way.

    It's worth keeping in mind that whether an argument is 'required' or not is determined in 2 different ways. There is an optional 'required' flag (default False). But this flag is not allowed for 'positionals'. Instead with those 'argparse' looks at 'nargs' ('?*' are not required).

    The 'required' attribute of an argument (Action) is ignored during 'parse_args' until the end. At that time it makes an inventory of 'required' arguments that have not been seen, and potentially raises an error. That testing was changed in a relatively recent patch, and produced an unintended change in whether 'subparsers' were required or not. (I could look up those issues in needed).

    I'll think about creating the alternative HelpFormatter.

    @paulj3
    Copy link
    Mannequin

    paulj3 mannequin commented Feb 13, 2014

    The attached file shows how the default argument groups could be redefined, using 'required' as the criteria.

    I've implemented it as a method that is added to a subclass of ArgumentParser. This method is invoked after arguments are defined, prior to generating the help.

    The help looks something like this:

    usage: alt_grouping.py [-h] [-f FOO] -g GOO pos [baz]
    
    required:
      pos                required positional
      -g GOO, --goo GOO  required optional
    
    optional:
      -h, --help         show this help message and exit
      -f FOO, --foo FOO  optional
      baz                optional positional
    

    I was thinking of implementing this as a formatter subclass, but given the way the help is assembled, invoking this method from the parser is simpler.

    @maggyero
    Copy link
    Mannequin

    maggyero mannequin commented Dec 3, 2019

    I have just run into the same issue here: https://bugs.python.org/issue38950

    • I prefer Terry J. Reedy's "keyword arguments" as it is clear and consistent with "positional arguments".
    • But Steven Bethard 's "flag arguments" looks fine since it is well known to shell users.
    • Martin Panter's "options" looks okay since it is the standard name in GNU Coreutils (https://www.gnu.org/software/coreutils/manual/coreutils.html#Common-options). However I don't like it very much as it is still ambiguous: "options", like "optional arguments", still suggests something that is non required. And "options" is less consistent with "positional arguments" (nobody seems to have suggested "option arguments").
    • Oliver Smith's "switches" does not look okay because it is not general enough since it is commonly restricted to Boolean arguments.

    Anyway, the first 3 solutions are better than the current "optional arguments". What is blocking the approval of Martin Panter's PR?

    @kkarbowiak
    Copy link
    Mannequin

    kkarbowiak mannequin commented Jul 20, 2020

    It seems the discussion has so far revolved around 'optional' arguments with required=True.

    What about the other way around?

    While trying to set required=False for a positional argument raises an exception, it is still possible to make the positional argument effectively optional by nargs='?' (it is then printed in brackets in usage message).

    @rhettinger rhettinger added 3.9 only security fixes 3.10 only security fixes labels Jul 21, 2020
    @rhettinger rhettinger assigned rhettinger and unassigned docspython Jul 21, 2020
    @rhettinger
    Copy link
    Contributor

    Based on the attached survey of practices, I propose a minimal edit to the help display. Instead of "optional arguments:", we say "the options are as follows:".

    The use of the word "option" is dominant is in the CLI world, followed by "action" and "switch". The noun form "option" doesn't seem to cause the same confusion that arises in the adjective form "optional arguments" which strongly implies "not required".

    For the documentation, I suggest adding a sentence or two in the introduction to explain the terminology used throughout the rest of the argparse docs.

    @rhettinger rhettinger added docs Documentation in the Doc dir type-bug An unexpected behavior, bug, or error and removed type-feature A feature request or enhancement labels Dec 19, 2020
    @rhettinger
    Copy link
    Contributor

    Since any chance to the help output will likely break tests, marking this as 3.10 only.

    @rhettinger rhettinger added stdlib Python modules in the Lib dir and removed docs Documentation in the Doc dir 3.9 only security fixes labels Dec 19, 2020
    @rhettinger
    Copy link
    Contributor

    I also like Eric'c suggestion of just using "options:" instead of "optional arguments".

    @rhettinger
    Copy link
    Contributor

    Since this change will break tests that rely matching help output exactly, I would like to hear if there are any objections to replacing "optional arguments" with "options".

    The words "switch" or "flag" don't work as well because they imply on/off and don't encompass option that take arguments.

    @ericvsmith
    Copy link
    Member

    I wouldn't let breaking these tests deter you from improving the output. I think using "options" is an improvement.

    @rhettinger
    Copy link
    Contributor

    New changeset 41b223d by Raymond Hettinger in branch 'master':
    bpo-9694: Fix misleading phrase "optional arguments" (GH-23858)
    41b223d

    @hroncok
    Copy link
    Mannequin

    hroncok mannequin commented Jan 8, 2021

    Coudl this please be mentioned on https://docs.python.org/3.10/whatsnew/3.10.html ?

    At least two packages fail tests because of the change (ipython and sphinxcontrib-autoprogram).

    @paulj3
    Copy link
    Mannequin

    paulj3 mannequin commented Jan 8, 2021

    Since this issue is closed it might be a good idea to open a new one with this problem. And if possible identify the failed tests.

    We forgot to allow for the fact that working code/tests might be checking for specific help messages, checks the will fail when this group label is changed.

    @hroncok
    Copy link
    Mannequin

    hroncok mannequin commented Jan 8, 2021

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    y0urself added a commit to greenbone/gvm-tools that referenced this issue Jun 14, 2022
    y0urself added a commit to greenbone/gvm-tools that referenced this issue Jun 16, 2022
    * Adjust python version and PR template
    
    * Fix: Update pyproject.toml and dependencies
    
    * string for 3.10
    
    * Change the tests, to skip for 3.10 because of python/cpython#53903
    
    * Format with black
    pelavarre added a commit to pelavarre/shell2py that referenced this issue Nov 11, 2022
    … of Opts
    
    1 )
    
    mostly let them say 'options:' like later Python 3,
        or say 'optional arguments:' like Python 2
    
    <=
    
    i've seen 'options:' as early as Jun/2021 Python 3.9.6,
        even before the Oct/2021 Python 3.10
    
    i've seen 'optional arguments:' as late as Apr/2020 Python 2.7.18
    
        python3 --version && \
            python3 -c 'import argparse; argparse.ArgumentParser().print_help()'
    
        # argparse required arguments displayed under "optional arguments"
        open python/cpython#53903
    
    2 )
    
    but also temporarily accept the macOs Zsh regression
    
        % echo '⌃ ⌥ ⇧ ⌘ ← → ↓ ↑ ⎋ ⇥ ⋮' |hexdump -C |sed 's,a,a,'
        00000000  e2 8c 83 20 e2 8c a5 20  e2 87 a7 20 e2 8c 98 20  |?.. ?.? ?.? ?.. |
        sed: RE error: illegal byte sequence
        %
    pelavarre added a commit to pelavarre/byoverbs that referenced this issue Dec 15, 2022
    default to convert 'grep' and 'grep -H' output to Portrait from Landscape, such as
    
        git grep -i Exit_If_ArgDoc_NE |bin/awk.py --
    
    details =>
    
    -- bin/awk.py --
    first commit,
    ported from 'pelavarre/byobash/awk/headgrafs.awk'
    
    -- bin/awk.py --
    -- bin/sed.py --
    
    surface the Buffer of
    
        copy input to buffer to output, while running Awk Code to edit the buffer
        copy input to buffer to output, while running Sed Code to edit the buffer
    
    -- bin/byotools.py --
    -- bin/cal.py --
    -- bin/open.py --
    -- demos/keycaps.py --
    
    churn and jitter
    
        rename to 'def exit_if_argdoc_ne' from 'def exit_unless_doc_eq'
    
        jitter back into 'parser =' from 'argparser =',
            for that was not 'argument_parser = '
                for instances of 'argparse.ArgumentParser'
    
    -- bin/byotools.py --
    
    cite the Python bugs that make deploying 'def exit_if_argdoc_ne' into the Past difficult
    
        python/cpython#53903  <= options: / optional arguments:
    
        https://bugs.python.org/issue38438  <= usage: [WORD ... ] / [WORD [WORD ...]]
    
    -- bin/find.py --
    
    start dreaming of defining 'find.py -- |grep ...' to mean
    
        find / -mount 2>/dev/null |grep ...
    
    -- bin/ssh.py --
    
    capture the 'ssh -F /dev/null $USER@localhost'
        way of sharing Sh Terminal experience before eventually consistent configuration
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants