--help broken when there's a required option set in pytest_addoption()

[jessecarbon]

Creating this after chatting on #pylib. When adding command line options using pytest_addoption(), if any of the arguments are set to be required (e.g. with required=True) it breaks the ability to run --help (unless you also provide the required option). This is in contrast to how a vanilla argparse config works.

Example of this not occurring with argparse:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--foo', required=True)
args = parser.parse_args()

--help still works without the required argument, but executing the file does not (expected behavior):

$ python vanilla.py --help
usage: vanilla.py [-h] --foo FOO

optional arguments:
  -h, --help  show this help message and exit
  --foo FOO

$ python vanilla.py
usage: vanilla.py [-h] --foo FOO
vanilla.py: error: the following arguments are required: --foo

Example of this not working as expected with pytest:

$ cat conftest.py
def pytest_addoption(parser):
    parser.addoption("--foo", required=True)

--help is broken, unless you also provide the required option:

$ pytest --help
usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: the following arguments are required: --foo

$ pytest --help --foo bar
usage: pytest [options] [file_or_dir] [file_or_dir] [...]

positional arguments:
  file_or_dir
...

My environment:

$ pip list
pip (8.1.2)
py (1.4.31)
pytest (3.0.3)
setuptools (20.10.1)

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.10.5
BuildVersion:   14F1912

• Include a detailed description of the bug or suggestion
• pip list of the virtual environment you are using
• pytest and operating system versions
• Minimal example if possible

[RonnyPfannschmidt]

@hpk42 is this potentially a regression slipped as we adapted argparse?

[hpk42]

I guess it's more because of extra things we do on top of argparse because @jessecarbon already noted that plain argparse works fine.

[RonnyPfannschmidt]

@hpk42 i meant it slipped in when we switched the parser

[segevfiner]

This is due to the way that argparse handles required arguments and it's default --help option.

Required arguments are checked during parse_[known]_args and in order to avoid this check, argparse defines an help action which will print the help and than immediatly exit the process (sys.exit). Pytest instead defines it's own --help argument with a store_true action, which means we get hit by the required argument validation done by argparse before we get a chance to even look at the parsed arguments.

Pytest will have to define it's own argparse action that will print help, and exit. And use it instead of store_true in order to fix this. Trying to implement required by our self will be difficult since argparse doesn't expose the set it uses to validate this. (seen_actions)

Workaround

Don't use required and validate that you have been given the options you require in pytest_sessionstart. You can raise pytest.UsageError to get a "nice" exit message.

Note though that both pytest_sessionstart and pytest.UsageError appear to be undocumented.

[RonnyPfannschmidt]

pytest_configure should be used to act on options,
the docs need a new issue

[segevfiner]

It won't work here since `pytest_configure` is also called before `--help` while `pytest_sessionstart` is only called while actually starting a test session and *not* when requesting `--help`. בתאריך 30 במאי 2017 12:44,‏ "Ronny Pfannschmidt" <notifications@github.com> כתב:

…

pytest_configure should be used to act on options, the docs need a new issue — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1999 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AXlg_xAzLyT3RFhx5YHD0YzDidK2dmDMks5r--TwgaJpZM4KT9dc> .

[RonnyPfannschmidt]

interesting point, sounds like a structural bug in pytest to me then

[The-Compiler]

@RonnyPfannschmidt "structural" how?

@segevfiner Your comment above seems spot-on to me - any chance you want to work on a pull request for this? 😉

[RonnyPfannschmidt]

@The-Compiler our cli integration api allows for configurations that actually break pytest --help

[The-Compiler]

@RonnyPfannschmidt It allows for required arguments, which is a good thing. It should just handle --help correctly, like @segevfiner explained above.

[segevfiner]

@The-Compiler I tried. But in order to do this I need access to the config object in the HelpAction but it needs to be registered in pytest_addoption which only has access to parser, and I think config only gets created later anyhow.

class HelpAction(Action):
    def __init__(self,
                 option_strings,
                 dest=SUPPRESS,
                 default=SUPPRESS,
                 help=None):
        super(_HelpAction, self).__init__(
            option_strings=option_strings,
            dest=dest,
            default=default,
            nargs=0,
            help=help)

    def __call__(self, parser, namespace, values, option_string=None):
        # Where should config come from?
        config._do_configure()
        showhelp(config)
        config._ensure_unconfigure()
        parser.exit()

Any suggestions on how to handle this?

[The-Compiler]

Ah, good point. I'm not sure what a good solution for this would be, but I'm not that familiar with that part of pytest's code.

[RonnyPfannschmidt]

we do have early and later config objects later in parse/preparse steps, but we would have to bring the help action into that

[segevfiner]

I'm still trying to work on a pull request BTW. I had a different idea on how to do this. But I still need to get the tests passing 😛