Rewrite argument parsing

[bmw]

We currently use the largely unmaintained configargparse and a number of hacks like magic default values and reaching into the internals of configargparse/argparse to get the behavior we want. We should rewrite our argument parsing to use another library like click or fire.

cc @erikrose, @ohemorange

[erikrose]

I've had excellent experiences with click. See https://github.com/mozilla/dxr/blob/master/dxr/cli/__init__.py for the top-level entrypoint in DXR's click-based CLI. click is particularly good at subcommands.

[ohemorange]

I don't know anything about click, but my friend wrote fire so we could probably get good support. I'm all for cleaner code everywhere!

[erikrose]

Then we should probably make a pro/con list and see what pops out! Nice to have 2 good choices.

[anarcat]

i don't know if click will cover the use cases currently covered by configargparse. first, it doesn't support config files out of the box - that might be done by a third party module like click_config_file. that config file support doesn't support writing config files the way certbot currently does either.

then click is way more than a commandline parser - it's a whole TUI (text user interface) driver. it hijacks the main control flow of your program and makes testing ... different, for example. it supports progress bars and ANSI art and whatnot. it is very opiniated about how things should be and you might find it difficult to work with if you deviate from canon.

fire is much simpler, but also has the limitation of not working with config files.

another option people often favor is docopt which also doesn't support config files (let alone writing them) but there is some sample code to parse config files that is surprisingly clean.

there aren't that many options here - rewriting the commandline parser is a huge deal in certbot. argparse, while it's weird and complicated, is one of the best things we have right now. i wonder if the best approach might not be to fix the issues in configargparse instead of rewriting half of certbot. ;) and there's been some action on the issue of finding a new maintainer on that side, so I wouldn't give up on it already...

[bmw]

Thanks for the thoughts @anarcat! For what it's worth, we don't use configargparse for writing configuration files and instead use configobj.

[anarcat]

well that makes things easier. for what it's worth, I ended up ditching all those methods for my own implementation and just rolled my own. I suggest you take that into consideration...

https://gitlab.com/anarcat/undertime/blob/acd4a7c73f7e9c727a8a572884a46bcb696682c9/undertime#L102

it's around 60 lines of code, but i think it's quite elegant, all things considered. it uses pyyaml to parse the config file, but that's easy to override... heck, it's short enough that I can just paste it here:

class ConfigAction(argparse.Action):
    """add configuration file to current defaults"""
    def __init__(self, *args, **kwargs):
        """the config action is a search path, so a list, so one or more argument"""
        kwargs['nargs'] = '+'
        super().__init__(*args, **kwargs)

    def __call__(self, parser, ns, values, option):
        """change defaults for the namespace, still allows overriding
        from commandline options"""
        for path in values:
            parser.set_defaults(**self.parse_config(path))

    def parse_config(self, path):
        """abstract implementation of config file parsing, should be overriden in subclasses"""
        raise NotImplementedError()


class YamlConfigAction(ConfigAction):
    """YAML config file parser action"""
    def parse_config(self, path):
        try:
            with open(os.path.expanduser(path), 'r') as handle:
                return yaml.safe_load(handle)
        except (FileNotFoundError, yaml.parser.ParserError) as e:
            raise argparse.ArgumentError(self, e)


class ConfigArgumentParser(argparse.ArgumentParser):
    """argument parser which supports parsing extra config files

    Config files specified on the commandline through the
    YamlConfigAction arguments modify the default values on the
    spot. If a default is specified when adding an argument, it also
    gets immediately loaded.

    This will typically be used in a subclass, like this:

            self.add_argument('--config', action=YamlConfigAction, default=self.default_config())

    """

    def _add_action(self, action):
        # this overrides the add_argument() routine, which is where
        # actions get registered. it is done so we can properly load
        # the default config file before the action actually gets
        # fired. Ideally, we'd load the default config only if the
        # action *never* gets fired (but still setting defaults for
        # the namespace) but argparse doesn't give us that opportunity
        # (and even if it would, it wouldn't retroactively change the
        # Namespace object in parse_args() so it wouldn't work).
        action = super()._add_action(action)
        if isinstance(action, ConfigAction) and action.default is not None:
            # fire the action, later calls can override defaults
            try:
                action(self, None, action.default, None)
            except argparse.ArgumentError:
                # ignore errors from missing default
                pass

    def default_config(self):
        """handy shortcut to detect commonly used config paths"""
        return [os.path.join(os.environ.get('XDG_CONFIG_HOME', '~/.config/'), self.prog + '.yml')]

I simply use it like this:

parser = ConfigArgumentParser()
parser.add_argument('--config', action=YamlConfigAction, default=ConfigArgumentParser.default_config())

This will load the config from ~/.config/argv0.yml as default values for other parameters on the commandline, or from whatever config file specified by the --config argument. It is overridable on the commandline and everything, but doesn't write its own config file either.

Anyways, I hope that helps... The TL;DR: of all the above is it's easy to preset defaults from a config file for simple parsers (parser.set_defaults() with a dict) but it becomes possibly more complicated with sub-parsers, in which case you might want to have add more tweaks to the above...

[ronhanson]

If you are having issues with Configargparse, or new feature requests, feel free to fill bug reports and do PR. I will try my best to move the project forward and fix the existing bugs.

[anarcat]

@ronhanson there are many issues in certbot that are ultimately upstream issues in configargparse. for example, bw2/ConfigArgParse#104 and #6049 are two issues here specific to the project...

[stale]

We've made a lot of changes to Certbot since this issue was opened. If you still have this issue with an up-to-date version of Certbot, can you please add a comment letting us know? This helps us to better see what issues are still affecting our users. If there is no activity in the next 30 days, this issue will be automatically closed.

[bmw]

If we ever get to this, it'll honestly probably be years from now, but I personally still think this should be done. Our current CLI parsing is very complex, relies on a bunch of hacky custom code, and has caused a number of issues for us (see some of the issues referencing this one for a few examples).

[ohemorange]

When we do this, we should keep in mind that verbose is a count argument, and therefore can be given with no arguments, or with a number in the cli.ini file. Then, we can deprecate the unlisted verbose-level argument.

[wgreenberg]

looking into this a bit, at first it seemed like click would be a great choice, but as mentioned earlier in this thread it's very opinionated. for example, it requires strict separation of arguments between commands and subcommands, e.g. certbot -d foo.com certonly means -d belongs to the certbot command, and thus you wouldn't be able to do certbot certonly -d foo.com. there's workarounds for this (see pallets/click#108), but nonetheless it'd be an annoying hack we'd have to pepper in everywhere.

fire looks like a convenient way to get a CLI up and running on a smaller project, but it doesn't look like it supports a lot of our use cases. for example, it doesn't seem to support multiple flags of the same type (-d foo.com -d bar.org), and it seems you can only set help text through docstrings.

i'll keep looking into the click hack and see if i can make it more palatable.

[bmw]

Ooo interesting. Thanks for doing that research! I'm very curious if any of these libraries help with our "default detection" stuff we've got going on.

Also, I think the Python CLI landscape has shifted a bit since this issue was initially created. Most of these are false leads, but I found https://python.libhunt.com/categories/181-command-line-application-development which lists all the libraries I think I've ever heard anyone talk about. In addition to docopt which was mentioned above, I've also heard good things about typer which is based on click.

[bmw]

Also also, I haven't looked into how hacky the workaround for click is, but in case you hadn't seen it already, I wanted to share that our support for arguments either on certbot or the subcommand is a hack on argparse too. See

certbot/certbot/certbot/_internal/cli/helpful.py

Lines 300 to 322 in 8037321

	def determine_verb(self) -> None:
	"""Determines the verb/subcommand provided by the user.
	This function works around some of the limitations of argparse.
	"""
	if "-h" in self.args or "--help" in self.args:
	# all verbs double as help arguments; don't get them confused
	self.verb = "help"
	return
	for i, token in enumerate(self.args):
	if token in self.VERBS:
	verb = token
	if verb == "auth":
	verb = "certonly"
	if verb == "everything":
	verb = "run"
	self.verb = verb
	self.args.pop(i)
	return
	self.verb = "run"

[wgreenberg]

updating this: although i ended up getting click to be flexible enough to handle certbot's (shall we say) freeform handling of CLI args, unfortunately argparse is built into some of our plugin interfaces, and so any migration away from it would almost certainly be a backwards compatibility-breaking change. since that would necessitate a major version change, i'll punt on it for now and instead focus on patching the existing argparse system to fix #9598

[github-actions]

We've made a lot of changes to Certbot since this issue was opened. If you still have this issue with an up-to-date version of Certbot, can you please add a comment letting us know? This helps us to better see what issues are still affecting our users. If there is no activity in the next 30 days, this issue will be automatically closed.

[github-actions]

This issue has been closed due to lack of activity, but if you think it should be reopened, please open a new issue with a link to this one and we'll take a look.