Skip to content

1.10.0

Latest

Choose a tag to compare

@bw2 bw2 released this 16 Nov 14:24

Behavior changes

Config file and env var lists no longer swallow subcommands or REMAINDER args

ConfigArgParse hands config file and env var values to argparse by adding them to the command line. For an option with nargs='+' or nargs='*', files = [a, b] becomes --files a b, and before 1.10.0 it was added just before the subcommand. So prog run was parsed as prog --files a b run: argparse read run as a third element in the list, and the subcommand silently never ran (files=['a', 'b', 'run'], command=None).

Now:

  • If an option comes before the subcommand, the list is added before that option, which ends the list: prog -v run gives files=['a', 'b'] and treats run as a subcommand.
  • Otherwise, if no other top-level options are specified, parsing stops with an error saying to set the list on the command line instead, or to specify another option before the subcommand.

The same applies to a nargs=REMAINDER positional: with files = [a, b], prog cmd x used to give files=['a', 'b', 'cmd', 'x'] and now stops with the same error.

Known issue in 1.10.0: with a subcommand, the error also happens when another option is set from a config file but the list comes from an env var or from a later config file. 1.9.0 handled those cases.

For nargs options, list elements that start with - or with an args-file prefix such as @ are now an error

For an option with nargs (eg. nargs='+'), a config file or env var list is added to the command line as separate args after the option, so files = [a, --admin] becomes --files a --admin. Argparse then reads --admin as an option rather than a file name. In 1.9.0 that silently set admin=True, even from an env var when --admin has no env var of its own. In the same way, with fromfile_prefix_chars='@', files = [a, @extra_args.txt] read more args from extra_args.txt. (A flat list for an action='append' option is passed as --files=a --files=--admin instead, and isn't affected.)

Now a list element that starts with - (or another of the parser's prefix_chars) or with one of its fromfile_prefix_chars stops parsing with an error that names the key. A lone - and negative numbers such as -2 still work.

Command-line values now win over config file and env var values in more cases

A config file or env var value is meant to be skipped when the same option is on the command line. 1.9.0 only spotted the option when it was written out in full, so it missed other forms argparse accepts and applied both values:

  • An abbreviated name: prog --ta e with tag = c in a config file gave tag=['c', 'e'] for an append option. Now tag=['e'].
  • Bundled short flags: prog -vv with verbose = 2 gave verbose=4 for a count option. Now verbose=2.
  • A value attached to a short option: prog -Ifoo with include = bar gave include=['bar', 'foo'] for an append option. Now include=['foo'].
  • An abbreviated name in a mutually exclusive group: with --level and --quiet in the group, prog --lev 3 with quiet = true failed with "argument --level: not allowed with argument --quiet". Now the config file's quiet is skipped, as it already was for prog --level 3.

When the command line has a subcommand or a --, this also affected options that keep a single value: prog --na y run with name = x in a config file gave name='x'; now it gives name='y'. Otherwise the command-line value came last and won anyway.

Options inside an args file now win too. With fromfile_prefix_chars='@', prog @args.txt, where args.txt holds --name and y on separate lines (argparse reads each line as one arg), gave the config file's name='x'; now it gives name='y'. A config file path given in an args file the same way (-c and my.conf on separate lines) is now opened; before, it was ignored.

Options that belong to a subcommand or to REMAINDER args no longer block config file and env var values

1.9.0 treated an option as already given when argparse actually hands it to a subcommand or to REMAINDER args. With name = x in a config file (or set by an env var), 1.9.0 dropped it (name=None) in each of these cases, and 1.10.0 gives name='x':

  • prog run --name y, where the subcommand run has its own --name that stores its value under another name (eg. dest='run_name').
  • prog cmd --name y, where everything from cmd on goes to a nargs=REMAINDER positional.
  • prog --exec cmd --name y, where --exec takes nargs=REMAINDER.

Likewise, a -h after -- or inside REMAINDER args (prog -- -h, prog cmd -h) made 1.9.0 skip config files entirely, as if help had been asked for. Config files are now read.

The config file option works the same way. 1.9.0 also opened a file named by a -c in a subcommand's args or in REMAINDER args, so prog --exec grep -c pattern failed with "Unable to open config file: pattern", and prog run -c run.conf (where run has its own -c) applied run.conf to the main parser too. Now only the main parser's own -c opens a file for it.

BooleanOptionalAction values from config files and env vars

For parser.add_argument('--feature', action=argparse.BooleanOptionalAction), a config file key can be written in its no- form, but 1.9.0 did the opposite of what it says: no-feature = true gave feature=True, and no-feature = false gave feature=False. Now no-feature = true gives feature=False, like --no-feature on the command line, and no-feature = false gives feature=True.

If the option also has a short alias, as in ("-f", "--feature"), a false value from a config file or env var used to set it to true (#367); it now sets it to false.

An option with only short names, such as -f and -F, has no no- form, so nothing on the command line can set it to false. When an env var asked for false anyway, 1.9.0 set it to true, or crashed with IndexError if the option had a single name. It now stops with an error such as "FEATURE can't set feature to False: none of -f/-F does".

parse_intermixed_args() now reads config files and env vars like parse_args() does

The same goes for parse_known_intermixed_args(). In 1.9.0, depending on the Python version, these either ignored config files and env vars or let their values override the command line.

exit_on_error=False now covers ConfigArgParse's own config file and env var errors

With exit_on_error=False, errors that ConfigArgParse itself finds in config files and env vars (eg. flag = maybe, or a line it can't parse) now raise argparse.ArgumentError instead of exiting the program. Errors that argparse reports, such as a wrong type, already did.

Other changes that can turn a previously accepted input into an error

  • Writing out a config file (with write_config_file() or an arg added with is_write_out_config_file_arg=True) now raises ValueError in these cases, where 1.9.0 wrote a file that read back differently:
    • In the default format, a string in square brackets, such as [1, 2] or the regex [a-z], was written as is and read back as a list.
    • With TomlConfigParser, some strings with a backslash, such as C:\temp\xdata (a backslash followed by x), were written so that the file couldn't be read back. Other backslashes, as in C:\Users\me, are written correctly in both versions.
    • TomlConfigParser([]) and IniConfigParser([], ...), which have no section to read from, wrote files that read back as nothing.
  • A number given for a count option in a config file or env var must now be digits only: verbose = -1 used to be ignored and +2 was accepted; both are now errors. true and false work as before.
  • Calling YAMLConfigFileParser().serialize() directly with an object such as a Path raises an error, instead of writing a !!python/object tag its own loader can't read.

Other reading changes

  • Default config format: a key with no = and a comment after it, such as verbose # be loud, is now read as verbose = true. 1.9.0 read the comment as the value, so it rejected that line for a flag and set name #value to #value. name = #value is unchanged.
  • Wildcards in default_config_files are read in alphabetical order, so the alphabetically last matching file wins. Before, the order depended on the file system.
  • IniConfigParser(['my_tool', 'DEFAULT'], ...): the named section now wins over [DEFAULT].

New

  • Python 3.15 is now tested and listed as supported. On 3.15, parser.print_help(file) no longer writes terminal color codes into file when stdout is a terminal but file is not (eg. a log file).
  • nargs=1 options accept a list from config files and env vars (size = [5]).
  • configargparse.already_on_command_line(), the helper that checks whether an option is already on a command line, takes an optional parser= argument. With it, the helper follows the rules above: it recognizes abbreviations, bundled short flags and attached values, and ignores args that go to a subcommand or to REMAINDER args. Calls without it work as before.

Fixes

Writing config files

  • In the default config format, list values are written as JSON (eg. files = ["a,b", "c"]), so they read back unchanged (#366); before, ['a,b', 'c'] read back as three items. ConfigArgParse 1.3 and later can read these files.
  • Options given in abbreviated or bundled form (--gen hg38, -vv) are written; they used to be left out.
  • Values that a type= function turned into other objects, such as type=json.loads dicts and FileType files, are written as the string they came from, so they read back.
  • Enum members, tuple defaults of multi-value options, nargs=1 values, append_const flags (each given once), and append/count options with defaults now read back with the values they had when the file was written.
  • Defaults that can't be read back (eg. default=sys.stdout, empty lists), defaults of mutually exclusive options, and the config file path option (when given on the command line, by an env var or as a default) are left out of written files.
  • YAML writes Path and IntEnum values as plain values. The default format and IniConfigParser quote values that would otherwise read back differently (tests from #368), and IniConfigParser with split_ml_text_to_list=True writes an empty list as [] instead of losing it.
  • An arg that writes the config file with type=pathlib.Path no longer crashes after writing.

Reading config files and env vars

  • Options inherited through parents= from an argparse parser created before ConfigArgParse was imported no longer crash parsing or help output.
  • Config file and env var values no longer cause an error or get lost when an option's value is a subcommand name (prog --target build build), when -- follows the subcommand (prog run -- a), or when a nargs=REMAINDER positional's args include an option (prog cmd --opt).
  • The config file path is found in a bundle of short flags (-vc conf), no longer confused with an option whose name starts the same (--set next to --settings), and, with allow_abbrev=False, no longer read from an abbreviation such as --sett.
  • IniConfigParser strips a value written on the line after key =.
  • Default format: line numbers in "Unexpected line N" errors start at 1.
  • A YAML key read as a bool or number (yes: 1), a stray % in an IniConfigParser value, flag = [a, b] for a store_true option, and verbose = lots for a count option now give a clear error instead of a traceback.
  • A TOML file that sets tool = "x" when the parser reads section tool.my_tool, and a bytes path in default_config_files, no longer crash. An empty or comment-only YAML file is now read as having no settings instead of stopping with an error.

Help and print_values()

  • Calling format_help() more than once gives the same text; later calls used to drop the config file paragraph.
  • Env var names from auto_env_var_prefix show in the help before anything is parsed.
  • CompositeConfigParser's help labels ConfigparserConfigFileParser instead of showing ??, and IniConfigParser's help says keys are lowercased.
  • print_values() writes to the current sys.stdout, shows @file args expanded, and no longer lists a positional's default when it was given.

Full changelog: v1.9.0...v1.10.0