Better path handling with argparse

[AustinScola]

BPO	42572
Nosy	@rhettinger, @ericvsmith, @masklinn, @miss-islington, @AustinScola
PRs	bpo-42572: Improve argparse docs for the type parameter. #23849 [3.9] bpo-42572: Improve argparse docs for the type parameter. (GH-23849) #23869

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2020-12-20.18:51:41.576>
created_at = <Date 2020-12-04.22:42:15.845>
labels = ['type-feature', '3.9', '3.10', 'docs']
title = 'Better path handling with argparse'
updated_at = <Date 2020-12-20.18:51:41.576>
user = 'https://github.com/AustinScola'

bugs.python.org fields:

activity = <Date 2020-12-20.18:51:41.576>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2020-12-20.18:51:41.576>
closer = 'rhettinger'
components = ['Documentation']
creation = <Date 2020-12-04.22:42:15.845>
creator = 'ascola'
dependencies = []
files = []
hgrepos = []
issue_num = 42572
keywords = ['patch']
message_count = 16.0
messages = ['382542', '382599', '382692', '382696', '382704', '382707', '382716', '382758', '382760', '382765', '382766', '382772', '383331', '383372', '383434', '383439']
nosy_count = 6.0
nosy_names = ['rhettinger', 'eric.smith', 'xmorel', 'paul.j3', 'miss-islington', 'ascola']
pr_nums = ['23849', '23869']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue42572'
versions = ['Python 3.9', 'Python 3.10']

[AustinScola]

One of the types of arguments that I find myself most often passing to argparse.ArgumentParser is paths. I think that I am probably not alone in frequent usage of paths as arguments. Given this, it would be extremely helpful to have an argparse.Action in argparse or an type similar to FileType that converted the string to a path. A path type factory could also have an arguments to optionally check if the path exists, or is a directory, or other similar predicates.

[ericvsmith]

I think this would be a type, not an action.

I'm not sure this would pass the bar of something that should be added to the stdlib. But in any event, it should be developed on PyPI first, perhaps by adding it to argparse-types.

[AustinScola]

Hey Eric,

Thanks for the response. I'm unfamiliar with the process of adding features to the language. Would you mind explaining to me what some of the qualifications are for getting something added to the stdlib? And also what role packages on PyPI play in that process?

I am also willing (and interested) in helping out in implementing this, but I don't want to step on anyone's toes so if the offer for help is rejected I will not be upset.

Thanks!

[ericvsmith]

Hi, Austin.

If it's something that can be implemented in a library (which this suggestion qualifies as), then we typically want to see it on PyPI and to gain some traction there. I only suggested argparse-types because it also has some argparse add-ons. But you could certainly make a new package on PyPI.

But even then I'm not sure this would make it into the stdlib. For add-on functionality we're usually happy to leave things on PyPI. On the other hand, paths are somewhat more fundamental than some of the things in argparse-types. You might want to bring it up on the python-ideas mailing list and see how much support you get there.

[AustinScola]

Awesome, thank you for the guidance Eric. I'll start a thread on the python-ideas mailing list, gauge the level of support, and go from there.

[paulj3]

What exactly do you do with a path argument?

Admittedly I'm not expert with os and os.path modules, but isn't a path just a string passed to a function such as cwd(), or joined to another create a filename.

I don't know what a 'path' type or action would do.

[masklinn]

What exactly do you do with a path argument?

Because they mention "convert[ing] the string to a path", I would expect an output of pathlib.Path, optionally checked for existence / non-existence and / or kind (file, directory, symlink, ...).

Obviously it is rather easy to bring your own, but OP's expectation is that paths input is common enough (especially in smaller standalone scripts I would expect) that having a helper in a standard library would be... helpful.

[paulj3]

The pathlib.Path is new since I paid much attention to os matters (I cut my teeth on py2.5).

Off hand it looks like the user could

    import pathlib
    parser.add_argument('-p', type=pathlib.Path)

to convert a string into a Path object.

A custom type function could call this, and apply any desired methods before returning the Path object. But that should be up to the user, not the argparse developers.

Importing path specific modules such as pathlib (which in turn has a lot of imports) goes against the attempt to reduce the number of unnecessary imports with modules like argparse.

https://bugs.python.org/issue30152
Reduce the number of imports for argparse

After this diet argparse still imports os, and uses:

os.path.basename

I might also note that argparse has only one custom 'type' function, the FileType factory. Other common type functions like 'int' and 'float' are Python builtin's. Adding a custom 'bool' (to be used instead of the builtin 'bool') has been rejected.

https://bugs.python.org/issue37564

json, yaml, and datetime types have also been rejected
https://bugs.python.org/issue35005

[ericvsmith]

The more I think about this, the more I think it shouldn't be in the stdlib. paul.j3 is correct that the simple case is just type=pathlib.Path.

For something more adventurous you could start with:

@dataclass(eq=True, frozen=True)
class ArgumentPath:
    must_exist: bool = False
    # Add other conditions as needed.

    def __call__(self, val):
        result = Path(val)
        if self.must_exist:
            if not result.exists():
                raise ValueError(f"path {result} must exist")
        return result

The reason I think this shouldn't be in the stdlib is that there are race conditions here between when you inspect the filesystem and when you'd actually use the path. What if the file was deleting, or it went from being a directory to a file?

I think the best advice is to use type=pathlib.Path, and handle anything else when you try to cd, or open, or whatever it is you're doing with the path.

It probably wouldn't hurt to document type=pathlib.Path in the argparse docs.

[rhettinger]

Agreed that we should not include new functionality here and instead just update the docs to show what can currently be done.

I'll write-up a documentation patch for this.