shutil.copytree behavior is inconsistent with copyfile

[josephhgarvin]

BPO	8125
Nosy	@pitrou, @briancurtin

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-03-12.16:36:03.973>
created_at = <Date 2010-03-12.16:25:06.057>
labels = ['type-bug', 'library']
title = 'shutil.copytree behavior is inconsistent with copyfile'
updated_at = <Date 2010-03-12.23:42:35.432>
user = 'https://bugs.python.org/josephhgarvin'

bugs.python.org fields:

activity = <Date 2010-03-12.23:42:35.432>
actor = 'joseph.h.garvin'
assignee = 'none'
closed = True
closed_date = <Date 2010-03-12.16:36:03.973>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2010-03-12.16:25:06.057>
creator = 'joseph.h.garvin'
dependencies = []
files = []
hgrepos = []
issue_num = 8125
keywords = []
message_count = 4.0
messages = ['100943', '100944', '100945', '100981']
nosy_count = 3.0
nosy_names = ['pitrou', 'brian.curtin', 'joseph.h.garvin']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8125'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2', 'Python 3.3']

[josephhgarvin]

shutil.copyfile's behavior is to replace the dst file if it already exists. shutil.copytree requires that the destination not already exist, and throws an OSError if it does. I see 3 problems with this behavior:

1. It's inconsistent with copyfile

2. It's inconsistent with 'cp', which is what users would be using if they were writing shell script. Given shutil's namesake I assume it's supposed to help make python a viable shell script replacement.

3. It makes it difficult to use copytree with tempfile.mkdtemp(). If I want to make a temporary directory and copy a folder into it, I need to copytree to a nonexist subfolder then move all the files down a level or resort to other unpythonic hacks. In my project I copy unit test resources to a temp folder because they're manipulated throughout the test and I want to keep the originals. I imagine that this is a common use case.

That said, if anyone depended on shutil.copytree failing to test whether or not a folder exists, changing this would break compatibility, so if it can't be changed, maybe a shutil.mergetree would be appropriate.

[pitrou]

1. It's inconsistent with copyfile

I think it's ok. When using copyfile you can easily check that the destination file doesn't exist, while with copytree it's harder to do the same for *all* destination files. Therefore it makes sense for copytree to raise an error in this case.

Given shutil's namesake I assume it's supposed to help make python a viable shell script replacement.

It doesn't mean the semantics have to be exactly the same. Actually, many Python users are under Windows where semantics will be different.

It makes it difficult to use copytree with tempfile.mkdtemp(). If I want to make a temporary directory and copy a folder into it, I need to copytree to a nonexist subfolder then move all the files down a level or resort to other unpythonic hacks.

I actually don't understand your problem. If you are creating a temporary directory, it is empty so there isn't any risk of hitting an already existing file.

In any case, I don't think your arguments are good enough to warrant breaking compatibility with older versions.

[briancurtin]

Also, keep in mind that shutil.copytree is more of an example than the be-all-end-all directory copy function, as stated in both the documentation and source code.

[josephhgarvin]

Sorry I wasn't terribly clear explaining the problem with mkdtemp. Say you create your temporary folder, /tmp/foo. Now you have a regular folder bar laid out like so:

bar/
blarg1.ext
blarg2.ext
subdir/
blarg3.ext
blarg4.ext

I'd like this to be the result of copying:

/tmp
/foo
blarg1.ext
blarg2.ext
subdir/
blarg3.ext
blarg4.ext

Basically, I'd like to copy bar/ to /tmp/foo such that the paths of the files relative to bar/ are the same as the paths of the copied files relative to /tmp/foo. AFAICT there's no easy way to do this with copytree's current behavior. Because tempfile.mkdtemp() creates /tmp/foo to start, you can't use copytree to do this.

But now I realize copy tree wants to copy the source directory as a directory, so you always get /tmp/foo/bar. What I wanted was copy(recursive_glob(src, "*"), dst). That would give you the effect you normally get from a utility like 'cp'.

It doesn't mean the semantics have to be exactly the same. Actually, many Python users are under Windows where semantics will be different.

I believe 'copy' has the same cp-like semantics I desire under Windows, but it's been quite some time since I used it.

So, not a bug, but sticks out to me as missing. Might make sense someday to have a keyword arg to copytree that would make it behave this way.