shutil.copytree doesn't give control over directory permissions

[jmcneil]

BPO	1666318
Nosy	@birkenfeld, @pitrou, @tiran, @devdanzin, @tarekziade, @merwok, @vajrasky
Files	makedirs_function.patch test1666318.patch: patch to test_shutil.py testing for copytree preserving permissions test1666318.patch: updated patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-08-16.17:38:07.237>
created_at = <Date 2007-02-22.16:26:35.000>
labels = ['easy', 'tests', 'type-feature', 'library']
title = "shutil.copytree doesn't give control over directory permissions"
updated_at = <Date 2013-08-16.17:38:07.213>
user = 'https://bugs.python.org/jmcneil'

bugs.python.org fields:

activity = <Date 2013-08-16.17:38:07.213>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2013-08-16.17:38:07.237>
closer = 'pitrou'
components = ['Library (Lib)', 'Tests']
creation = <Date 2007-02-22.16:26:35.000>
creator = 'j_mcneil'
dependencies = []
files = ['24816', '31081', '31316']
hgrepos = []
issue_num = 1666318
keywords = ['patch', 'easy']
message_count = 16.0
messages = ['31307', '31308', '31309', '141822', '155580', '155613', '155628', '193900', '193904', '194025', '194028', '194031', '194981', '195355', '195374', '195376']
nosy_count = 13.0
nosy_names = ['georg.brandl', 'jlgijsbers', 'thomaswaldmann', 'pitrou', 'catherinedevlin', 'christian.heimes', 'ajaksu2', 'tarek', 'eric.araujo', 'daniel.ugra', 'mcjeff', 'python-dev', 'vajrasky']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1666318'
versions = ['Python 3.3', 'Python 3.4']

[jmcneil]

I am using shutil.copytree to setup new user home directories within an automated system. The copy2 function is called in order to copy individual files and preserve stat data.

However, copytree simply calls os.mkdir and leaves directory creation at the mercy of my current umask (in my case, that's daemon context - 0).

I've got to then iterate through the newly copied tree and set permissions on each individual subdirectory.

Adding a simple copystat(src, dst) on line 112 of shutil.py fixes the problem.

The result should be uniform; either preserve permissions across the board, or leave it to the mercy of the caller. I know there's an enhancement request already open to supply a 'func=' kw argument to copytree.

[jmcneil]

python -V
Python 2.4.3

on

Linux marvin 2.6.18-1.2257.fc5smp #1 SMP Fri Dec 15 16:33:51 EST 2006 i686 i686 i386 GNU/Linux

[thomaswaldmann]

I can confirm this bug.

For MoinMoin wiki, we solved that by duplicating this function to our own tree and we did exactly the proposed change:

Adding a simple copystat(src, dst) on line 112 of shutil.py (right below os.mkdir) fixes this problem.

I also suggest to remove that XXX comment in that function. With the proposed fix, it gets usable and thus is not only example code any more.

BUT: for avoiding problems on win32, a second fix needs to be done. For MoinMoin, we call this wrapper around shutil.copystat from our copytree function:

def copystat(src, dst):
    """Copy stat bits from src to dst

This should be used when shutil.copystat would be used on directories
on win32 because win32 does not support utime() for directories.

According to the official docs written by Microsoft, it returns ENOACCES if the
supplied filename is a directory. Looks like a trainee implemented the function.
"""
if sys.platform == 'win32' and S_ISDIR(os.stat(dst)[ST_MODE]):
    if os.name == 'nt':
        st = os.stat(src)
        mode = S_IMODE(st[ST_MODE])
        if hasattr(os, 'chmod'):
            os.chmod(dst, mode) # KEEP THIS ONE!
    #else: pass # we are on Win9x,ME - no chmod here
else:
    shutil.copystat(src, dst)

As you see, some special treatment is needed for win32/nt and directories - and directories are exactly the use case used by the fixed copytree, so a fixed copystat is needed or copytree will crash on win32/nt.

[merwok]

The result should be uniform; either preserve permissions across the
board, or leave it to the mercy of the caller.

I’m surprised by this report, as the code does a copystat(sourcedir, targetdir) since 2004 (bpo-1048878). Anyhow, I think the second option would be better.

I know there's an enhancement request already open to supply a
'func=' kw argument to copytree.

This is now implemented (copy_function argument), but does not help with directories, only with files. Recently I needed to call copytree and change the permissions of the destination, so I wanted to use a two-line custom copy_function (one line to call shutil.copy, one line for os.chmod), but I hadn’t understood that the custom function was not used for directory creation. I found out that shutil does a copystat on the created directories behind my back. So, I think this problem can be solved in a nice and b/w-compatible way by adding a new keyword argument, makedirs_func. I would like to fix this in stable versions too, but if another core developer argues that it should be considered a new feature, I won’t argue.

[mcjeff]

I made the change suggested in the last comment, patch is attached. Trying to clean up any bugs I've got my name on!