shutil.move raises OSError when copystat fails

[piman]

BPO	1438480
Nosy	@merwok, @websurfer5, @eshan Singhal
Files	shutil.py.diff: Fix copy2/copy

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2006-02-25.05:26:29.000>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'shutil.move raises OSError when copystat fails'
updated_at = <Date 2021-12-26.15:58:26.319>
user = 'https://bugs.python.org/piman'

bugs.python.org fields:

activity = <Date 2021-12-26.15:58:26.319>
actor = 'ajaksu2'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2006-02-25.05:26:29.000>
creator = 'piman'
dependencies = []
files = ['1911']
hgrepos = []
issue_num = 1438480
keywords = ['patch']
message_count = 6.0
messages = ['27612', '27613', '27614', '141827', '221789', '358270']
nosy_count = 5.0
nosy_names = ['thomaswaldmann', 'piman', 'eric.araujo', 'Jeffrey.Kintscher', 'eshan-singhal']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1438480'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-42948: Fix shutil.move() on permission-restricted filesystems #141697
• gh-42948: add preserve_timestamps to shutil.copystat and shutil.copy2 #142098

[piman]

If you are on a Linux system, and shutil.move a file
from anywhere onto a partition where you have write
permission but are not the owner, os.utime will fail
with an EPERM OSError. This can (and did) happen moving
a file on a vfat partition mounted with umask=0000, so
every user had read/write/execute but all files were
owned by root.

This happens in shutil.copystat, so shutil.move doesn't
remove the old file. The resulting error code (OSError,
EPERM) is not distinguishable from several other
permission errors that can happen during shutil.move,
even though a failure to set a utime is not fatal for
most move operations (mv(1) succeeds, for example).

I would suggest either ignoring an EPERM from copystat,
or catching it in shutil.copy2 and raising a more
specific exception so that it can be easily
distinguished from genuine failure due to permissions.

[piman]

Logged In: YES
user_id=796

The attached patch causes shutil.copy and shutil.copy2 (and
so copytree, and move) to ignore EACCES and EPERM from
copymode and copystat respectively. Other errors are
reraised with the original traceback. Any error copying the
file, rather than the metadata, is still considered fatal.
User calls to copymode and copystat still fail as before.

Justification: On Linux, utime(2) says the distinction
between EACCES and EPERM is blurry and "Linux is not careful
to distinguish between the EACCES and EPERM" so the patch
ignores both. In my opinion, anyone wanting to deal with
low-level details like mtimes and modes is probably not
using shutil, so ignoring is better than a new exception
(which would just result in complicated new code, and
surprise old code). It's also easy for a caller to check if
the mode/utimes were copied, if it's important to them.

[thomaswaldmann]

See also http://sourceforge.net/tracker/index.php?func=detail&aid=1666318&group_id=5470&atid=105470 for a similar problem with copystat.

[merwok]

In my opinion, anyone wanting to deal with low-level details like
mtimes and modes is probably not using shutil

As a moderately experienced UNIX user, I would say that file permissions are not low-level details. I see shutil functions as convenient wrappers to automate looping and calling a function, but I do care about file permissions.

If people are still interested in this modification, I think it would need a backward-compatible change, for example in the form of a new argument ignore_permission_errors, or a more generic onerror callback like rmtree has.

[BreamoreBoy]

The patch would need changing to allow for the follow_symlinks parameter and the backward compatibility issues mention in msg141827. Do we wait for an updated patch, close as "won't fix" or what?

[EshanSinghal]

Amending shutil.copy and shutil.copy2 to expose a method of ignoring the permission errors rather than changing the current behaviour would not fix the original issue without further changes in consumers of these functions to pass through the new parameter (i.e. shutil.move in this scenario).

This may be undesirable as copy_function is an optional parameter to shutil.move and so there would have to be a specialcase guard/check to see if the copy_function is shutil.copy or shutil.copy2 when passing in a new ignore_permission_errors parameter.

I propose a new issue to track adding functionality to optionally ignore permission issues for shutil.copy/shutil.copy2, and this issue can remain open.