shutil.move doesn't handle ENOTSUP raised by chflags on OS X

[grobian]

BPO	14662
Nosy	@pitrou, @ned-deily, @merwok, @hynek
Files	expand-chflags-catch-2.7.diff: Patch to catch also ENOTSUP in copystat() for Python 2.7 expand-chflags-catch-tip.diff expand-chflags-catch-3.2.diff expand-chflags-catch-2.7-v2.diff expand-chflags-catch-3.2-v2.diff expand-chflags-catch-tip-v2.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/hynek'
closed_at = <Date 2012-05-11.01:20:47.729>
created_at = <Date 2012-04-24.17:58:51.591>
labels = ['type-bug', 'library']
title = "shutil.move doesn't handle ENOTSUP raised by chflags on OS X"
updated_at = <Date 2012-05-11.01:20:47.727>
user = 'https://bugs.python.org/grobian'

bugs.python.org fields:

activity = <Date 2012-05-11.01:20:47.727>
actor = 'ned.deily'
assignee = 'hynek'
closed = True
closed_date = <Date 2012-05-11.01:20:47.729>
closer = 'ned.deily'
components = ['Library (Lib)']
creation = <Date 2012-04-24.17:58:51.591>
creator = 'grobian'
dependencies = []
files = ['25382', '25383', '25384', '25434', '25435', '25436']
hgrepos = []
issue_num = 14662
keywords = ['patch']
message_count = 22.0
messages = ['159178', '159184', '159185', '159192', '159405', '159407', '159417', '159419', '159422', '159423', '159479', '159480', '159481', '159737', '159795', '159796', '159798', '159803', '159947', '159970', '160395', '160396']
nosy_count = 6.0
nosy_names = ['pitrou', 'ned.deily', 'eric.araujo', 'grobian', 'python-dev', 'hynek']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14662'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[grobian]

With current working dir an NFS-mounted ZFS share, and /var/tmp (OSX default) HFS+:

% echo "test" > /var/tmp/testfile
% python
Python 2.7.3 (default, Apr 24 2012, 19:33:45) 
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.move("/var/tmp/testfile", "./testfile");
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 299, in move
    copy2(src, real_dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 129, in copy2
    copystat(src, dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 45] Operation not supported: './testfile'
>>> 
% ls /var/tmp/testfile ./testfile
./testfile  /var/tmp/testfile

The problem likely is that the flags stored on the HFS+ volume cannot be applied to the NFS-mounted ZFS volume. This likely also occurs when doing a bit more regular things, like e.g. moving/copying to a mounted USB disk (with FAT32 filesystem).

I believe this is a "regression" introduced by http://bugs.python.org/issue8746. Python-2.7.2 works fine.

While preserving flags is nice, it is questionable whether failure to do so in this case is worth dying for. In particular, leaving behind both the original as well as the copy is a bit messy.

[hynek]

I guess a “best effort” approach would be best here.

I presume Python 3.2+ have the same behavior?

[grobian]

I presume Python 3.2+ have the same behavior?

I cannot compile that or get it working normally, so I can't tell for sure. Judging from the code, I'd say yes.

[hynek]

Now that’s odd. I just looked into the code at http://hg.python.org/cpython/file/2.7/Lib/shutil.py#l103 and there is a guard against EOPNOTSUPP:

try:
os.chflags(dst, st.st_flags)
except OSError, why:
if (not hasattr(errno, 'EOPNOTSUPP') or
why.errno != errno.EOPNOTSUPP):
raise

Does your /Library/Gentoo/usr/lib/python2.7/shutil.py look the same? Would you mind adding a print why.errno just before the raise?

I have tried move'ing files to NTFS and FAT32 and it works just fine here.

[grobian]

% echo "test" > /var/tmp/testfile
% python
Python 2.7.3 (default, Apr 26 2012, 19:06:37) 
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.move("/var/tmp/testfile", "./testfile");
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 299, in move
    copy2(src, real_dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 129, in copy2
    copystat(src, dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 45] Operation not supported: './testfile'
>>> 
% vi /Library/Gentoo/usr/lib/python2.7/shutil.py
% python
Python 2.7.3 (default, Apr 26 2012, 19:06:37) 
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.move("/var/tmp/testfile", "./testfile");
45
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 300, in move
    copy2(src, real_dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 130, in copy2
    copystat(src, dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 45] Operation not supported: './testfile'
>>> 
% grep 45 /usr/include/sys/errno.h
#define ENOTSUP         45              /* Operation not supported */

I tried with a FAT16 formatted USB-disk, but there it doesn't fail, so I did some further digging. MS-DOS FS (under OSX) just seems to support setting flags (I tried with stat.UF_HIDDEN, Finder no longer displays the file).

NFS, however, does NOT support any chflags call.

% python
Python 2.7.3 (default, Apr 26 2012, 19:06:37) 
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import errno
>>> print hasattr(errno, 'EOPNOTSUPP')
True
>>> print errno.EOPNOTSUPP
102
>>> 

102 obviously != 45

% grep 102 /usr/include/sys/errno.h
#define EOPNOTSUPP      102             /* Operation not supported on socket */

I believe Python got it mixed up here, we're looking for ENOTSUP, but that one doesn't exist, at least not here.

>>> print hasattr(errno, 'ENOTSUP')
False

[grobian]

it seems errnomodule.c has no idea of ENOTSUP, and that's not the only missing one.

OSX 10.7:
$ grep "^#define\sE" /usr/include/sys/errno.h | awk '{print $2}' | while read line ; do grep -q ${line} Modules/errnomodule.c || echo "missing: $line" ; done
missing: ENOTSUP
missing: EBADRPC
missing: ERPCMISMATCH
missing: EPROGUNAVAIL
missing: EPROGMISMATCH
missing: EPROCUNAVAIL
missing: EFTYPE
missing: EAUTH
missing: ENEEDAUTH
missing: EPWROFF
missing: EDEVERR
missing: EBADEXEC
missing: EBADARCH
missing: ESHLIBVERS
missing: EBADMACHO
missing: ECANCELED
missing: ENOATTR
missing: ENOPOLICY
missing: ENOTRECOVERABLE
missing: EOWNERDEAD
missing: ELAST

Solaris 10:
$ grep "^#define\sE" /usr/include/sys/errno.h | awk '{print $2}' | while read line ; do grep -q ${line} Modules/errnomodule.c || echo "missing: $line" ; done
missing: ECANCELED
missing: ENOTSUP
missing: EOWNERDEAD
missing: ENOTRECOVERABLE
missing: ELOCKUNMAPPED
missing: ENOTACTIVE

[ned-deily]

Thanks for the analysis. Yes, it looks like there's a difference between OS X and current FreeBSDs, for example. chflags(2) on the latter is documented as returning EOPNOTSUPP and on the former ENOTSUP. shutil should check for both. A quick search of the source tree did not find any other users of chflags in the standard library. As far as adding other missing errnos, that could be handled as a separate issue as it more of an enhancement. Anyone interested in contributing a patch for either or both?

https://developer.apple.com/library/mac/#documentation/darwin/reference/manpages/man2/chflags.2.html

http://www.freebsd.org/cgi/man.cgi?query=chflags&apropos=0&sektion=2&manpath=FreeBSD+9.0-RELEASE&arch=default&format=html

[hynek]

I had this text ready before ned chimed in, I’ll post it anyway because it was a lot of work ;):

You're right, 2.7’s errnos are incomplete compared to 3.2. Antoine added ENOTSUP in c370866f30a7 and it runs as “Solaris-specific”.

So it’s currently in 3.2 and later. Shouldn’t hurt to back port it?

EOPNOTSUP is obviously wrong in your case and it doesn’t really sound right at all by the description. However, maybe on some other architecture (FreeBSD?) it’s the way to go? The commit (2e0d58adadbe) states it’s because of ZFS on OS X.

As the code is unchanged in 3.2+, this bug also applies to them.

Suggestion:

For 3.2+3.3: I’d extend the catch to also catch ENOTSUP
For 2.7: I’d also backport the err code.

NB I’m fine if Fabian wants to do it himself, it’s his issue.

[grobian]

I don't want to go through the paperwork nonsense just for a trivial patch, hence I didn't supply one, but instead provided all the information for you guys to make the correct fix.

[merwok]

Trivial patches don’t require paperwork; non-trivial patches require a simple contributor agreement (print, sign, scan, email). We don’t like that either but it is required. If you have any suggestion to make the process simpler, please share them on python-dev.

[hynek]

This is a fix for 2.7.

As Benjamin said in http://bugs.python.org/issue14682#msg159477 it’s okay to back port ENOTSUP, I did it as part of the patch here. I wasn’t sure whether we should document it?

I’m porting the patch to tip right now, reviews/opinions welcome.

[hynek]

This one is against tip.

[hynek]

And finally against 3.2