shutil.copystat utime lookup fails on certain Android file systems

[JerryA]

BPO	28141
Nosy	@iritkatriel
Files	copyoops2.py: reproduce the utime failure of copystat

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-12-29.22:34:48.263>
created_at = <Date 2016-09-14.06:04:10.197>
labels = ['type-bug', 'library']
title = 'shutil.copystat utime lookup fails on certain Android file systems'
updated_at = <Date 2021-12-29.22:34:48.262>
user = 'https://bugs.python.org/JerryA'

bugs.python.org fields:

activity = <Date 2021-12-29.22:34:48.262>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-12-29.22:34:48.263>
closer = 'iritkatriel'
components = ['Library (Lib)']
creation = <Date 2016-09-14.06:04:10.197>
creator = 'Jerry A'
dependencies = []
files = ['44650']
hgrepos = []
issue_num = 28141
keywords = []
message_count = 3.0
messages = ['276385', '407931', '409330']
nosy_count = 2.0
nosy_names = ['Jerry A', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28141'
versions = ['Python 2.7', 'Python 3.5']

[JerryA]

shutil.copystat fails on certain Android filesystems when trying lookup("utime") (line 194 of https://hg.python.org/cpython/file/3.5/Lib/shutil.py)

Using Python 2.7 and Python 3.5 on Android 6.0.1., when I am on the "typical" user file system, and I run a script that uses shutil.copy2, that copy2 will fail when it calls shutil.copystat and that copystat fails performing utime on the destination file.

I am attaching a script that can be used to reproduce this. I run the script using python 3.5 within termux.

Below is a transcript my running of the attacked script.

$ pwd
/data/data/com.termux/files/home

$ ls --l
Dropbox  storage  tmp  venv  workspace

$ ls -la
total 52
drwx------ 10 u0_a73 u0_a73    4096 Sep 13 16:07 .
drwx------  4 u0_a73 u0_a73    4096 Jul 10 02:18 ..
-rw-------  1 u0_a73 u0_a73    7011 Sep 13 20:34 .bash_history
drwx------  3 u0_a73 u0_a73    4096 Jul 10 02:30 .cache
drwxrwx--- 12 u0_a73 everybody 4096 Sep 13 15:35 .emacs.d
drwx------  2 u0_a73 u0_a73    4096 Jul 10 15:58 .gnupg
-rw-------  1 u0_a73 u0_a73     145 Sep 13 16:07 .python_history
drwx------  2 u0_a73 everybody 4096 Sep 13 12:45 .ssh
drwx------  3 u0_a73 u0_a73    4096 Sep 13 14:10 Dropbox
drwx------  2 u0_a73 u0_a73    4096 Jul 10 03:42 storage
drwx------  2 u0_a73 u0_a73    4096 Sep 13 14:24 tmp
drwx------  5 u0_a73 u0_a73    4096 Jul 10 02:33 venv
lrwxrwxrwx  1 u0_a73 u0_a73      30 Jul 10 15:54 workspace -> /storage/emulated/0/workspace/


$ cat /proc/mounts
rootfs / rootfs ro,seclabel,relatime 0 0
tmpfs /dev tmpfs rw,seclabel,nosuid,relatime,mode=755 0 0
devpts /dev/pts devpts rw,seclabel,relatime,mode=600 0 0
none /dev/memcg cgroup rw,relatime,memory 0 0
none /dev/cpuctl cgroup rw,relatime,cpu 0 0
adb /dev/usb-ffs/adb functionfs rw,relatime 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,seclabel,relatime 0 0
selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
none /sys/fs/cgroup tmpfs rw,seclabel,relatime,mode=750,gid=1000 0 0
none /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
/sys/kernel/debug /sys/kernel/debug debugfs rw,seclabel,relatime 0 0
none /acct cgroup rw,relatime,cpuacct 0 0
tmpfs /mnt tmpfs rw,seclabel,relatime,mode=755,gid=1000 0 0
/dev/fuse /mnt/runtime/default/emulated fuse rw,nosuid,nodev,noexec,noatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0
/dev/fuse /mnt/runtime/read/emulated fuse rw,nosuid,nodev,noexec,noatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0
/dev/fuse /mnt/runtime/write/emulated fuse rw,nosuid,nodev,noexec,noatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0
/dev/block/platform/dw_mmc.0/by-name/system /system ext4 ro,seclabel,relatime 0 0
/dev/block/platform/dw_mmc.0/by-name/cache /cache ext4 rw,seclabel,nosuid,nodev,noatime,journal_checksum,journal_async_commit,noauto_da_alloc,errors=panic,data=ordered 0 0
/dev/block/platform/dw_mmc.0/by-name/userdata /data ext4 rw,seclabel,nosuid,nodev,noatime,journal_checksum,journal_async_commit,noauto_da_alloc,errors=panic,data=ordered 0 0
/dev/block/platform/dw_mmc.0/by-name/efs /factory ext4 ro,seclabel,relatime,data=ordered 0 0
tmpfs /storage tmpfs rw,seclabel,relatime,mode=755,gid=1000 0 0
/dev/fuse /storage/emulated fuse rw,nosuid,nodev,noexec,noatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0
tmpfs /storage/self tmpfs rw,seclabel,relatime,mode=755,gid=1000 0 0
$

$ cd workspace
$ cd foo
$ cd copystatbug/

$ echo "this is Mr. Lorem Ipsum" > foo.bar
$ cat foo.bar
this is Mr. Lorem Ipsum

$ python copyoops2.py
copyfoo
A problem lookingup utime(<built-in function utime>) of foo.foo; [Errno 1] Operation not permitted
A problem copying the file metadata of foo.bar to foo.foo; [Errno 1] Operation not permitted: 'foo.foo'
copybar
A problem lookingup utime(<built-in function utime>) of bar.bar; [Errno 1] Operation not permitted
Traceback (most recent call last):
  File "copyoops2.py", line 155, in <module>
    copybar("foo.bar", "bar.bar")
  File "copyoops2.py", line 151, in copybar
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "copyoops2.py", line 81, in copystat
    lookup("chmod")(dst, mode, follow_symlinks=follow)
PermissionError: [Errno 1] Operation not permitted: 'bar.bar'
$ ls -la
total 36
drwxrwx--- 2 root everybody 4096 Sep 13 22:40 .
drwxrwx--- 4 root everybody 4096 Sep 13 20:42 ..
-rw-rw---- 1 root everybody   24 Sep 13 22:40 bar.bar
-rw-rw---- 1 root everybody 4846 Sep 13 22:38 copyoops.py
-rw-rw---- 1 root everybody 4838 Sep 13 22:39 copyoops2.py
-rw-rw---- 1 root everybody   24 Sep 13 20:57 foo.bar
-rw-rw---- 1 root everybody   24 Sep 13 22:40 foo.foo
$ python --version
Python 3.5.2

[iritkatriel]

Jerry, is this still a problem with current python versions? (2.7/3.5 are no longer being maintained).

[iritkatriel]

Please create a new issue if this is still a problem on a currentl version of python (>= 3.9).