Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) #77852
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
assignee = None closed_at = <Date 2018-06-12.21:48:06.592> created_at = <Date 2018-05-28.16:17:23.634> labels = ['3.8', 'library', 'performance'] title = 'Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)' updated_at = <Date 2021-07-31.19:15:48.402> user = 'https://github.com/giampaolo'
activity = <Date 2021-07-31.19:15:48.402> actor = 'eric.araujo' assignee = 'none' closed = True closed_date = <Date 2018-06-12.21:48:06.592> closer = 'giampaolo.rodola' components = ['Library (Lib)'] creation = <Date 2018-05-28.16:17:23.634> creator = 'giampaolo.rodola' dependencies =  files = ['47621'] hgrepos =  issue_num = 33671 keywords = ['patch', 'needs review'] message_count = 18.0 messages = ['317878', '317880', '317905', '317906', '317932', '317989', '319401', '319405', '319484', '319486', '319980', '320247', '320456', '336492', '336502', '336514', '336516', '398660'] nosy_count = 21.0 nosy_names = ['facundobatista', 'ncoghlan', 'pitrou', 'scoder', 'vstinner', 'giampaolo.rodola', 'gps', 'StyXman', 'tarek', 'eric.araujo', 'r.david.murray', 'petr.viktorin', 'asvetlov', 'methane', 'SilentGhost', 'neologix', 'socketpair', 'python-dev', 'martin.panter', 'josh.r', 'Anthony Sottile'] pr_nums = ['7160', '7681', '12016', '27516'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'performance' url = 'https://bugs.python.org/issue33671' versions = ['Python 3.8']
The text was updated successfully, but these errors were encountered:
Patch in attachment uses platform specific zero-copy syscalls on Linux and Solaris (os.sendfile(2)), Windows (CopyFileW) and OSX (fcopyfile(2)) speeding up shutil.copyfile() and other functions using it (copy(), copy2(), copytree(), move()).
Average speedup for a 512MB file copy is +24% on Linux, +50% on OSX and +48% on Windows by copying file on the same partition (SSD disk was used).
Follows some benchmarks.
Create 128K, 8M, 512M file:
$ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(128 * 1024))" $ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(8 * 1024 * 1024))" $ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(512 * 1024 * 1024))"
$ time ./python -m timeit -s 'import shutil; p1 = "f1"; p2 = "f2"' 'shutil.copyfile(p1, p2)'
128K copy (+13%):
with patch: 1000 loops, best of 5: 198 usec per loop real 0m1.464s user 0m0.281s sys 0m0.958s
8MB copy (+24%):
with patch: 50 loops, best of 5: 7.78 msec per loop real 0m2.447s user 0m0.086s sys 0m1.682s
512MB copy (+26%):
with patch: 1 loop, best of 5: 646 msec per loop real 0m5.475s user 0m0.037s sys 0m2.959s
128K copy (+8.5%):
with patch: 500 loops, best of 5: 464 usec per loop real 0m2.798s user 0m0.379s sys 0m2.031s
8MB copy (+67%):
with patch: 20 loops, best of 5: 10.8 msec per loop real 0m1.860s user 0m0.079s sys 0m0.719s
512MB copy (+50%):
128K copy (+69%):
8M copy (+64%):
512M copy (+48%):
That possibly should be used under Linux in order to really acheive zero-copying. Just like modern cp command.
Yes, I tried changing benchmarks order and zero-copy variants are always faster. As for instantaneous CoW copy, it is debatable. E.g. "cp" command does not do it by default:
For future reference, as per #7160 discussion, we decided not to use CopyFileEx on Windows and instead increase read() buffer size from 16KB to 1MB (Windows only) resulting in a 40.8% speedup (instead of 48%). Also copyfileobj() has been optimized on all platforms by using readinto()/memoryview()/bytearray().
128KB copy (+27%)
8MB copy (+45.6%)
512MB copy (+40.8%)