Tarfile superfluous truncate calls slows extraction.

[fried]

BPO	27194
Nosy	@gustaebel, @asvetlov, @ambv, @fried
Files	truncate.patch: patch to move truncate for only sparse tar entries. test.py: test file to generate random tar for benchmark

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ambv'
closed_at = <Date 2016-06-12.00:02:19.913>
created_at = <Date 2016-06-03.04:56:33.633>
labels = ['library', 'performance']
title = 'Tarfile superfluous truncate calls slows extraction.'
updated_at = <Date 2016-06-12.00:02:19.911>
user = 'https://github.com/fried'

bugs.python.org fields:

activity = <Date 2016-06-12.00:02:19.911>
actor = 'lukasz.langa'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2016-06-12.00:02:19.913>
closer = 'lukasz.langa'
components = ['Library (Lib)']
creation = <Date 2016-06-03.04:56:33.633>
creator = 'fried'
dependencies = []
files = ['43138', '43139']
hgrepos = []
issue_num = 27194
keywords = ['patch']
message_count = 4.0
messages = ['267035', '267037', '268297', '268298']
nosy_count = 5.0
nosy_names = ['lars.gustaebel', 'asvetlov', 'lukasz.langa', 'python-dev', 'fried']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue27194'
versions = ['Python 3.5']

[fried]

With large tar file extracts I noticed that tarfile was slower than it should be. Seems in linux that for large files (10MB) truncate is not always a free operation even when it should be a no-op. ex: File is already 10mb seek to end and truncate.

I created a script to test the validity of this patch. It generates two random tar archives containing 1024 files of 10mb each. The files are randomized so disk caching should not interfere.

So to extract those 1g tar files the following was observed
Time Delta for TarFile: 148.23699307441711
Time Delta for FastTarFile: 107.71058106422424
Time Diff: 40.52641201019287 0.27338932859929255

[fried]

I ran this on Linux ext4. I didn't see much improvement on OSX with SSD.

[python-dev]

New changeset b63474aa8a5f by Łukasz Langa in branch '3.5':
Issue bpo-27194: superfluous truncate calls in tarfile.py slow down extraction
https://hg.python.org/cpython/rev/b63474aa8a5f

New changeset a4f918de25e5 by Łukasz Langa in branch 'default':
Merge 3.5, issue bpo-27194
https://hg.python.org/cpython/rev/a4f918de25e5

[ambv]

Thanks for the patch, Jason. This is now merged and will be available in 3.5.2 and 3.6.