TarFile expose copyfileobj bufsize to improve throughput

[fried]

BPO	27199
Nosy	@gustaebel, @asvetlov, @ambv, @fried
Files	buftest.py: test file to generate two random tar files and test extraction time improvements copybufsize.patch: patch to expose the copy buffer size

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ambv'
closed_at = <Date 2016-09-10.02:51:11.259>
created_at = <Date 2016-06-03.18:55:37.748>
labels = ['library', 'performance']
title = 'TarFile expose copyfileobj bufsize to improve throughput'
updated_at = <Date 2016-09-10.02:51:11.258>
user = 'https://github.com/fried'

bugs.python.org fields:

activity = <Date 2016-09-10.02:51:11.258>
actor = 'lukasz.langa'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2016-09-10.02:51:11.259>
closer = 'lukasz.langa'
components = ['Library (Lib)']
creation = <Date 2016-06-03.18:55:37.748>
creator = 'fried'
dependencies = []
files = ['43158', '43159']
hgrepos = []
issue_num = 27199
keywords = ['patch']
message_count = 4.0
messages = ['267134', '268234', '275546', '275547']
nosy_count = 5.0
nosy_names = ['lars.gustaebel', 'asvetlov', 'lukasz.langa', 'python-dev', 'fried']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue27199'
versions = ['Python 3.6']

[fried]

The default of 16k while good for memory usage it is not well suited for all cases. if we increased this to 4MB we saw a pretty large improvement to tar file creation and extraction on linux servers.

For a 1gb tar file containing 1024 random files each of 10MB in size.
Time Delta for TarFile: 146.3240258693695
Time Delta for FastTarFile 4MB copybufsize: 102.76440262794495
Time Diff: 43.55962324142456 0.2976928975444698

[ambv]

New feature -> 3.6.

[python-dev]

New changeset 0bac85e355b5 by Łukasz Langa in branch 'default':
Issue bpo-27199: TarFile expose copyfileobj bufsize to improve throughput
https://hg.python.org/cpython/rev/0bac85e355b5

[ambv]

Thanks for the patch!