Configurable blocksize in HTTP(S)Connection

[nirs]

BPO	31945
Nosy	@brettcannon, @nirs, @vstinner, @serhiy-storchaka, @1st1
PRs	bpo-31945: Configurable blocksize in HTTP(S)Connection #4279

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-11-06.21:21:57.942>
created_at = <Date 2017-11-04.22:32:30.582>
labels = ['3.7', 'type-feature', 'library']
title = 'Configurable blocksize in HTTP(S)Connection'
updated_at = <Date 2017-11-06.21:21:57.941>
user = 'https://github.com/nirs'

bugs.python.org fields:

activity = <Date 2017-11-06.21:21:57.941>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-11-06.21:21:57.942>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2017-11-04.22:32:30.582>
creator = 'nirs'
dependencies = []
files = []
hgrepos = []
issue_num = 31945
keywords = ['patch']
message_count = 5.0
messages = ['305571', '305624', '305678', '305679', '305680']
nosy_count = 5.0
nosy_names = ['brett.cannon', 'nirs', 'vstinner', 'serhiy.storchaka', 'yselivanov']
pr_nums = ['4279']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31945'
versions = ['Python 3.7']

[nirs]

blocksize is hardcoded to 8192 in send() and _read_readable(), preventing
efficient upload when using file-like body.

Users of the module that are not interested in chunked encoding can rewrite
the copy loop using HTTPConnection.send():

   conn = HTTPSConnection(...)
   conn.putrequest(...)
   conn.putheader(...)
   conn.endheaders()

   while True:
       chunk = file.read(512*1024)
       if not chunk:
          break
       conn.send(chunk)

But fixing send() to use a configurable blocksize seems more useful.

Also, users of requests do not have access the underlying connection, so
they cannot use preferred buffer size.

When reading from /dev/zero and uploading to server that drop the received
data, larger buffer size gives 3X more throughput *and* 1/3 of cpu time.
With real storage and network, the effect will probably be much smaller.

[nirs]

When using highlevel request() api, users can control the block size by
wrapping the file object with an iterator:

    class FileIter:

        def __init__(self, file, blocksize):
            self.file = file
            self.blocksize = blocksize

        def __iter__(self):
            while True:
                datablock = self.file.read(self.blocksize)
                if not datablock:
                    break
                yield datablock

Adding configurable block size will avoid this workaround.

[vstinner]

New changeset ad455cd by Victor Stinner (Nir Soffer) in branch 'master':
bpo-31945: Configurable blocksize in HTTP(S)Connection (bpo-4279)
ad455cd

[vstinner]

Thank you Nir Soffer for this nice enhancement. Sadly, since it's a new feature, it cannot be backport to Python 3.6 nor 2.7, since it's a new feature.

[vstinner]

The commit message contains much more information than the NEWS and What's New entries. Maybe the What's New entry can be completed, I don't know. In the meanwhile, I close the issue :-) Nir: Feel free to create a new PR if you want to complete the What's New entry ;-)