urllib.request: opener not resetting content-length

[terryjreedy]

BPO	16464
Nosy	@terryjreedy, @orsenthil, @vstinner, @asvetlov
Files	issue16464.diff issue16464_fixed.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/asvetlov'
closed_at = <Date 2012-11-27.21:19:34.090>
created_at = <Date 2012-11-13.01:57:33.230>
labels = ['type-bug']
title = 'urllib.request: opener not resetting content-length'
updated_at = <Date 2014-03-19.16:34:43.692>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2014-03-19.16:34:43.692>
actor = 'vstinner'
assignee = 'asvetlov'
closed = True
closed_date = <Date 2012-11-27.21:19:34.090>
closer = 'asvetlov'
components = []
creation = <Date 2012-11-13.01:57:33.230>
creator = 'terry.reedy'
dependencies = []
files = ['28015', '28074']
hgrepos = []
issue_num = 16464
keywords = ['patch']
message_count = 10.0
messages = ['175485', '175814', '176108', '176109', '176114', '176492', '176493', '184732', '213097', '214109']
nosy_count = 6.0
nosy_names = ['terry.reedy', 'orsenthil', 'vstinner', 'asvetlov', 'python-dev', 'kachayev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16464'
versions = ['Python 3.4']

[terryjreedy]

Code based on python-list post by a do-not-wish-to-register urllib user.

import urllib.request
opener = urllib.request.build_opener()
request = urllib.request.Request("http://example.com/", headers =
        {"Content-Type": "application/x-www-form-urlencoded"})
print(request.data, '\n', request.header_items())

opener.open(request, "1".encode("us-ascii"))
print(request.data, '\n', request.header_items())

opener.open(request, "123456789".encode("us-ascii"))
print(request.data, '\n', request.header_items())
>>> 
None 
 [('Content-type', 'application/x-www-form-urlencoded')]
b'1' 
 [('Content-length', '1'), ('Host', 'example.com'), ('User-agent', 'Python-urllib/3.3'), ('Content-type', 'application/x-www-form-urlencoded')]
b'123456789' 
 [('Content-length', '1'), ('Host', 'example.com'), ('User-agent', 'Python-urllib/3.3'), ('Content-type', 'application/x-www-form-urlencoded')]

The first opener.open adds data and several headers to request, including content-length. The second changes the data but not the content-length. The docs do not say anything about this either way.

[kachayev]

This is special case for more "general" problem. When request is executed with HTTP client and data is not None, it calculates content length and adds special header to request. Then one can change request.data attribute value, but header "Content-length" is not changed in this case.

Patch is attached.

I implemented request.data as property and added method "remove_header" to deal problem. Test cases are also provided.

[asvetlov]

I'm agree with solution, see my comments in review for the patch.

[asvetlov]

Perhaps it's a bit new behavior and should be applied to 3.4 only.

[kachayev]

Fixed patch is attached.
Documentation is updated.

[python-dev]

New changeset 618ea5612e83 by Andrew Svetlov in branch 'default':
Issue bpo-16464: reset Request's Content-Length header on .data change.
http://hg.python.org/cpython/rev/618ea5612e83

[asvetlov]

Pushed.
Thanks, Alexey.

[python-dev]

New changeset b1579eb4e1bc by R David Murray in branch 'default':
bpo-17485: Delete the Content-Length header if the data attribute is deleted.
http://hg.python.org/cpython/rev/b1579eb4e1bc

[python-dev]

New changeset e6d862886e5c by R David Murray in branch 'default':
whatsnew: urllib Request objects are now reusable.
http://hg.python.org/cpython/rev/e6d862886e5c

[vstinner]

changeset: 89857:ad0c75b7bd7d
tag: tip
parent: 89855:9120196b3114
parent: 89856:68335b8afb1f
user: Victor Stinner <victor.stinner@gmail.com>
date: Wed Mar 19 17:34:12 2014 +0100
description:
(Merge 3.4) Skip test_urllib2.test_issue16464() is the ssl module is missing

changeset: 89856:68335b8afb1f
branch: 3.4
parent: 89852:c44258b4b7a4
user: Victor Stinner <victor.stinner@gmail.com>
date: Wed Mar 19 17:31:20 2014 +0100
files: Lib/test/test_urllib2.py
description:
Skip test_urllib2.test_issue16464() is the ssl module is missing