cgi.parse_multipart() requires undocumented CONTENT-LENGTH in Python 3.7

[yan12125]

BPO	34226
Nosy	@freddrake, @orsenthil, @ned-deily, @PierreQuentel, @vadmium, @yan12125, @miss-islington, @tirkarthi, @Roger, @0x3333, @YtvwlD
PRs	bpo-34226: fix cgi.parse_multipart without content_length #8530 [3.9] bpo-34226: fix cgi.parse_multipart without content_length (GH-8530) #20890 [3.8] bpo-34226: fix cgi.parse_multipart without content_length (GH-8530) #20891 [3.7] bpo-34226: fix cgi.parse_multipart without content_length (GH-8530) #20892

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = <Date 2020-06-15.16:04:52.499>
created_at = <Date 2018-07-25.14:55:40.577>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.7', 'library', 'docs']
title = 'cgi.parse_multipart() requires undocumented CONTENT-LENGTH in Python 3.7'
updated_at = <Date 2020-06-15.17:59:28.374>
user = 'https://github.com/yan12125'

bugs.python.org fields:

activity = <Date 2020-06-15.17:59:28.374>
actor = 'orsenthil'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2020-06-15.16:04:52.499>
closer = 'orsenthil'
components = ['Documentation', 'Library (Lib)']
creation = <Date 2018-07-25.14:55:40.577>
creator = 'yan12125'
dependencies = []
files = []
hgrepos = []
issue_num = 34226
keywords = ['patch', '3.7regression']
message_count = 11.0
messages = ['322362', '322541', '353545', '371340', '371350', '371367', '371557', '371563', '371565', '371566', '371583']
nosy_count = 12.0
nosy_names = ['fdrake', 'orsenthil', 'ned.deily', 'docs@python', 'quentel', 'martin.panter', 'yan12125', 'miss-islington', 'xtreak', 'rogerduran', 'Tercio Gaudencio Filho', 'ytvwld']
pr_nums = ['8530', '20890', '20891', '20892']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34226'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

[yan12125]

In Python 3.7, cgi.parse_multipart() requires "CONTENT-LENGTH" in the second parameter pdict. This is not necesary in Python 3.6.

For example, the following code runs with 3.6:

$ python3.6
Python 3.6.6 (default, Jun 27 2018, 13:11:40) 
[GCC 8.1.1 20180531] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cgi, io
>>> data = b'--heyDavid\r\nContent-Disposition: form-data; name="cfield"\r\n\r\njust a string\r\n\r\n--heyDavid--\r\n'
>>> cgi.parse_multipart(io.BytesIO(data), {"boundary": b"heyDavid"})
{'cfield': [b'just a string\r\n']}

While not on 3.7:

$ python3.7
Python 3.7.0 (default, Jul 15 2018, 10:44:58) 
[GCC 8.1.1 20180531] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cgi, io
>>> data = b'--heyDavid\r\nContent-Disposition: form-data; name="cfield"\r\n\r\njust a string\r\n\r\n--heyDavid--\r\n'
>>> cgi.parse_multipart(io.BytesIO(data), {"boundary": b"heyDavid"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/cgi.py", line 220, in parse_multipart
    headers['Content-Length'] = pdict['CONTENT-LENGTH']
KeyError: 'CONTENT-LENGTH'

I looked into the source code of CPython, and found CONTENT-LENGTH in nowhere but cgi.py and test_cgi.py. I guess it has the same meaning as the Content-Length header in HTTP or CONTENT_LENGTH environment variable in CGI? It would be great to document such a backward-incompatible behavior.

Environment: Arch Linux with Python 3.6.6 from [extra] repo and 3.7.0 from [staging] repo.

CC the author and the reviewer of bpo-29979, where relevant codes are introduced.

[Roger]

working on this on europython

[vadmium]

Looks like the change causing this is revision cc3fa20. I would remove the reference to pdict['CONTENT-LENGTH'].

[ned-deily]

I note this is marked as a 3.7regression and still open. Since the cutoff for the final 3.7 bugfix mode release is in a few days, I'm assuming this means that 3.7 users will have to live with this regression. If you feel that is a problem, speak up now.

[YtvwlD]

Since there's the easy workaround of just setting pdict["CONTENT-LENGTH"] I personally don't have a problem with this staying the way it is.

But it would be really nice if this were documented somewhere.

[orsenthil]

I will spend time on this on 12-June or 13-June, 2020.
At the very least, if no changes are required, I will document this for 3.7 and higher versions.

[miss-islington]

New changeset d8cf351 by roger in branch 'master':
bpo-34226: fix cgi.parse_multipart without content_length (GH-8530)
d8cf351

[miss-islington]

New changeset aa83935 by Miss Islington (bot) in branch '3.7':
[3.7] bpo-34226: fix cgi.parse_multipart without content_length (GH-8530) (GH-20892)
aa83935

[miss-islington]

New changeset b87453f by Miss Islington (bot) in branch '3.9':
bpo-34226: fix cgi.parse_multipart without content_length (GH-8530)
b87453f

[miss-islington]

New changeset c72b7f7 by Miss Islington (bot) in branch '3.8':
bpo-34226: fix cgi.parse_multipart without content_length (GH-8530)
c72b7f7

[orsenthil]

This is now fixed in all active versions of python. No documentation changes required for this change.