subprocess should open stdin in mode w+b on windows

[JasonGross]

BPO	18016
Nosy	@tjguk, @zware, @zooba, @csabella

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-03-19.22:59:02.567>
created_at = <Date 2013-05-19.20:29:13.836>
labels = ['type-bug', 'library']
title = 'subprocess should open stdin in mode w+b on windows'
updated_at = <Date 2019-03-19.22:59:02.564>
user = 'https://bugs.python.org/JasonGross'

bugs.python.org fields:

activity = <Date 2019-03-19.22:59:02.564>
actor = 'cheryl.sabella'
assignee = 'none'
closed = True
closed_date = <Date 2019-03-19.22:59:02.567>
closer = 'cheryl.sabella'
components = ['Library (Lib)']
creation = <Date 2013-05-19.20:29:13.836>
creator = 'Jason.Gross'
dependencies = []
files = []
hgrepos = []
issue_num = 18016
keywords = []
message_count = 4.0
messages = ['189624', '222892', '223117', '338414']
nosy_count = 6.0
nosy_names = ['tim.golden', 'Jason.Gross', 'BreamoreBoy', 'zach.ware', 'steve.dower', 'cheryl.sabella']
pr_nums = []
priority = 'normal'
resolution = 'works for me'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue18016'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[JasonGross]

Files opened on Windows in mode 'wb' have a limit on the number of characters that can be written to them (at once?); attempting to write too many bytes gives the confusing error “IOError: [Errno 22] Invalid argument”. See http://support.microsoft.com/default.aspx?scid=kb;en-us;899149 and http://stackoverflow.com/questions/4226941/python-ioerror-errno-22-invalid-argument-when-using-cpickle-to-write-large. I have gotten this error when using subprocess.communicate(input=<large string>). Please fix the subprocess library (and any other library that uses mode 'wb') to use mode 'w+b', at least on Windows.

[BreamoreBoy]

@jason I'm very sorry about the delay in getting back to you.

Can our Windows gurus shed any light on this one?

[zooba]

With 2.7 and 3.4 (same for 32- and 64-bit):

>>> f = open('test.bin', 'wb')
>>> f.write(b' ' * (1024*1024*100))
104857600
>>> f.close()
>>> import os
>>> os.stat('test.bin').st_size
104857600

The linked KB only applies to VS 2003 and VS 2005 (VC7 and VC8), so I'm not entirely surprised that this doesn't repro with VC9 or VC10.

For the subprocess case (on 3.4):

>>> import sys, subprocess
>>> p=subprocess.Popen([sys.executable, '-c', 'print(len(input()))'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> p.communicate(b' ' * (1024*1024*100))
(b'104857600\r\n', b'')

I'm inclined to close this as no repro unless we get an actual repro.

[csabella]

Since Steve was inclined to close this in 2014 and there haven't been any additional comments since then, I am going to close this now.