PIPE_BUF is invalid on AIX

[sable]

BPO	9862
Nosy	@amauryfa, @bitdancer
Files	patch_broken_pipe_buf.diff patch_broken_pipe_buf_updated.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/bitdancer'
closed_at = <Date 2010-10-16.00:49:07.914>
created_at = <Date 2010-09-15.16:30:33.584>
labels = ['type-bug', 'library']
title = 'PIPE_BUF is invalid on AIX'
updated_at = <Date 2010-10-18.09:24:10.790>
user = 'https://bugs.python.org/sable'

bugs.python.org fields:

activity = <Date 2010-10-18.09:24:10.790>
actor = 'sable'
assignee = 'r.david.murray'
closed = True
closed_date = <Date 2010-10-16.00:49:07.914>
closer = 'r.david.murray'
components = ['Library (Lib)']
creation = <Date 2010-09-15.16:30:33.584>
creator = 'sable'
dependencies = []
files = ['18937', '18944']
hgrepos = []
issue_num = 9862
keywords = ['patch']
message_count = 7.0
messages = ['116467', '116932', '116952', '117031', '117040', '118849', '118998']
nosy_count = 3.0
nosy_names = ['amaury.forgeotdarc', 'sable', 'r.david.murray']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9862'
versions = ['Python 2.7', 'Python 3.2']

[sable]

On AIX, the test test_communicate_pipe_buf in test_subprocess will hang forever (in py3k and py27):

test_communicate_pipe_buf (main.ProcessTestCase) ...

File "Lib/test/test_subprocess.py", line 386, in test_communicate_pipe_buf
(stdout, stderr) = p.communicate(string_to_write)
File "/san_cis/home/cis/data/bamboo-home-agent-runtime/xml-data/build-dir/RTAIX30-SUP/Python-2.7-svn/Lib/subprocess.py", line 740, in communicate
return self._communicate(input)
File "/san_cis/home/cis/data/bamboo-home-agent-runtime/xml-data/build-dir/RTAIX30-SUP/Python-2.7-svn/Lib/subprocess.py", line 1257, in _communicate
stdout, stderr = self._communicate_with_poll(input)
File "/san_cis/home/cis/data/bamboo-home-agent-runtime/xml-data/build-dir/RTAIX30-SUP/Python-2.7-svn/Lib/subprocess.py", line 1320, in _communicate_with_poll
input_offset += os.write(fd, chunk)
KeyboardInterrupt

The comment in this test indicates:
# communicate() with writes larger than pipe_buf
# This test will probably deadlock rather than fail, if
# communicate() does not work properly.

So I guess it means communicate() does not work properly on AIX.

[sable]

The problem does not happen with Python 2.6.
The difference is that:

• in Python 2.6, the subprocess module would try to write at most 512 bytes
  cf L1221 in subprocess.py
  chunk = input[input_offset : input_offset + 512]

• in Python 2.7 and py3k, the subprocess module will try to write at most _PIPE_BUF bytes
  cf L1319 in subprocess.py
  chunk = input[input_offset : input_offset + _PIPE_BUF]

When forcing PIPE_BUF to 512 in selectmodule.c, the test will pass.
PIPE_BUF seems to be broken on AIX.

[sable]

PIPE_BUF in unistd.h is defined to 32768.

The test works with PIPE_BUF changed up to 6144 but won't work with 7168.

We could probably use 4096 as a safe value for faster result if needed.
I just do not define the value and leave the default of 512 be used.

Here is my patch which has been verified to work OK and should be applied to py3k and py27.

[amauryfa]

Removing select.PIPE_BUF does not seem a good idea to me because this breaks compatibility. I suggest to simply set it to 512 on AIX.

(An ideal solution would be to really determine the actual buffer size in ./configure; this is probably overkill)

[sable]

OK for me.
Here is a new patch that defines PIPE_BUF to 512 instead of removing select.PIPE_BUF.

[bitdancer]

Committed to py3k in r85554, 2.7 in r85556. Sébastien, from what you say it sounds like this does not apply to 3.1, so I blocked it there. If this is incorrect let me know and I'll backport it.

[sable]

Thanks R. David.

I checked in 3.1 and PIPE_BUF is not defined in the select module, so the default value of 512 is used in subprocess.
So no correction is needed for that version.