Not a full replacement for subprocess of Python 2.7

[tlandschoff-scale]

Hey there,

thanks for making subprocess32 available, it really helped us fighting random lock-ups in our application. I would not have dared to backport subprocess from Python 3 and resorted to workarounds.

After issues with mixups between subprocess32 and subprocess (some libraries still use subprocess and know nothing about subprocess32) we decided to heed this advice:

Or if you fully control your POSIX Python 2.7 installation, this can serve as a replacement for its subprocess module. Users will thank you by not filing concurrency bugs.

So far this looked good. Out of curiousity I ran the Python 2.7 stdlib test suite against subprocess32 and was surprised that it failed.

I offer this Dockerfile to reproduce this:

$ cat Dockerfile 
FROM debian:9

RUN apt-get update
RUN apt-get install -y python2.7 libpython2.7-testsuite python-pip

# Install subprocess32, replace subprocess
RUN pip install subprocess32
RUN cp /usr/local/lib/python2.7/dist-packages/subprocess32.py /usr/lib/python2.7/subprocess.py
RUN cp /usr/local/lib/python2.7/dist-packages/_posixsubprocess32.so /usr/lib/python2.7/lib-dynload/_posixsubprocess32.so

# Run regrtest
RUN python -m test.regrtest --verbose test_subprocess

This results into these failures:

$ sudo docker build .
Sending build context to Docker daemon  2.048kB
Step 1/7 : FROM debian:9
 ---> de8b49d4b0b3
Step 2/7 : RUN apt-get update
 ---> Using cache
 ---> 21f3d4930519
[...]
Step 7/7 : RUN python -m test.regrtest --verbose test_subprocess
 ---> Running in 58064776ae73
== CPython 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516]
==   Linux-4.15.0-47-generic-x86_64-with-debian-9.6 little-endian
==   /tmp/test_python_7
Testing with flags: sys.flags(debug=0, py3k_warning=0, division_warning=0, division_new=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, tabcheck=0, verbose=0, unicode=0, bytes_warning=0, hash_randomization=0)
[1/1] test_subprocess
test_call_kwargs (test.test_subprocess.ProcessTestCase) ... ok
[...]
test test_subprocess failed -- multiple errors occurred
skipped 'mswindows only'

======================================================================
ERROR: test_communicate_epipe_only_stdin (test.test_subprocess.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 714, in test_communicate_epipe_only_stdin
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 712, in communicate
    self.stdin.write(input)
IOError: [Errno 32] Broken pipe

======================================================================
ERROR: test_exceptions (test.test_subprocess.POSIXProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 803, in test_exceptions
    self.assertIn("os.chdir", c.exception.child_traceback)
AttributeError: 'exceptions.OSError' object has no attribute 'child_traceback'

======================================================================
ERROR: test_preexec_errpipe_does_not_double_close_pipes (test.test_subprocess.POSIXProcessTestCase)
Issue16140: Don't double close pipes on preexec error.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 868, in test_preexec_errpipe_does_not_double_close_pipes
    stderr=subprocess.PIPE, preexec_fn=raise_it)
  File "/usr/lib/python2.7/test/test_subprocess.py", line 827, in __init__
    subprocess.Popen.__init__(self, *args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 614, in __init__
    restore_signals, start_new_session)
TypeError: _execute_child() takes exactly 18 arguments (20 given)

======================================================================
ERROR: test_communicate_epipe (test.test_subprocess.ProcessTestCaseNoPoll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 706, in test_communicate_epipe
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 724, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python2.7/subprocess.py", line 1538, in _communicate
    orig_timeout)
  File "/usr/lib/python2.7/subprocess.py", line 1684, in _communicate_with_select
    bytes_written = os.write(self.stdin.fileno(), chunk)
OSError: [Errno 32] Broken pipe

======================================================================
ERROR: test_communicate_epipe_only_stdin (test.test_subprocess.ProcessTestCaseNoPoll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 714, in test_communicate_epipe_only_stdin
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 712, in communicate
    self.stdin.write(input)
IOError: [Errno 32] Broken pipe

----------------------------------------------------------------------
Ran 146 tests in 16.424s

Some tests are bound to fail given that they access the internals of the Python 2.7 implementation. But these look relevant:

======================================================================
ERROR: test_communicate_epipe_only_stdin (test.test_subprocess.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 714, in test_communicate_epipe_only_stdin
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 712, in communicate
    self.stdin.write(input)
IOError: [Errno 32] Broken pipe

======================================================================
ERROR: test_communicate_epipe (test.test_subprocess.ProcessTestCaseNoPoll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 706, in test_communicate_epipe
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 724, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python2.7/subprocess.py", line 1538, in _communicate
    orig_timeout)
  File "/usr/lib/python2.7/subprocess.py", line 1684, in _communicate_with_select
    bytes_written = os.write(self.stdin.fileno(), chunk)
OSError: [Errno 32] Broken pipe

======================================================================
ERROR: test_communicate_epipe_only_stdin (test.test_subprocess.ProcessTestCaseNoPoll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/test/test_subprocess.py", line 714, in test_communicate_epipe_only_stdin
    p.communicate("x" * 2**20)
  File "/usr/lib/python2.7/subprocess.py", line 712, in communicate
    self.stdin.write(input)
IOError: [Errno 32] Broken pipe

I intent to find out the cause and fix this, but for reference I'd like to have this issue public in the mean time.

BTW: There is no mention of epipe in the test_subprocess32.py tests:

~/workspace/python-subprocess32$ grep -i epipe test_subprocess32.py 

Greetings, Torsten

[gpshead]

At first glance it appears that the fix for https://bugs.python.org/issue10963 might not have made it into subprocess32? (at the very least, the test isn't there)

python/cpython@4f61b02

seems to be the relevant upstream commit.