Mix callable alias with pure subprocess.run: Bad file descriptor

[sbliven]

xonfig

+------------------+----------------------+
| xonsh            | 0.12.1               |
| Git SHA          | cd334e44             |
| Commit Date      | Apr 15 13:10:59 2022 |
| Python           | 3.10.2               |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 3.0.22               |
| shell type       | readline             |
| history backend  | json                 |
| pygments         | 2.10.0               |
| on posix         | True                 |
| on linux         | False                |
| on darwin        | True                 |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
| xontrib 1        | coreutils            |
| RC file          | []                   |
+------------------+----------------------+

Expected Behavior

I am trying to create a complex alias with redirected stdin. As a test I created the following wrapper function for grep, since it's a command that accepts input either from stdin or a file:

def _mygrep(args, stdin=None, stdout=None, stderr=None):
    return subprocess.run(["/usr/bin/grep"]+args,  stdin=stdin, stdout=stdout, stderr=stderr).returncode
aliases['mygrep']=_mygrep

I expect this alias to work exactly like grep. For instance:

$ echo hovercraft|mygrep raft
hovercraft

Current Behavior

With an empty rc file the above function works as expected. However, if I first load the coreutils xontrib I see an OSError about 50% of the time. The error prints directly to the console, so redirecting the output doesn't hide the error.

$ echo hovercraft|mygrep raft a>/dev/null
xonsh: To log full traceback to a file set: $XONSH_TRACEBACK_LOGFILE = <filename>
Traceback (most recent call last):
  File "/usr/local/Cellar/xonsh/0.11.0/libexec/lib/python3.10/site-packages/xonsh/procs/__amalgam__.py", line 2328, in run
    r = self.f(self.args, sp_stdin, sp_stdout, sp_stderr, spec, spec.stack)
  File "/usr/local/Cellar/xonsh/0.11.0/libexec/lib/python3.10/site-packages/xonsh/procs/__amalgam__.py", line 2117, in proxy_four
    return f(args, stdin, stdout, stderr)
  File "<stdin>", line 2, in _mygrep
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1804, in _execute_child
    os.close(errpipe_read)
OSError: [Errno 9] Bad file descriptor

Surprisingly, this behavior happens even for commands like grep that are not replaced by coreutils.

Traceback (if applicable)

traceback

Steps to Reproduce

xonsh --no-rc
xontrib load coreutils
$XONSH_SHOW_TRACEBACK = True
import subprocess
def _mygrep(args, stdin=None, stdout=None, stderr=None):
    return subprocess.run(["/usr/bin/grep"]+args,  stdin=stdin, stdout=stdout, stderr=stderr).returncode
aliases['mygrep']=_mygrep
echo hovercraft|mygrep raft
# hovercraft
echo hovercraft|mygrep raft

xonsh: To log full traceback to a file set: $XONSH_TRACEBACK_LOGFILE = <filename>
hovercraft
Traceback (most recent call last):
  File "/usr/local/Cellar/xonsh/0.11.0/libexec/lib/python3.10/site-packages/xonsh/procs/__amalgam__.py", line 2328, in run
    r = self.f(self.args, sp_stdin, sp_stdout, sp_stderr, spec, spec.stack)
  File "/usr/local/Cellar/xonsh/0.11.0/libexec/lib/python3.10/site-packages/xonsh/procs/__amalgam__.py", line 2117, in proxy_four
    return f(args, stdin, stdout, stderr)
  File "<stdin>", line 2, in _mygrep
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1804, in _execute_child
    os.close(errpipe_read)
OSError: [Errno 9] Bad file descriptor

For community

⬇️ Please click the 👍 reaction instead of leaving a +1 or 👍 comment

[anki-code]

The xontrib/coreutils.py module is pretty simple. To trace this we can just comment all code and uncomment line by line. As result I see that the line that leads to error:

xonsh/xontrib/coreutils.py

Line 33 in 9f0bf03

xsh.aliases["echo"] = echo

Here echo becomes callable alias and we see:

xonsh --no-rc

$XONSH_SHOW_TRACEBACK = True
$XONSH_TRACE_SUBPROC = 3

import subprocess
@aliases.register('mygrep')
def _mygrep(args, stdin=None, stdout=None, stderr=None):
    return subprocess.run(["/usr/bin/grep"]+args,  stdin=stdin, stdout=stdout, stderr=stderr).returncode

echo hovercraft | mygrep raft  # here echo is binary

# Trace run_subproc({'cmds': (['echo', 'hovercraft'], '|', ['mygrep', 'raft']), 'captured': 'hiddenobject'})
# 0: {'cmd': ['echo', 'hovercraft'], 'cls': 'subprocess.Popen', 'binary_loc': '/opt/homebrew/Cellar/coreutils/9.5/libexec/gnubin/echo', 'threadable': True, 'background': False, 'stdout': 7, 'captured': 'hiddenobject'}
# 1: {'cmd': ['mygrep', 'raft'], 'cls': 'xonsh.procs.proxies.ProcProxyThread', 'alias_name': 'mygrep', 'threadable': True, 'background': False, 'stdin': 3, 'stdout': <_io.TextIOWrapper name=9 mode='w' encoding='utf-8'>, 'stderr': <_io.TextIOWrapper name=11 mode='w' encoding='utf-8'>, 'captured': 'hiddenobject', 'captured_stdout': <_io.TextIOWrapper name=8 mode='r' encoding='utf-8'>, 'captured_stderr': <_io.TextIOWrapper name=10 mode='r' encoding='utf-8'>}
# hovercraft

from xonsh.xoreutils.echo import echo
aliases["echo"] = echo

echo hovercraft | mygrep raft   # here echo is callable alias and first call is working
echo hovercraft | mygrep raft

# Trace run_subproc({'cmds': (['echo', 'hovercraft'], '|', ['mygrep', 'raft']), 'captured': 'hiddenobject'})
# 0: {'cmd': ['echo', 'hovercraft'], 'cls': 'xonsh.procs.proxies.ProcProxyThread', 'alias_name': 'echo', 'threadable': True, 'background': False, 'stdout': 7, 'captured': 'hiddenobject'}
# 1: {'cmd': ['mygrep', 'raft'], 'cls': 'xonsh.procs.proxies.ProcProxyThread', 'alias_name': 'mygrep', 'threadable': True, 'background': False, 'stdin': 3, 'stdout': <_io.TextIOWrapper name=9 mode='w' encoding='utf-8'>, 'stderr': <_io.TextIOWrapper name=11 mode='w' encoding='utf-8'>, 'captured': 'hiddenobject', 'captured_stdout': <_io.TextIOWrapper name=8 mode='r' encoding='utf-8'>, 'captured_stderr': <_io.TextIOWrapper name=10 mode='r' encoding='utf-8'>}
# Exception

# Exception in thread {'cls': 'ProcProxyThread', 'name': 'Thread-17', 'func': FuncAlias({'name': 'mygrep', 'func': '_mygrep'}), 'alias': 'mygrep', 'pid': None}
# hovercraft
# Traceback (most recent call last):
  File "/Users/pc/.local/xonsh-env/lib/python3.12/site-packages/xonsh/procs/proxies.py", line 470, in run
    r = run_with_partial_args(
        ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pc/.local/xonsh-env/lib/python3.12/site-packages/xonsh/cli_utils.py", line 381, in run_with_partial_args
    return func(**kwargs)
           ^^^^^^^^^^^^^^
  File "/Users/pc/.local/xonsh-env/lib/python3.12/site-packages/xonsh/aliases.py", line 86, in __call__
    return self.func(*func_args)
           ^^^^^^^^^^^^^^^^^^^^^
  File "<stdin>", line 8, in _mygrep
  File "/Users/pc/.local/xonsh-env/lib/python3.12/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pc/.local/xonsh-env/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/pc/.local/xonsh-env/lib/python3.12/subprocess.py", line 1914, in _execute_child
    os.close(errpipe_read)
OSError: [Errno 9] Bad file descriptor

# NOTE! The error `os.close(errpipe_read): Bad file descriptor` 
# occurs in `subprocess.run(['/usr/bin/grep', 'raft'], ...)` call.

As you can see subprocess likes to close the pipe. This is why you can see no errors on the first call and the exception on the second call - subprocess closed the pipe after the first call.

So let's set close_fds=False:

$XONSH_SHOW_TRACEBACK = True

import subprocess
@aliases.register('mygrep')
def _mygrep(args, stdin=None, stdout=None, stderr=None):
    print('***',stdin,stdout,stderr)
    return subprocess.run(["/usr/bin/grep"]+args, stdin=stdin, stdout=stdout, stderr=stderr, close_fds=False).returncode

from xonsh.xoreutils.echo import echo
aliases["echo"] = echo

echo hovercraft | mygrep raft
echo hovercraft | mygrep raft
echo hovercraft | mygrep raft
echo hovercraft | mygrep raft
echo hovercraft | mygrep raft
# It works.

Conclusions:

1. Mixing management of std from callable alias machinery with calling python subprocess is not safe because you need to manage Popen calls on your own. So you need to set close_fds=False manually.

2. The error

     File "/Users/pc/.local/xonsh-env/lib/python3.12/subprocess.py", line 1026, in __init__
       self._execute_child(args, executable, preexec_fn, close_fds,
     File "/Users/pc/.local/xonsh-env/lib/python3.12/subprocess.py", line 1914, in _execute_child
       os.close(errpipe_read)
   OSError: [Errno 9] Bad file descriptor

   looks like a Python bug (src) because errpipe_read was created and used inside _execute_child during subprocess.run(['/usr/bin/grep', 'raft'], ...) call. It will be cool to investigate, create pure python example and report to the general Python issue tracker.

3. Maybe the provided example in this issue is just replacing the alias by another command. There is PR that introduces the alias that just returns new command - Alias that returns modified command #5473.

[anki-code]

I'm going to close this because the case fully captured.