fork-exec and pipe with I/O redirection -- make Python a sane alternative to non-trivial shell scripts
Switch branches/tags
Nothing to show
Pull request Compare This branch is 1 commit behind aht:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

extproc -- fork-exec and pipe with I/O redirection


Design goals:

  • Easy to fork-exec commands, wait or no wait
  • Easy to capture stdout/stderr of children (command substitution)
  • Easy to express I/O redirections
  • Easy to construct pipelines
  • Use short names for easy interactive typing

In effect, make Python a sane alternative to non-trivial shell scripts.

Technically, extproc is a layer on top of subprocess. The subprocess module support a rich API but is clumsy for many common use cases, namely sync/async fork-exec, command substitution and pipelining, all of which is trivial to do on system shells. [1][2]

This module depends on Python 2.6, or where subprocess is available. Doctests require /bin/sh to pass. Tested on Linux.

This is an alpha release. Expect bugs.

Let's start forking!

Cmd(), Sh() and Pipe()

Those objects hold information to prepare for a fork-exec (or in case of Pipe, a series thereof).

The first argument to Cmd() should be a list of command argurments. If a string, it is passed to shlex.split(). Thus,

>>> Cmd(['grep', 'my stuff']) == Cmd('grep "my stuff"')

Sh(cmd) is equivalent to Cmd(['/bin/sh','-c', cmd]). It is also a subclass.

To construct a Pipe(), pass in a list of Cmd's.


run() performs a fork-exec-wait and return the child(ren)'s exit status(es), e.g.

>>> assert Cmd('/bin/true').run() == 0
>>> found_deadbeaf = Pipe(Cmd('dmesg'), Cmd('grep deadbeaf')).run() is a shorthand function:

>>> assert run('/bin/false') == 1


spawn() performs a fork-exec and returns a subprocess.Popen object. The following is equivalent to a gvim -f & on Unix shells:

>>> gvim = Cmd(['gvim', '-f']).spawn()

You may do what you wish to the Popen object, for instance,

>>> gvim.kill(15)


capture() also performs a fork-exec-wait but capture the child's stdin/stderr as file objects when possible, e.g.

>>> Sh('echo -n foo').capture(1)
>>> Sh('echo -n bar >&2').capture(2)

The full return is a namedtuple (stdin, stdout, exit_status), e.g.

>>> out, err, status = Sh('echo -n foo; echo -n bar >&2').capture(1, 2)

Capturing is equivalent to shell backquotes aka command substitution (but sh cannot capture stderr separate from stdout):

$ out=`echo -n foo`
$ outerr=$(echo -n foo; echo -n bar 2>&1 >&2)

extproc.cmd, and extproc.pipe are safe shortcuts that setup the capture of the child(ren)'s stdout, then read and close it, e,g.

>>> sh('echo -n foo')

The following finds files modified in the last 30 minutes and pipes to dmenu(1) to select a single item:

>>> item = pipe(Cmd('find -mmin +30'), Cmd('dmenu'))

I/O redirection

I/O redirections are performed by specifying a fd argument which should be a dict mapping a subset of file descriptors [0, 1, 2] to either open files, strings, or existing file descriptors, e.g.

>>> sh('echo -n foo; echo -n bar >&2', fd={2: 1})

The following append the child's stdout to the file 'abc' (equiv. to echo foo >> abc)

>>> sh('echo foo', {1: open('abc', 'a')})

os.devnull (which is just the string '/dev/null' on Unix) also works:

>>> Sh('echo -n ERROR >&2; echo bogus stuff', {1: os.devnull}).capture(2)

In fact you can pass in fd=SILENCE, which will send everything straight to hell, hmm... I mean /dev/null.


See docstrings for now.


The main interpreter process had better be a single thread, since forking multithreaded programs is not well understood by mortals. [3]

It is not a good idea to reuse Cmd's objects.


capture() use temporary files and is synchronous. It might be worth adding an async=True option to use PIPE for client code that knows what it is doing.

It is really too bad that subprocess does not support full I/O redirection.

See also: ./TODO


[1] sh(1) --

[2] The Scheme Shell --