-
Notifications
You must be signed in to change notification settings - Fork 1
Home
Proclib is a high-level library that wraps over the standard library
subprocess module. It provides helpful conveniences like automatic
splitting of commands using shlex.split
. Mandatory code example:
>>> from proclib.api import spawn
>>> r = spawn(['cat'], data='stdin\n')
>>> r.out
'stdin\n'
>>> r.status_code
0
"But it reads the entire stdout!", you say. For long running, expensive
processes it is recommended that you stream them- streaming processes
allow you to read from stdout
and stderr
asynchronously
(the process need not exit and can run while you consume the output).
For example:
>>> from contextlib import closing
>>> with closing(spawn(['yes', 'head -n 2'])) as r:
... for item in r.stdout:
... print(item)
...
y
y
With long running processes you often need to pass huge files of data
(preferrably not XML!) in. To do that you simply need to pass a
file object to the data
parameter. Or you can also pass in any
iterable that yields data that can be written, e.g. generator
functions, lists, StringIO
objects, etc.
>>> spawn(['head -n 20', 'tail -n 4'], data=open('README.rst'))
<Response [tail]>
The streaming feature will hopefully cater more to users with heavier workloads than normal. With the streaming feature, Proclib allows you to use a sane language to glue together components of your system to create a data processing pipeline that is written in a variety of languages. Also note that proclib is streaming by default- so the things that look "non streaming" are usually conveniences added for common usage patterns.