Is there a way to run a system process asynchronously?

[hoisie]

Is there a way to run system commands asynchronously? For instance, let's say you're given an image as input, and you'd like to run some image conversion or optimization program in the background. Can you do this without blocking?

[greut]

web.by fires up a thread; I've done it by using an external worker.

[lucky]

You probably want subprocess.

[pitchfork]

or celery

[yashh]

The better way to handle this is using a message queue. You push a job onto a queue and a worker picks it up and process it. Check out RabbitMQ, pyres, gearman, bealstalkd etc.

[fictorial]

+1 for yashh

[ghost]

you can find an example at http://brianglass.wordpress.com/2009/11/29/asynchronous-shell-commands-with-tornado/

[parente]

The multiprocessing package in Python 2.6 std lib works well with Tornado's IO loop.

[alexdong]

I wrote a module for tornado similar to the AsyncHTTPClient just for this purpose.
http://github.com/alexdong/tornado/blob/master/tornado/async_process.py

[yashh]

Is there a way to run a set of mysql queries in parallel. Bret Taylor's post on mysql explains that they query all mysql shards in parallel. Can we do this in tornado ?

[alexdong]

Well, you might want to leverage the async_process and pipe the result into tornado.

[mark-kubacki]

Put the data (e.g. filename of the image in question) in a queue Q and process that queue in another process. Python comes already with everything you need:
http://docs.python.org/library/multiprocessing.html

If you want to do that distributed, check out ZeroMQ http://www.zeromq.org

[bdarnell]

Tornado 3.0 introduced tornado.process.Subprocess: http://www.tornadoweb.org/en/stable/process.html#tornado.process.Subprocess