Is there a way to run a system process asynchronously? #2

Closed
hoisie opened this Issue Sep 10, 2009 · 12 comments

Comments

Projects
None yet

hoisie commented Sep 10, 2009

Is there a way to run system commands asynchronously? For instance, let's say you're given an image as input, and you'd like to run some image conversion or optimization program in the background. Can you do this without blocking?

greut commented Sep 11, 2009

web.by fires up a thread; I've done it by using an external worker.

lucky commented Oct 3, 2009

You probably want subprocess.

@ghost

ghost commented Dec 17, 2009

or celery

yashh commented Dec 31, 2009

The better way to handle this is using a message queue. You push a job onto a queue and a worker picks it up and process it. Check out RabbitMQ, pyres, gearman, bealstalkd etc.

+1 for yashh

parente commented Feb 21, 2010

The multiprocessing package in Python 2.6 std lib works well with Tornado's IO loop.

I wrote a module for tornado similar to the AsyncHTTPClient just for this purpose.
http://github.com/alexdong/tornado/blob/master/tornado/async_process.py

yashh commented Oct 24, 2010

Is there a way to run a set of mysql queries in parallel. Bret Taylor's post on mysql explains that they query all mysql shards in parallel. Can we do this in tornado ?

Well, you might want to leverage the async_process and pipe the result into tornado.

wmark commented Oct 23, 2011

Put the data (e.g. filename of the image in question) in a queue Q and process that queue in another process. Python comes already with everything you need:
http://docs.python.org/library/multiprocessing.html

If you want to do that distributed, check out ZeroMQ http://www.zeromq.org

@zhaozhouping zhaozhouping pushed a commit to zhaozhouping/tornado that referenced this issue Mar 19, 2013

zhaozhouping add ###2 for test bf14c79
Owner

bdarnell commented Mar 30, 2013

Tornado 3.0 introduced tornado.process.Subprocess: http://www.tornadoweb.org/en/stable/process.html#tornado.process.Subprocess

bdarnell closed this Mar 30, 2013

JackTandy referenced this issue Oct 19, 2014

Closed

deleted #1226

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment