Best practices for starting new threads/processes (that are not part of our WSGI app) from within our WSGI app

[EB123]

Hi,

We need some advice about an issue we're facing and i hope you might be able to give us some insights.

So, we have a few python microservices (and some more are currently in development), all of them are a flask app wrapped by gunicorn (currently we're mostly using the gevent worker type).

We've come to realize that sometime a request-response type of work is not what we need, and a more "worker-type" microservice will be more suitable, so we're currently in the process of developing a in-house library that will help developers in our company to create a worker-app, basically it's some wrapper above the pika library that will help them to easily create consumers that will get jobs from rabbitmq.

The thing is, most of our "worker-type" microservices will still need some endpoints, so our initial plan was that the rabbitmq consumers will be initialized from the main.py of our flask app. But, when we started to dig a bit deeper, this all started to seem a bit complex.

So, here is what i'm asking - is there a "best-practice" for starting some threads/processes (the pika consumers) from within our WSGI app, but the threads/processes are NOT part of our WSGI app?

Here's for example one of our concierns, maybe it will also help to make my question clearer :) :

• workers recycling - if a worker didnt handle any request for X time (X > worker_timeout), but the worker process is currently also running a pika thread that is doing some heavy cpu work, will gunicorn consider this worker to be "silent" and restart it?

We're aware that the best solution might just to separate them both and not initialize the workers from within our flask app, but most likely that at least for the near future it wont be an option (or a very undesirable option).

So, are there any best-practices for this kind of use case?
Thanks!

[tilgovi]

Best practice is probably to make them separate. Even if they run together it is best to have two entry points rather than try to spawn a background thread for Gunicorn.

For example, with k8s you could have a pod with two containers. For Amazon ECS, a task with two containers. For some process supervisor, start two daemons.

I hope that helps!

[haku-d]

I also have same problem, my application need to publish a message to rabbitmq on every coming request. I started by using pika's blocking connection on each worker, but after X time the connection closed automatically, so any next incoming request raised a connection exception.

I don't know should I start a pika's SelectionConnection on other thread using ioloop to keep the connection still alive?