use gunicorn

[ryandawsonuk]

First steps towards addressing #383

See the link I posted on that issue. Seems to be common to use just flask on local machine and add gunicorn within container. See for example https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xix-deployment-on-docker-containers

Have also done this in https://github.com/SeldonIO/seldon-core/blob/master/seldon-request-logger/Dockerfile

[axsaucedo]

Just as a heads up it's worth taking into consideration that the Gunicorn server actually spins up multiple workers by default, which normally is indeed desired), howver given that we (at least currently) offload load balancing and replicas to the gateway components, we may have to avoid having more than one worker running. I do think that the benefits make it a great choice, and it's not too hard to set up 👍 we could even use this as an opportunity to discuss normalising the microservice vs seldonEngine APIs

[ukclivecox]

We have considered gUnicorn as a standard way to wrap the Flask server we use for the python wrapper. This is also used by Sagemaker. Another option is to remove Flask and use Tornado.

[ryandawsonuk]

My thinking is that gunicorn on k8s will help to get maximum use out of each container. The articles indicate that you can reliably handle a greater throughput of requests in a single container by adding gunicorn. So I’d expect that to carry over to multiple resource-constrained containers. To be sure we could do new benchmarking https://docs.seldon.io/projects/seldon-core/en/latest/reference/benchmarking.html

My impression is it doesn’t seem to negatively affect cold-start.

We could instead just add a note to the docs saying that gunicorn isn’t included by default and how to add it if desired.

[ryandawsonuk]

Closing this in favour of #684