WIP: Update python wrapper to use gunicorn

[ukclivecox]

• Adds a command line arg --workers defaults to 4 (only uses gunicorn if workers>1)
• Adds an optional load() method to user_model prototype. This is called after init in each worker.
  • Needed for Tensorflow where loading your graph needs to be done in each process and not before gunicorn forks

Fixes #453
Fixes #383
Fixes #674

[axsaucedo]

It woudl be worth adding to the flag to the s2i wrapper (the run file) so that the number of workers can be changed when deploying.

The reason why it is important to be able to change the number of workers is because there would be situations where you would only want to have 1 worker per pod - an example would be tasks that have heavy resource usage, such as memory.

If you have an ML graph that takes 50gb RAM for example, you may want your scaling to be horizontally, as if you request 50gb for the pods, as a user they may be confused that their pods keep going over their requests.

[ukclivecox]

@axsaucedo You can set the env vairaible GUNICORN_WORKERS to limit the number of workers so don't think an s2i change is needed.

[axsaucedo]

Ohh my bad, I had not seen the os.environ.get("GUNICORN_WORKERS", "4") - awesome! Thanks for the clarification @cliveseldon

[alexlatchford]

I've been evaluating Seldon this week and this was on my list, nice job as I'm guessing this will land well within our time frame!

Not sure if this is the place to request this but would be great to include an option to choose the worker type, some of our models at inference time go out to feature store to enrich input requests. We'd be doing at as a part of an input_transformer component likely which we could set to just have a single worker and have it feed a model component with many times more workers.

PS. Apologies if there is already a different pattern for this!

[alexlatchford]

Digging in some more, looking at the server implementations as they load the models in memory on the first prediction. This would be done post process fork resulting in the model being loaded into memory as many times are there are workers.

If you also expose the preload_app option for Gunicorn and set it to True and if you've loaded the model during the initial phase (ie. not on the first network call) then it'll keep only one copy in memory and not duplicate the memory space but be accessible for all child processes. This is something we do here at Zillow to keep the memory footprint down in our current iteration of serving infrastructure but keeping a larger number of worker processes 😄

This does have some downsides as it's incompatible with the reload option which enables hot-code reloading (useful for local development) but we just turn off preload_app locally to fix that.

[axsaucedo]

Very interesting @alexlatchford, that's one of the challenges I outlined above, the preload_app functionality would certainly allow for the workers to only load one copy of the model reducing memory footprint. Of course there is still a challenge if the features extracted are big but that is a different challenge. I am not sure how much work may be involved to extend the wrapper to support this, definitely worth checking out, but depending on that we may have to open a separate issue to do this once the initial/basic move to gunicorn is landed.

[axsaucedo]

We could use log.debug as convention, not critical tho

[lennon310]

If you are using gunicorn workers and make this variable overridable, you probably also need to consider (1) the preload option to reduce the memory usage, and (2) max_requests (probably along with max_requests_jitter) to restart the workers to avoid their memory leak. These are the opts in gunicorn command, but I guess there should be corresponding APIs that provide those options too.

Ref: http://docs.gunicorn.org/en/stable/settings.html

[ukclivecox]

I think we would need to make peload optional as not sure it will work for Tensorflow graphs that need a separate session in each fork.