Use Gunicorn with Flask to serve a Pytorch Model

[kalyangvs]

gunicorn is used as gunicorn app:app --preload --workers 3
Preload is used to share the resources among the workers.
Set the OMP_NUM_THREADS to 2.

app.py contains the following code

from flask import Flask, jsonify
import torch  
from create_model import testtype or paste code here
app = Flask(__name__) 
model = torch.load('model.pt')

@app.route('/predict',methods = ['POST', 'GET']) 
def prediction(): 
    constant_input = torch.randn(20, 16, 50, 100)
    prediction = model(constant_input)
    return jsonify(prediction)

model.pt is created using create_model.py containing

import torch
import torch.nn as nn
import torch.nn.functional as F

class test(nn.Module):
    def __init__(self):
        super(test, self).__init__()
        self.conv1 = nn.Conv2d(16, 33, 3, stride=2)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return x

m = test()
input = torch.randn(20, 16, 50, 100)
print(m(input))
torch.save(m, 'model.pt')

But I am not able to infer it .
Instead of using a torch model if I use some numpy operation and just return its output, it is able to.

1. Though using gunicorn app:app --preload --workers 3 --threads 2 I am able to infer. But anyone please tell me why does it differ only when threads are used .
2. Even with -k gthreads it works fine instead of threads
   Thanks.

[benoitc]

what do you mean by "But I am not able to infer it ." does it fails?

[kalyangvs]

Yes it fails.
But works in the other two cases.

[benoitc]

do you have a trace of the failure? what does it raises?

On Mon 11 Nov 2019 at 04:50 gvskalyan ***@***.***> wrote: Yes it fails. But works in the other two cases. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2157?email_source=notifications&email_token=AAADRIXBUKCTZXPQQX5RRH3QTDI65A5CNFSM4JLDFBUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDVSU6Q#issuecomment-552282746>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADRIUZYX5UOSKPPAIMXC3QTDI65ANCNFSM4JLDFBUA> .

-- Sent from my Mobile

[kalyangvs]

The worker gets timed out . I tried with various high timeout values and with working cases as in threads=2 it works!.

[tilgovi]

I tried with various high timeout values and with working cases as in threads=2 it works!

Does this mean that the issue can be closed?

[kalyangvs]

Can you please address why this does not work with not having threads?
Else can you please elaborate on why having gthreads works?

The worker gthread is a threaded worker. It accepts connections in the main loop, accepted connections are added to the thread pool as a connection job. On keepalive connections are put back in the loop waiting for an event. If no event happen after the keep alive timeout, the connection is closed.

[tilgovi]

The default, synchronous worker will be killed if it does not generate a response within the timeout (default 30s). The threaded worker can signal liveness on a separate thread, so it generally does not time out (unless there is a bug in the interpreter, or unsafe C code that leads to a deadlock, or something like this).

How long do your requestts take?

[kalyangvs]

It depends on no of words present in a sentence. But for the same sentence which took around 250 ms for serving the inference. The default setting with sync workers got timedout.

[kalyangvs]

The reload option is not reloading the app. it detects a change is made, the debug log shows workers got reinitiated, but the changes I made which caused the reload did not get reflected.

Even put tried using on_reload in conf_ini but it does not enter here.

[tilgovi]

Reload is incompatible with preload.

[benoitc]

closing the issue as it seems answered. Feel free to reopen it if needed.