pytorch： multi workers

[tp-nan]

mosec/examples/resnet50_server_pytorch.py

Line 49 in 1627a9b

class Inference(Worker):

Hi, for multi-instances usage, by default the pytorch use default cuda stream, making it not able to run instances concurrently. So how do you solve this problem? multi-process？

[lkevinzc]

Hi @ShiyangZhang , yes we use multi-process. All the concurrent requests will be queued and multiple processes will consume the queue