New fastai inference API

[tcapelle]

Hey Zach,
Let's work here prototyping the inference.
What I would like to have (Santa's wishful letter).

• Streamlined torchscript support on all fastai models, simple models should be compatible with jit.trace and more complex ones, with decisions with jit.script. The guys at facebook may be able to help here, they are super interested on this right now. The user should have simple image preprocessing/posprocessing to make inference work once the models is exported on plain pytorch. If jeremy splits the fastai lib on core/vision/etc... we could depende on the fastai core's.
• ONNX: Exporting on all models, image encoders should work out of the box, some layers are missing for Unet's (PixelShuffle). Tabular should work also. Without being an expert, I would expect that torchscript replaces the ONNX pipeline in the future, one less layer.
• RTTorch: We should start discussing with them probably, as the TensorRT frameworks is super fast for GPU inference. This could be done latter, once we have ONNX exports. I have a contact at NVIDIA that could help us export to TensorRT.
• DeepStream? Stas Beckman is a guru on this topic, we could ask him what he thinks about it.

We should have tests that periodically verify that this functionality is not broken, and the performance is maintained. This is something fastai does not have right now and it needs, e.g., fastai's unet is slower than before, noted this the other day.

Another cool thing, would be to directly serve the model with torch.serve directly from fastai. Like,

learn.serve(port=5151)

and get a service running to make inference over HTTP.

[muellerzr]

Hey Thomas, Sure but I’d need to make a new branch. fastinference already does ONNX completely, so 2 is already done. To me I think jit, ONNX, and torchscript is what we should support, as that’s already quite a lot. Maybe TensorRT, I’ve worked with it before so it’s not the most difficult to do. On the weekends is when I can put in a bit of effort into this. Later today I can make a new branch though.

…

On Wed, Jan 20, 2021 at 2:45 AM Thomas Capelle ***@***.***> wrote: Hey Zach, Let's work here prototyping the inference. What I would like to have (Santa's wishful letter). - Streamlined torchscript support on all fastai models, simple models should be compatible with jit.trace and more complex ones, with decisions with jit.script. The guys at facebook may be able to help here, they are super interested on this right now. - ONNX: Exporting on all models, image encoders should work out of the box, some layers are missing for Unet's (PixelShuffle). Tabular should work also. Without being an expert, I would expect that torchscript replaces the ONNX pipeline in the future, one less layer. - RTTorch: We should start discussing with them probably, as the TensorRT frameworks is super fast for GPU inference. This could be done latter, once we have ONNX exports. I have a contact at NVIDIA that could help us export to TensorRT. - DeepStream? Stas Beckman is a guru on this topic, we could ask him what he thinks about it. We should have tests that periodically verify that this functionality is not broken, and the performance is maintained. This is something fastai does not have right now and it needs, e.g., fastai's unet is slower than before, noted this the other day. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#32>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3YCV55RSBKTHUEZQGC3RLS22JZZANCNFSM4WKKGRDA> .

[muellerzr]

One aspect I think we should also include is exporting of the DataLoaders, similar to how learn.export does. It’s not too hard to do, and can help people who want the speed of ONNX, etc, but still use the fastai pipeline for data preprocessing. Experts can of course go ahead and make their own preprocessing pipeline if they know what they’re doing, but many do not.

…

On Wed, Jan 20, 2021 at 2:54 AM Zachary Mueller ***@***.***> wrote: Hey Thomas, Sure but I’d need to make a new branch. fastinference already does ONNX completely, so 2 is already done. To me I think jit, ONNX, and torchscript is what we should support, as that’s already quite a lot. Maybe TensorRT, I’ve worked with it before so it’s not the most difficult to do. On the weekends is when I can put in a bit of effort into this. Later today I can make a new branch though. On Wed, Jan 20, 2021 at 2:45 AM Thomas Capelle ***@***.***> wrote: > Hey Zach, > Let's work here prototyping the inference. > What I would like to have (Santa's wishful letter). > > - Streamlined torchscript support on all fastai models, simple models > should be compatible with jit.trace and more complex ones, with > decisions with jit.script. The guys at facebook may be able to help > here, they are super interested on this right now. > - ONNX: Exporting on all models, image encoders should work out of > the box, some layers are missing for Unet's (PixelShuffle). Tabular should > work also. Without being an expert, I would expect that torchscript > replaces the ONNX pipeline in the future, one less layer. > - RTTorch: We should start discussing with them probably, as the > TensorRT frameworks is super fast for GPU inference. This could be done > latter, once we have ONNX exports. I have a contact at NVIDIA that could > help us export to TensorRT. > - DeepStream? Stas Beckman is a guru on this topic, we could ask him > what he thinks about it. > > We should have tests that periodically verify that this functionality is > not broken, and the performance is maintained. This is something fastai > does not have right now and it needs, e.g., fastai's unet is slower than > before, noted this the other day. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#32>, or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AB3YCV55RSBKTHUEZQGC3RLS22JZZANCNFSM4WKKGRDA> > . >

[tcapelle]

You don't sleep!