Does predictImage support other models besides image classification models?

[cyrillkuettel]

I was just wondering what type of image models predictImage actually supports. I'm assuming here image classification models are supported, but what about other types of models, like for example object detection?

Is the implementation different for these or does this work as well.

Thank you for this plugin by the way, very nice work.

[jimjam-slam]

I'm also hoping to use an object detection model that I've developed with PyTorch Mobile and would love to see an example of it being used if it's supported!

[cyrillkuettel]

I will take up the challenge.
For Object Detection it's a bit trickier, because you'll essentially have to implement your own draw function (for the bounding boxes). This is commonly done in OpenCV – I'll have to check if it's possible to call native OpenCV from dart.

If you only need either Android or iOS, you can checkout the official pytorch android-demo-app or ios-demo-app for object detection. So far as I have tested they work like a charm.

I should have something usable for both iOS an Android by the end of the year. Will update here once I'm there.

[jimjam-slam]

Mmmm, that's what I was thinking! I've been looking at the Flutter Android/iOS interfaces in this package and was hoping to compare them to pytorch_mobile. "End of the year" would be a target for me too, since I'm using it for a side project. Happy to trade notes, @cyrillkuettel!

[jimjam-slam]

@cyrillkuettel:

So I can't speak to the part where you draw the boxes back onto the original image, but as far as actually running the inference goes, I have a long, rambling stream of consciousness as I dove into this plugin, android-demo-app and ios-demo-app, and my tl;dr is that there's absolutely nothing different about running an object detection model compared to running an image classifier - it's just the structure of the output tensor that will be different.

I first thought this meant that you could use customModel in this plugin, provided you understood how to pack the input tensor properly, but I actually think the input tensor structure is identical between the image classifier and the object detector - and the PyTorch "module" (not module in a compiler sense; module in a model sense) is determined entirely by the input file.

I see that the difference between getImagePrediction and getImagePredictionList in this plugin is just that the former does a bit of extra work, processing the results to find the most likely rated classification, whereas the latter just passes them back essentially as is. I'm going to try using this function from the plugin and see what I get back :) Maybe it can work with just a bit of modification (or even none!).

[jimjam-slam]

So, I'm able to sub in getImagePredictionList and add a button to run inspect on that (although I'm not getting much useful info on it), but when I try to switch in an object detection model I've trained, the app crashes with:

Unknown type name 'NoneType':
Serialized   File "code/__torch__/torchvision/models/detection/transform.py", line 11
  image_std : List[float]
  size_divisible : int
  fixed_size : NoneType
               ~~~~~~~~ <--- HERE
  def forward(self: __torch__.torchvision.models.detection.transform.GeneralizedRCNNTransform,
    images: List[Tensor],
*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** -[__NSArrayM insertObject:atIndex:]: object cannot be nil'

Not sure if I'm just doing something wrong with the model I've trained and exported or if there's some adjustment required to load the model in.

(If the code the error is referencing is the fixed_size argument in this file, then it looks like it's supposed to be optional and so should be allowed to be nil?)

[jimjam-slam]

Okay, I've tried loading in YOLOv5s and got the same error. (Heads up for issue #17)

[cyrillkuettel]

@jimjam-slam Over the past week, I've read the source code of flutter_pytorch_mobile as well as the android-demo-app and ios-demo-app. The later being two almost identical copies, simply implemented in a different programming language. (well, almost ;))

I have come to the same conclusion as you:

it's just the structure of the output tensor that will be different

It's important to know the structure of the output tensor. For example, in deeplab Segmentation, the output tensor has the shape [21, 400, 400] because it has 21 classes, width and height is 400. This is one thing that is variable between different models.

As for the errors when loading models: I have discovered that for this particular case, pytorch 11 environment is required else loading models will produce weird behavior.
The reason for this is the pytorch mobile versions are running behind the python version, (for example, android-demo-app uses version 1.10, same for ios: 1.10 and it can't handle models exported with newer versions of pytorch.

Here is the complete example of exporting the model, using pytorch 11.I have used pretrained models for demonstration purposes, but it should work with any model that can be scripted.

import torch
from torch.utils.mobile_optimizer import optimize_for_mobile
model = torch.hub.load('pytorch/vision: v0.11.0', 'deeplabv3_resnet50', pretrained=True)
model.eval()
scripted_module = torch.jit.script(model)
optimized_model = optimize_for_mobile(scripted_module)
optimized_model._save_for_lite_interpreter("deeplabv3_scripted.ptl")

I exported them exactly like shown above. Note that the line _save_for_lite_interpreter is only needed if you use Pytorch lite (LibTorch-Lite). If you use Libtorch you can skip this line. for anyone not wanting to bother setting up a virtual environment, I have uploaded models, I can confirm these worked. You can find the links here:

• deeplabv3_scripted.ptl
• yolov5s.torchscript.ptl

[jimjam-slam]

This is great to see, @cyrillkuettel! I'd noted that the Android and iOS apps use LibTorch Lite whereas this plugin uses LibTorch (or is it the other way around?), so I'm glad you caught the thread and unravelled it. I'll double-check my Python and package versions too.

I'm away for the weekend, but I'm looking forward to applying your work later next week! Great work! 🥳

[cyrillkuettel]

@jimjam-slam I'm happy to see this is useful to anyone, I spent a lot of time tracking down this issue. Yes the plugin uses LibTorch as far as i can see.
Libtorch-Lite is a faster implementation. Fortunately, it's quite easy to switch between the two, you simply need to change a couple of lines. There's nice guide illustrating the migration in the SegmentationModel Readme.

[jimjam-slam]

@cyrillkuettel Thank again for your guidance on this! I'm having a look, and it seems like I trained my model (based on torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained = True)) with torch 1.13.0+cu116 and torchvision 0.14.0+cu116.

The notebook I had based my work on, one of Google's, actually downloads the pytorch/vision repo and checks 0.8.2 out, but it looks like it only uses this to grab some of the scripts in the detection folder out, leading to a confusing situation where torch and torchvision themselves are new but some of the modules from vision and older. Ugh.

I'll see if I get the older version running (I'm not quite as proficient with Python, so I struggle a bit with its package management!) and retry!

[jimjam-slam]

I managed to regree to the older PyTorch and Torchvision versions, but still hit errors loading models:

Could not run 'aten::empty_strided' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build).

I think this is probably because I've been training models with CUDA. I'm having a crack at training one on just the CPU (just for one epoch so that I'm not sitting around forever).

If that works, I'll revert to using a GPU for training and see if I can convert the model back to a CPU backend after training by saving, switching devices, loading with the map_location argument, and saving again.

[cyrillkuettel]

That's a very peculiar error. I always assumed that training with CUDA backend does not change the actual structure of the exported model.

What I do know, is that there do exist some models that simply cannot be scripted in a straightforward way. For example, if you use fancy stuff like fastai, or some other very non-standard model architecture, they can't be scripted.

[jimjam-slam]

Yeah, I'll have to do some more testing here. I tried training on the GPU and then switching to the CPU and saving again, but I still got the error, so I'm not sure if I did it properly. Will have to have another crack at this when I can 😮‍💨

[jimjam-slam]

@cyrillkuettel So it looks like I didn't regress far enough: I didn't clock that this Flutter plugin uses PyTorch 1.8.0, and I was training on PyTorch 1.11.

Unfortunately, I start getting other errors when I try to train all the way back on PyTorch 1.8.0 and Torchvision 0.9.0. I'm wondering whether it'd be hard to do the opposite and bump this plugin's dependencies up a version or two.

Two places I see references to the PyTorch version are:

https://github.com/fynnmaarten/flutter_pytorch_mobile/blob/3a8848d6ef726e2d491d79a9f355f7f7171b71dc/ios/pytorch_mobile.podspec#L24

https://github.com/fynnmaarten/flutter_pytorch_mobile/blob/3a8848d6ef726e2d491d79a9f355f7f7171b71dc/android/build.gradle#L38

Would updating the plugin be as simple as forking, updating those two references, and adding the forked package as a dependency with a local path?

[cyrillkuettel]

@jimjam-slam Actually that was my mistake, i thought the pytorch version was higher than that. I'm very sorry.

Would updating the plugin be as simple as forking, updating those two references, and adding the forked package as a dependency with a local path?

Yes it's really that simple. You can even add a git url as a plugin dependency:

flutter_pytorch_mobile:
    git: git@github.com:name/path-to-forked-repo.git

One minor thing, on iOS, after updating LibTorch version, I had to:

pod update LibTorch

For the changes to take effect.

[jimjam-slam]

Not at all! Thanks very much for your guidance on this—I'll have a crack at upgrading the plugin tonight! 🥳

[jimjam-slam]

Okay, I've made some progress after spending a lot of time trying to get the versions of everything lined up. (This plugin also includes Torchvision in the Android build but not in the iOS build, which was also causing problems.) The changes I've made so far are at: jimjam-slam#1

For folks playing along at home, in my training notebook I'm using:

pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 --extra-index-url https://download.pytorch.org/whl

And when getting the training scripts I'm using:

git clone https://github.com/pytorch/vision.git
cd vision
git checkout v0.12.0

cp references/detection/utils.py ../
cp references/detection/transforms.py ../
cp references/detection/coco_eval.py ../
cp references/detection/engine.py ../
cp references/detection/coco_utils.py ../

I'm now able to load my custom model, but when I make the inference I get:

[log] >>> Running image model...
[W LegacyTypeDispatch.h:79] Warning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (function operator())
forward() Expected a value of type 'List[Tensor]' for argument 'images' but instead found type 'Dynamic<1>'.
[log] >>> Image model inference back!
[log] >>> Inference:
[log] null

I'll have a bit of a closer look at this to make sure I'm providing the plugin with the right type!