U2-net model for Pytorch Live

[lavandaboy]

Tutorial Select

Prepare Custom Model

Feedback

Hi Pytorch Live community,

I am trying to implement the (U2-net model) instead of AnimeGan model which is provided as an example to test how it works with Pytorch Live. The steps I do:

1. Conversion of the u2netp model (the small version of the U2-net model) using this manual and the same live.spec.
2. After the conversion I got this model - https://github.com/lavandaboy/deeplabv3-pytorch-live/blob/master/u2netp_small_live.ptl
3. I deploy the example with AnimeGan model, and everything works perfectly
4. I change the link to the model of Animegan in Model.ts file to this (https://github.com/lavandaboy/deeplabv3-pytorch-live/blob/master/u2netp_small_live.ptl)
5. I deploy everything on my ios device

As the result, nothing happens ("Processing..." is )I receive the following error: Possible Unhandled Promise Rejection (id:0): Object { "message": " ",}

Could you please advise how to implement the U2-net model into Pytorch Live? If you need anything else to provide from me please let me know.

Thanks!

[raedle]

EDIT: It was an error on my end. The wget command doesn't work with files committed to a GitHub repo. I got a step further, and will let you know how it goes.

Ignore message below for now :)

Hi @lavandaboy, I tried loading the model as lite interpreter model in a Google Colab, but get a RuntimeError.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-4-86c4d700d487>](https://localhost:8080/#) in <module>()
      1 from torch.jit.mobile import _load_for_lite_interpreter
      2 
----> 3 model = _load_for_lite_interpreter('/content/u2netp_small_live.ptl')
      4 
      5 print(model)

[/usr/local/lib/python3.7/dist-packages/torch/jit/mobile/__init__.py](https://localhost:8080/#) in _load_for_lite_interpreter(f, map_location)
     46 
     47     if isinstance(f, str) or isinstance(f, pathlib.Path):
---> 48         cpp_module = torch._C._load_for_lite_interpreter(f, map_location)
     49     else:
     50         cpp_module = torch._C._load_for_lite_interpreter_from_buffer(f.read(), map_location)

RuntimeError: Format error

Can you provide the steps how you exported the model and saved it for the lite interpreter runtime (e.g., as a Google Colab)? Ideal would be if you could provide an example in the Google Colab how to use the model.

1. With PyTorch Live and the react-native-pytorch-core release 0.2.0, the live spec is no longer needed. The data processing can be done in JavaScript-only
2. See above comment about RuntimeError
3. Are the model inputs and output the same as with the AnimeGANv2 model?

Happy to give it a try if you can provide additional info as mentioned above 🙂

[raedle]

Hey @lavandaboy, I was able to use your exported model and build a simple on-device demo with PyTorch Live. You can find the code in the following repo:

https://github.com/raedle/UNetExample

Can you provide instructions here or as a PR on the repo how you exported the model for PyTorch Mobile lite interpreter? That would be rad!

Thank you!

[lavandaboy]

Hi Roman,

As always, thanks a lot for your help!

The conversion manual is the following:

1. Clone https://github.com/NathanUA/U-2-Net.git
2. Open file u2net_test.py
3. Go to row 57 and keep model_name='u2net' if you want to use the full model or change to model_name='u2netp' if you want to use the small model (less than 5Mb)
4. Go to row 94 (just below "net.eval()" code)
5. Add the following code:

scripted_model = torch.jit.script(net)
optimized_model = optimize_for_mobile(scripted_model)
spec = Path("live.spec.json").read_text()
extra_files = {}
extra_files["model/live.spec.json"] = spec
optimized_model._save_for_lite_interpreter("u2net_small_live.ptl", _extra_files=extra_files)

6. Add the following code to the beginning of the file u2net_test.py:

from pathlib import Path
from torch.utils.mobile_optimizer import optimize_for_mobile

7. Add live.spec.json file into the root of the U2net folder with the following code inside:

{
  "pack": {
    "type": "tensor_from_image",
    "image": "image",
    "transforms": [
      {
        "type": "image_to_image",
        "name": "center_crop"
      },
      {
        "type": "image_to_image",
        "name": "scale",
        "width": 224,
        "height": 224
      },
      {
        "type": "image_to_tensor",
        "name": "rgb_norm",
        "mean": [0.485, 0.456, 0.406],
        "std": [0.229, 0.224, 0.225]
      }
    ]
  },
  "unpack": {
    "type": "argmax",
    "dtype": "float",
    "key": "maxIdx",
    "valueKey": "confidence"
  }
}

8. Upload the pretrained model from U2-net repo into the folder /saved_models/unetp/ if you are using the small version or into /saved_models/unet/ if you are using the full model
9. Run u2net_test.py
10. You will see your converted model for PyTorch Live u2net_small_live.ptl in the folder of U2-Net

Hope this manual helps to convert more models to Pytorch Live!

Now I will play with your working example :) Thanks, Roman!