yolov5s.torchscript.ptl

[bairock]

Version

1.1.0

Problem Area

react-native-pytorch-core (core package)

Steps to Reproduce

run model yolov5s.torchscript.ptl

Expected Results

get results (bbox, class)

Code example, screenshot, or link to repository

Error Possible Unhandled Promise Rejection (id: 0):
Error: End of input at character 0 of promiseMethodWrapper

Did I follow this instruction to convert .pt to .ptl or am I doing something wrong?
https://github.com/pytorch/android-demo-app/tree/master/ObjectDetection
I ran android project pytorch mobile model worked there.

As I understand it, you need live.spec.json but where to get it for yolov5?

It's him ? I used it but it still gives an error or I need to project it?
{'config.txt': '{"shape": [1, 3, 640, 640], "stride": 32, "names": ["person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light", "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow", "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard", "cell phone", "microwave", "oven", "toaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush"]}'}

here is the link to the issue

[chrisklaiber]

Hello @bairock, thank you for your interest in PyTorch Live! As far as I’m aware, there isn’t yet a Live Spec defined by the community for YOLOv5. Are you planning to create one?

You can follow the tutorial here for instructions about how to adapt existing models to PyTorch Live (https://pytorch.org/live/docs/tutorials/prepare-custom-model), which includes creating a Live Spec. A Live Spec defines how to convert from an input in JavaScript to the tensor the model expects (this is the pack step) and how to convert from the model’s output to what will be returned to JavaScript (the unpack step). Existing operations for use in the pack and unpack steps can be found in the API for Model Specification (https://pytorch.org/live/docs/api/model-spec/). Examples of existing Live Specs that ship with the PyTorch Live example app can be found at https://github.com/pytorch/live/tree/main/react-native-pytorch-core/example/models

When you are successful, please do share a code pointer to the Live Spec you created! If you find that the Live Spec does not offer the operations you require, please let us know.

[bairock]

Hello @bairock, thank you for your interest in PyTorch Live! As far as I’m aware, there isn’t yet a Live Spec defined by the community for YOLOv5. Are you planning to create one?

You can follow the tutorial here for instructions about how to adapt existing models to PyTorch Live (https://pytorch.org/live/docs/tutorials/prepare-custom-model), which includes creating a Live Spec. A Live Spec defines how to convert from an input in JavaScript to the tensor the model expects (this is the pack step) and how to convert from the model’s output to what will be returned to JavaScript (the unpack step). Existing operations for use in the pack and unpack steps can be found in the API for Model Specification (https://pytorch.org/live/docs/api/model-spec/). Examples of existing Live Specs that ship with the PyTorch Live example app can be found at https://github.com/pytorch/live/tree/main/react-native-pytorch-core/example/models

When you are successful, please do share a code pointer to the Live Spec you created! If you find that the Live Spec does not offer the operations you require, please let us know.

Is there an example of unpuck type tuple? found no example anywhere.

[Aisenapio]

Hi, what the structure of the items in the tuple unpack should be, here is an example:

#	xmin	ymin	xmax	ymax	confidence	class	name
0	749.50	43.50	1148.0	704.5	0.874023	0	person
1	433.50	433.50	517.5	714.5	0.687988	27	tie
2	114.75	195.75	1095.0	708.0	0.624512	0	person
3	986.00	304.00	1028.0	420.0	0.286865	0	tie

[dongdv95]

Hi, what the structure of the items in the tuple unpack should be, here is an example:

xmin ymin xmax ymax confidence class name

0 749.50 43.50 1148.0 704.5 0.874023 0 person
1 433.50 433.50 517.5 714.5 0.687988 27 tie
2 114.75 195.75 1095.0 708.0 0.624512 0 person
3 986.00 304.00 1028.0 420.0 0.286865 0 tie

Hi @Aisenapio , have you created Live Specs of yolov5 ? or an example of unpuck type tuple.

[JohnZcp]

Hello, I am also trying to create the spec.json for yolo_v5. How is your progress? @bairock

[bairock]

Hello, I am also trying to create the spec.json for yolo_v5. How is your progress? @bairock

hachiko mode

[JohnZcp]

Do you have any idea about the unpack? Currently, I guess the output type should be set as object "models.common.Detections". About I don't think torch live recognizes this type as output according to tutorial.

[jslok]

Any progress on exporting yolov5 to .ptl?

[zhiqwang]

Just FYI,

The official torchscript exported by yolov5 only contains the general model inferencing part, one must implement the pre-preprocess (letterbox) and post-proprocess (mainly the nms op) if they want to deploy this torchscript.

We did some experiments on yolort to embed the pre-processing and post-processing into the torchscript graph following the strategy of TorchVision's object detection banks, as such we don't need to write the pre-process and post-process within torchlive. So it will be very easy to deploy YOLOv5 with yolort if we could deploy the models in TorchVision on torchlive.

[raedle]

We can take a look, but would need a working torchscript-ed model.

Does anyone have a working torchscript-ed version of a YOLOv5 model that accepts an image tensor as input and can share it here for download?

[zhiqwang]

Hi @raedle , I export a YOLOv5 torchscript model that accepts an image tensor as the input with yolort, and I upload the torchscript and optimized model at

• https://huggingface.co/spaces/zhiqwang/assets/blob/main/yolov5s_scripted.pt
• https://huggingface.co/spaces/zhiqwang/assets/blob/main/yolov5s_scriptmodule.ptl

I only embed the post-processing (nms) into the torchscript graph, and the model is same with YOLOv5, check out this docs for more details, and feel free to pin me if anyone have any questions about how to export the torchscript-ed model or the relationship between YOLOv5 and yolort.

And you can use the following scripts to test the inference output using this torchscript-ed models (install yolort with pip install yolort==0.6.3 fist):

import cv2
import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile
from yolort.utils import Visualizer, read_image_to_tensor
from yolort.v5 import letterbox, scale_coords, attempt_download

img_size = 640
stride = 32
device = torch.device('cpu')

export_scripted_source = "https://huggingface.co/spaces/zhiqwang/assets/resolve/main/yolov5s_scripted.pt"
export_scripted_path = attempt_download(export_scripted_source)

export_optimized_path = "yolov5s_scriptmodule.ptl"

### Load the TorchScript model

scripted_model = torch.jit.load(export_scripted_path)

optimized_model = optimize_for_mobile(scripted_model)
# Or download the optimized model from https://huggingface.co/spaces/zhiqwang/assets/blob/main/yolov5s_scriptmodule.ptl
optimized_model._save_for_lite_interpreter(export_optimized_path)

scripted_model = scripted_model.eval()
scripted_model = scripted_model.to(device)

### Load the image

img_source = "https://huggingface.co/spaces/zhiqwang/assets/resolve/main/bus.jpg"
# img_source = "https://huggingface.co/spaces/zhiqwang/assets/resolve/main/zidane.jpg"
img_path = attempt_download(img_source)
img_raw = cv2.imread(img_path)

image = letterbox(img_raw, new_shape=(img_size, img_size), stride=stride)[0]
image = read_image_to_tensor(image)
image = image.to(device)
image = image[None]

with torch.no_grad():
    out_script = scripted_model(image)

# Don't forget to rescale the coordinates back to original image scale
scale_coords(image.shape[2:], out_script[1][0]['boxes'], img_raw.shape[:-1])

print(out_script)

And the out_script[1][0] is the inference result of this image.

{'scores': tensor([0.89617, 0.87025, 0.85156, 0.84933, 0.53494]),
 'labels': tensor([0, 0, 0, 5, 0]),
 'boxes': tensor([[6.71788e+02, 3.95372e+02, 8.10000e+02, 8.78361e+02],
         [2.20657e+02, 4.08141e+02, 3.46167e+02, 8.67381e+02],
         [4.92508e+01, 3.89991e+02, 2.48078e+02, 9.12459e+02],
         [1.26507e+01, 2.23378e+02, 8.09707e+02, 7.88516e+02],
         [4.53959e-02, 5.52411e+02, 6.78823e+01, 8.75375e+02]])}

[raedle]

Thanks for sharing the inference code, @zhiqwang! That was incredibly helpful!

Here is a screencast of an early working version. I will need to finalize the changes and then make it available for everyone as soon as possible.

Thanks again for the model and the example!

yolov5s.mp4

[zhiqwang]

Hi @raedle , It would be my pleasure and I can add a more detailed scripts for exporting the torchscript-ed module of YOLOv5 if you need it.

• First, we can use the following script to verify the inference output between YOLOv5 and yolort.

  Click to show the verification details

  # Prepare environment, image and model weights to test
  
  import cv2
  import torch
  from yolort.models.yolo import YOLO
  from yolort.utils import Visualizer, read_image_to_tensor
  from yolort.v5 import load_yolov5_model, letterbox, non_max_suppression, scale_coords, attempt_download
  
  device = torch.device('cpu')
  
  img_source = "https://huggingface.co/spaces/zhiqwang/assets/resolve/main/bus.jpg"
  # img_source = "https://huggingface.co/spaces/zhiqwang/assets/resolve/main/zidane.jpg"
  img_path = attempt_download(img_source)
  img_raw = cv2.imread(img_path)
  
  # Downloaded from 'https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt'
  model_path = 'yolov5s.pt'
  checkpoint_path = attempt_download(model_path)
  
  img_size = 640
  stride = 32
  score_thresh = 0.25
  nms_thresh = 0.45
  
  # Here we use the letterbox of YOLOv5 to do the image pre-processing.
  
  image = letterbox(img_raw, new_shape=(img_size, img_size), stride=stride)[0]
  image = read_image_to_tensor(image)
  image = image.to(device)
  image = image[None]
  
  ## Load model as ultralytics and inference
  
  model_yolov5 = load_yolov5_model(checkpoint_path, fuse=True)
  model_yolov5 = model_yolov5.to(device)
  model_yolov5 = model_yolov5.eval()
  
  with torch.no_grad():
      yolov5_dets = model_yolov5(image)[0]
      yolov5_dets = non_max_suppression(yolov5_dets, score_thresh, nms_thresh, agnostic=False)[0]
  
  ## Updating model weights from ultralytics to yolort and inference
  
  model_yolort = YOLO.load_from_yolov5(
      checkpoint_path,
      score_thresh=score_thresh,
      nms_thresh=nms_thresh,
      version="r6.0",
  )
  
  model_yolort = model_yolort.eval()
  model_yolort = model_yolort.to(device)
  
  with torch.no_grad():
      yolort_dets = model_yolort(image)
  
  print(f"Detection boxes with yolort:\n{yolort_dets[0]['boxes']}")
  print(f"Detection scores with yolort:\n{yolort_dets[0]['scores']}")
  print(f"Detection labels with yolort:\n{yolort_dets[0]['labels']}")
  
  ## Verify the detection results between yolort and ultralytics
  
  # Testing boxes
  torch.testing.assert_allclose(yolort_dets[0]['boxes'], yolov5_dets[:, :4])
  # Testing scores
  torch.testing.assert_allclose(yolort_dets[0]['scores'], yolov5_dets[:, 4])
  # Testing labels
  torch.testing.assert_allclose(yolort_dets[0]['labels'], yolov5_dets[:, 5].to(dtype=torch.int64))
  
  print("Exported model has been tested, and the result looks good!")
  
  # Detection output visualisation
  
  # Hah, that's the trick to rescale the box correctly. We need to scale the inference results back to the original scale of the image.
  scale_coords(image.shape[2:], yolort_dets[0]['boxes'], img_raw.shape[:-1])
  
  # This only works on Jupyter now
  v = Visualizer(img_raw, model_yolov5.names)
  # Prepare the prediction labels for the Visualizer
  v.draw_instance_predictions(yolort_dets[0])
  v.imshow(scale=0.5)

• Then we could export the torchscript-ed module of YOLOv5 (containing the nms op)

  ## Export the TorchScript-ed model
  
  import torch
  import torchvision
  from torch.utils.mobile_optimizer import optimize_for_mobile
  from yolort.models import YOLO
  from yolort.v5 import attempt_download
  
  # Prepare some parameters for the exported torchscript-ed module
  score_thresh = 0.25
  nms_thresh = 0.45
  device = torch.device("cpu")
  
  # Downloaded from 'https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt'
  model_path = "yolov5s.pt"
  checkpoint_path = attempt_download(model_path)
  
  model = YOLO.load_from_yolov5(checkpoint_path, score_thresh=score_thresh, nms_thresh=nms_thresh)
  
  model = model.eval()
  model = model.to(device)
  
  export_scripted_path = "yolov5s_scripted.pt"
  export_optimized_path = "yolov5s_scriptmodule.ptl"
  
  scripted_model = torch.jit.script(model)
  scripted_model.save(export_scripted_path)
  optimized_model = optimize_for_mobile(scripted_model)
  optimized_model._save_for_lite_interpreter(export_optimized_path)

• Last, let's verify the inference result between the PyTorch and TorchScript

  Click to show the verification details

  scripted_model = torch.jit.load(export_scripted_path)
  scripted_model = scripted_model.eval()
  scripted_model = scripted_model.to(device)
  
  ## Verify the detection results between PyTorch and TorchScript
  with torch.no_grad():
      out_script = scripted_model(image)
  
  scale_coords(image.shape[2:], out_script[1][0]['boxes'], img_raw.shape[:-1])
  
  for k, v in yolort_dets[0].items():
      torch.testing.assert_allclose(out_script[1][0][k], v, rtol=1e-07, atol=1e-09)
  
  print("Exported model has been tested with libtorch, and the result looks good!")

[limzh123]

Hi, do anyone know how to deploy a custom model into PyTorch live? I am encountering some bugs.

[JohnZcp]

@raedle I also used the yolot generate the .ptl file for yolov5, but received the error "Could not convert downloaded file into Torch Module". I compared my model to the example of MobileNetV3, and they are all RecursiveScriptModule. Since yolot and yolov5 have the same model structure, I am not sure the reason for this error. I checked the code of the package, and it said "TorchModule.load will set an empty string if the model file is not bundled inside the model file". Can you provide more explanation?

[zhiqwang]

Hi @JohnZcp ,

and it said "TorchModule.load will set an empty string if the model file is not bundled inside the model file".

May I ask do you use the yolort model with the pre-processing (letterbox) part? I also encountered some problems when including pre-processing, I'm not quite sure why this is happening. But everything works fine when exporting yolov5 with only post-processing (nms) as above (and I copied the detailed conversion script below).

## Export the TorchScript-ed model

import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile
from yolort.models import YOLO
from yolort.v5 import attempt_download

# Prepare some parameters for the exported torchscript-ed module
score_thresh = 0.25
nms_thresh = 0.45
device = torch.device("cpu")

# Downloaded from 'https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt'
model_path = "yolov5s.pt"
checkpoint_path = attempt_download(model_path)

model = YOLO.load_from_yolov5(checkpoint_path, score_thresh=score_thresh, nms_thresh=nms_thresh)

model = model.eval()
model = model.to(device)

export_scripted_path = "yolov5s_scripted.pt"
export_optimized_path = "yolov5s_scriptmodule.ptl"

scripted_model = torch.jit.script(model)
scripted_model.save(export_scripted_path)
optimized_model = optimize_for_mobile(scripted_model)
optimized_model._save_for_lite_interpreter(export_optimized_path)

[JohnZcp]

@zhiqwang Yes. And I think we still need to provide the live.spec.json as the extra file for generating ptl version. But, I don't think missing this json file will cause the error I mentioned above. It only effects library's data processing for input and output.

[raedle]

@JohnZcp, @Aisenapio, @dongdv95, @bairock, @jslok, and @zhiqwang, the latest release 0.2.0 of PyTorch Live has a JavaScript API to pre-/post-process data. The live.spec.json is no longer needed.

I create a YOLOv5 example app with the new PyTorch Live API.

It would be fantastic if this could also work with the YOLOv5 Runtime Stack!

[chrisklaiber]

Closing this issue because it appears to be resolved. Feel free to re-open if more info is needed. Also note, there is now a YOLOv5 tutorial available: https://playtorch.dev/docs/tutorials/snacks/yolov5/