Backward compatibility of ptl with live.spec on iOS

[SomaKishimoto]

Version

0.2.0

Problem Area

react-native-pytorch-core (core package)

Steps to Reproduce

1. make ptl file with live.spec.json
2. build a project with ptl on iOS
3. inference a input image
4. get this error

code: EUNSPECIFIED, 
domain: react_native_pytorch_core.BaseIValuePackerError, 
message: decodeObjectsError, 
name: Error, 
nativeStackIOS: [Array], 
stack: 
Error: decodeObjectsError

the project works well in 0.1.3, but the error comes up after upgrade to 0.2.0.
Is there no backward compatibility for ptl with live.spec?

No problem on Android.

Expected Results

No error

Code example, screenshot, or link to repository

No response

[chrisklaiber]

Hi @SomaKishimoto. Live Spec support will be removed in the future, but in 0.2.0 it should continue to work as before.

Is it possible the model is returning a different format than before, or that the Live Spec has changed?

Here is the code where that error is raised: https://github.com/facebookresearch/playtorch/blob/v0.2.0/react-native-pytorch-core/ios/ML/Processing/Unpacker/BoundingBoxesUnpacker.swift#L19-L34

That code is looking for a specific shape for the values in the Live Spec (variable modelSpec) as well as in the output from the model (variable ivalue). It raises the error you are seeing when there is a mismatch in either.

[SomaKishimoto]

@chrisklaiber, thank you for the prompt response.

Is it possible the model is returning a different format than before, or that the Live Spec has changed?

No, I haven't changed the Live Spec and any format in the model.
The model hasn't been changed before and after the upgrade(0.1.3 → 0.2.0)

Here is the code where that error is raised: https://github.com/facebookresearch/playtorch/blob/v0.2.0/react-native-pytorch-core/ios/ML/Processing/Unpacker/BoundingBoxesUnpacker.swift#L19-L34

Yes, I checked it several times, but didn't get hints to solve this error.

Please let me know if you have any idea to solve the problem than the mentioned above.
I'll check it a bit more!

[raedle]

@SomaKishimoto, can you provide the live.spec.json that's packed with the model and the forward signature of the model?

Alternatively, we recommend migrating to the PlayTorch SDK JavaScript API. We can provide support when the live.spec.json is provided

[SomaKishimoto]

@raedle, Here is live.spec.json and def forward.
The model using in my project is YOLOX and the project was created based on the DETR example , so I changed the output of YOLXO to match it.

live.spec.json

{
  "pack": {
    "type": "tensor_from_image",
    "image": "image",
    "transforms": [
      {
        "type": "image_to_image",
        "name": "scale",
        "width": "$scaleWidth",
        "height": "$scaleHeight"
      },
      {
        "type": "image_to_tensor",
        "name": "rgb_norm",
        "mean": [0.0, 0.0, 0.0],
        "std": [0.0039216, 0.0039216, 0.0039216]
      }
    ]
  },
  "unpack": {
    "type": "bounding_boxes",
    "key": "boundingBoxes",
    "dtype": "float",
    "probabilityThreshold": "$probabilityThreshold",
    "classes": ["dog", "cat", "person"]
  }
}

def forward

    def forward(self, x, targets=None):
        # fpn output content features of [dark3, dark4, dark5]
        fpn_outs = self.backbone(x)

        if self.training:
            assert targets is not None
            loss, iou_loss, conf_loss, cls_loss, l1_loss, num_fg = self.head(fpn_outs, targets, x)
            outputs = {
                "total_loss": loss,
                "iou_loss": iou_loss,
                "l1_loss": l1_loss,
                "conf_loss": conf_loss,
                "cls_loss": cls_loss,
                "num_fg": num_fg,
            }
        else:
            num_classes = 3
            conf_thre = 0.6
            nms_thre = 0.45

            prediction = self.head(fpn_outs)
            box_corner = prediction.new(prediction.shape)
            box_corner[:, :, 0] = prediction[:, :, 0] / 416
            box_corner[:, :, 1] = prediction[:, :, 1] / 416
            box_corner[:, :, 2] = prediction[:, :, 2] / 416
            box_corner[:, :, 3] = prediction[:, :, 3] / 416
            prediction[:, :, :4] = box_corner[:, :, :4]

            image_pred = prediction.squeeze(0)

            class_conf, _ = torch.max(image_pred[:, 5 : 5 + num_classes], 1, keepdim=True)
            _, indexes = torch.topk(class_conf, k=20, dim=0)
            boxes, test, confidences = torch.split(image_pred, (4, 1, 3), dim=1)

            conf_mask = (test[indexes] >= conf_thre).squeeze()
            indexes = indexes[conf_mask]

            sorted_confidences = torch.gather(confidences, 0, indexes.expand(-1, 3))
            sorted_boxes = torch.gather(boxes, 0, indexes.expand(-1, 4))

            detections = {
                "pred_logits": sorted_confidences.unsqueeze(0),
                "pred_boxes": sorted_boxes.unsqueeze(0),
            }
        return detections

Maybe it's time to migrate to PlayTorch SDK JavaScript API.

[raedle]

@SomaKishimoto, we found the culprit. After upgrading the LibTorch-Lite library from version 1.10.0 to 1.12.0, the kind support changed to c10::DynamicType. Unfortunately, the unit tests didn't catch this. We have a fix in #99. When we merged the change, it will be available in the react-native-pytorch-core@nightly or the next planned release 0.2.1.

Alternatively, as mentioned, it might be worth upgrading to the new PlayTorch SDK JavaScript API. The useObjectDetection.ts in the repo should be compatible with your model's interface (according to the live spec).

You'd have to change the mean and std for the normalize function in the packFn (see code).

// Normalize the tensor image with mean and standard deviation
const normalize = T.normalize([0.0, 0.0, 0.0], [0.0039216, 0.0039216, 0.0039216]);
tensor = normalize(tensor);

The scaleWidth and scaleHeight can be set in the resize function in the packFn (see code)

// Resize the image tensor to 3 x min(height, width)
const resize = T.resize([scaleHeight, scaleWidth]);
tensor = resize(tensor);

In the unpackFn, use the probabilityThreshold and resolve the classes (see code)

Let me know if you have question or need advice!

[SomaKishimoto]

@raedle , thanks for resolving the issue! I've verified that your fix works fine.

I will try to upgrade to the new PlayTorch SDK JavaScript API by referring to your explanation and let you know if I get stuck.

Thank you always!

[raedle]

Fantastic to hear that the issue is resolved, @SomaKishimoto! I'll close the issue, then