Strange behavior in detection reference code

[rvandeghen]

🐛 Describe the bug

Hi,

I am using your training procedure for object detection https://github.com/pytorch/vision/blob/main/references/detection/train.py with a custom dataset. When I evaluate my model, the output seems correct for the very first epoch but for the following epochs, the metrics fall to 0.
However, this is not related to the model performance as I can use a checkpoint and evaluate it in another process, which gives back expected values.

From the code, the difference between COCO dataset and a custom dataset happens here:

vision/references/detection/coco_utils.py

Line 203 in f4fd193

return convert_to_coco_api(dataset)

I suppose that the current behavior is not expected. Have you ever faced a similar issue and how can I correct it ?

Renaud

Versions

Collecting environment information...
PyTorch version: 1.10.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: CentOS Linux release 8.2.2004 (Core) (x86_64)
GCC version: (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.28

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.18.0-193.6.3.el8_2.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
Nvidia driver version: 450.57
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.10.1
[pip3] torchvision==0.11.2
[conda] blas 1.0 mkl
[conda] cudatoolkit 11.3.1 h2bc3f7f_2
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] mkl 2021.4.0 h06a4308_640
[conda] mkl-service 2.4.0 py39h7f8727e_0
[conda] mkl_fft 1.3.1 py39hd3c417c_0
[conda] mkl_random 1.2.2 py39h51133e4_0
[conda] numpy 1.21.2 py39h20f2e39_0
[conda] numpy-base 1.21.2 py39h79a1101_0
[conda] pytorch 1.10.1 py3.9_cuda11.3_cudnn8.2.0_0 pytorch
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torchvision 0.11.2 py39_cu113 pytorch

cc @datumbox

[rvandeghen]

Solved it by adding a copy of the inital dataset object in the convert_to_coco_api function, which gives the following:

def convert_to_coco_api(ds):
    ds_copy = copy.deepcopy(ds) # Create copy here
    coco_ds = COCO()
    # annotation IDs need to start at 1, not 0, see torchvision issue #1530
    ann_id = 1
    dataset = {'images': [], 'categories': [], 'annotations': []}
    categories = set()
    for img_idx in range(len(ds_copy)):
        # find better way to get target
        # targets = ds.get_annotations(img_idx)
        img, targets = ds_copy[img_idx]
        image_id = targets["image_id"].item()
        img_dict = {}
        img_dict['id'] = image_id
        img_dict['height'] = img.shape[-2]
        img_dict['width'] = img.shape[-1]
        dataset['images'].append(img_dict)
        bboxes = targets["boxes"]
        bboxes[:, 2:] -= bboxes[:, :2]
        bboxes = bboxes.tolist()
        labels = targets['labels'].tolist()
        areas = targets['area'].tolist()
        iscrowd = targets['iscrowd'].tolist()
        if 'masks' in targets:
            masks = targets['masks']
            # make masks Fortran contiguous for coco_mask
            masks = masks.permute(0, 2, 1).contiguous().permute(0, 2, 1)
        if 'keypoints' in targets:
            keypoints = targets['keypoints']
            keypoints = keypoints.reshape(keypoints.shape[0], -1).tolist()
        num_objs = len(bboxes)
        for i in range(num_objs):
            ann = {}
            ann['image_id'] = image_id
            ann['bbox'] = bboxes[i]
            ann['category_id'] = labels[i]
            categories.add(labels[i])
            ann['area'] = areas[i]
            ann['iscrowd'] = iscrowd[i]
            ann['id'] = ann_id
            if 'masks' in targets:
                ann["segmentation"] = coco_mask.encode(masks[i].numpy())
            if 'keypoints' in targets:
                ann['keypoints'] = keypoints[i]
                ann['num_keypoints'] = sum(k != 0 for k in keypoints[i][2::3])
            dataset['annotations'].append(ann)
            ann_id += 1
    dataset['categories'] = [{'id': i} for i in sorted(categories)]
    coco_ds.dataset = dataset
    coco_ds.createIndex()
    return coco_ds

@datumbox do you think it is useful to make a PR with this change to solve this issue ?

[datumbox]

@rvandeghen Thanks for raising this.

It is very difficult to tell what's the problem given that I can't reproduce the issue without having your custom dataset. What's unclear to me is why you need to deep-copy the ds given you don't modify it.

do you think it is useful to make a PR with this change to solve this issue ?

The reference script works fine with Coco which is its intended use-case. Moreover the script serves as a starting point for how one can build their own loops. With the info I have at this point, I don't think applying the deep-copy patch in the general case is worth it.

[rvandeghen]

@datumbox I tested it with fake data and it produces the same behavior. You can try it by yourself.

import torch
import torchvision
from pycocotools.coco import COCO

class MyDataset(torch.utils.data.Dataset):
    
    def __init__(self, size):
        
        self.size = size
        
        self.targets = list()
        
        self.images = torch.rand((self.size, 224, 224))
        
        for k in range(self.size):
            boxes = list()
            labels = list()
            areas = list()
            iscrowds = list()
            for b in range(torch.randint(1, 5, (1,)).item()):
                x1 = torch.randint(100, (1,)).item()
                y1 = torch.randint(100, (1,)).item()
                x2 = torch.randint(101, 200, (1,)).item()
                y2 = torch.randint(101, 200, (1,)).item()
                boxes.append((x1, y1, x2, y2))
                labels.append(torch.randint(3, (1,)).item())
                a = (x2 - x1)*(y2 - y1)
                areas.append(a)
                iscrowds.append(0)
            image_id = k
            self.targets.append({'boxes': torch.tensor(boxes),
                                 'labels': torch.tensor(labels),
                                 'image_id': torch.tensor(image_id),
                                 'area': torch.tensor(areas),
                                 'iscrowd': torch.tensor(iscrowds)})
        

    def __len__(self,):
        return self.size
    
    def __getitem__(self, idx):
        
        image = self.images[idx]
        targets = self.targets[idx]
        
        return image, targets

ds = MyDataset(10)

print(ds.__getitem__(0))
>>> (tensor([[0.0886, 0.9420, 0.6616,  ..., 0.3945, 0.5109, 0.5262],
         [0.8148, 0.2545, 0.0768,  ..., 0.8070, 0.3861, 0.0709],
         [0.8251, 0.4180, 0.5144,  ..., 0.9156, 0.5318, 0.1327],
         ...,
         [0.6207, 0.3496, 0.5851,  ..., 0.2702, 0.8043, 0.7516],
         [0.0099, 0.3456, 0.4506,  ..., 0.7782, 0.9289, 0.6935],
         [0.5059, 0.7233, 0.4494,  ..., 0.3241, 0.9573, 0.3146]]),
 {'boxes': tensor([[ 23,   3, 159, 187],
          [ 30,  61, 117, 180]]),
  'labels': tensor([0, 0]),
  'image_id': tensor(0),
  'area': tensor([25024, 10353]),
  'iscrowd': tensor([0, 0])})

coco = get_coco_api_from_dataset(ds)
print(ds.__getitem__(0))
>>> (tensor([[0.0886, 0.9420, 0.6616,  ..., 0.3945, 0.5109, 0.5262],
         [0.8148, 0.2545, 0.0768,  ..., 0.8070, 0.3861, 0.0709],
         [0.8251, 0.4180, 0.5144,  ..., 0.9156, 0.5318, 0.1327],
         ...,
         [0.6207, 0.3496, 0.5851,  ..., 0.2702, 0.8043, 0.7516],
         [0.0099, 0.3456, 0.4506,  ..., 0.7782, 0.9289, 0.6935],
         [0.5059, 0.7233, 0.4494,  ..., 0.3241, 0.9573, 0.3146]]),
 {'boxes': tensor([[ 23,   3, 136, 184],
          [ 30,  61,  87, 119]]),
  'labels': tensor([0, 0]),
  'image_id': tensor(0),
  'area': tensor([25024, 10353]),
  'iscrowd': tensor([0, 0])})

As you can see, the last 2 values of the boxes are changed in place with respect to the Dataset data. Maybe something is wrong with my Dataset class but I still managed to counter this issue with the deepcopy.

I totally agree that the reference script is a starting point but I suppose that if the problem comes from the function convert_to_coco_api it can still be solved in the script. Let me know if you have the same issue with the fake data and the best solution to solve it.

[datumbox]

@rvandeghen Thanks for providing an example that reproduces clearly the problem. This helps a lot.

The problem seems to be due to the inplace modification that happens at:
bboxes[:, 2:] -= bboxes[:, :2]

To avoid this we should clone that target boxes to avoid modifying the original dataset:
bboxes = targets["boxes"].clone()

From what I see, no other record is modified inplace but it's worth confirming. Could you please apply the patch and let me know if it works for you? If it does, I'm happy to review a PR that fixes the issue.

[rvandeghen]

@datumbox Indeed the problem comes from bboxes[:, 2:] -= bboxes[:, :2] and is solved by cloning it. I also checked that the masks were not changed with the current version and it works fine.