Question about pad_shape

[dingguo1996]

I make a custom datasets and put it into train ,but I got error as following:

Traceback (most recent call last):
  File "/home/siting/files/DingGuo/mmdetection/tools/train.py", line 90, in <module>
    main()
  File "/home/siting/files/DingGuo/mmdetection/tools/train.py", line 86, in main
    logger=logger)
  File "/home/siting/files/DingGuo/mmdetection/mmdet/apis/train.py", line 59, in train_detector
    _non_dist_train(model, dataset, cfg, validate=validate)
  File "/home/siting/files/DingGuo/mmdetection/mmdet/apis/train.py", line 121, in _non_dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/siting/.conda/envs/maskrcnn_benchmark/lib/python3.7/site-packages/mmcv/runner/runner.py", line 355, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/siting/.conda/envs/maskrcnn_benchmark/lib/python3.7/site-packages/mmcv/runner/runner.py", line 257, in train
    for i, data_batch in enumerate(data_loader):
  File "/home/siting/.conda/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 513, in __next__
    return self._process_next_batch(batch)
  File "/home/siting/.conda/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 534, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
  File "/home/siting/.conda/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/siting/.conda/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/siting/files/DingGuo/mmdetection/mmdet/datasets/custom.py", line 169, in __getitem__
    data = self.prepare_train_img(idx)
  File "/home/siting/files/DingGuo/mmdetection/mmdet/datasets/custom.py", line 240, in prepare_train_img
    scale_factor, flip)
  File "/home/siting/files/DingGuo/mmdetection/mmdet/datasets/transforms.py", line 109, in __call__
    mmcv.impad(mask, pad_shape[:2], pad_val=0) for mask in masks
  File "/home/siting/files/DingGuo/mmdetection/mmdet/datasets/transforms.py", line 109, in <listcomp>
    mmcv.impad(mask, pad_shape[:2], pad_val=0) for mask in masks
  File "/home/siting/.conda/envs/maskrcnn_benchmark/lib/python3.7/site-packages/mmcv/image/transforms/geometry.py", line 186, in impad
    pad[:img.shape[0], :img.shape[1], ...] = img
ValueError: could not broadcast input array from shape (1094,1604) into shape (1094,1600)

Actually, I put my dataset into detectron or maskbenchmark and it works well, but it doesn't work here. I think it's the problem in mask_pad

[jichilen]

This maybe a problem about mmcv.imread(), in which cv2.imread() is used. In some situation, the image is taken by ceilphones and a rotation angle is saved in the image. However the image will not be rotated when using cv2.imread(). But when annotating a image, the rotated image is used.
So you may change
img = mmcv.imread(osp.join(self.img_prefix, img_info['filename']))
to
img = np.array(Image.open(osp.join(self.img_prefix, img_info['filename'])).convert("RGB"))
img = img[...,::-1]
in custom.py
maybe this will help you

[dingguo1996]

It still doesn't work, I think the problem is attributed to padding the masks which makes the size mismatch with img

[jichilen]

It still doesn't work, I think the problem is attributed to padding the masks which makes the size mismatch with img

you are right, it is the problem of shape mismatch between orign-image and mask-image which is generated from the annotation

Do you use python setup.py install to insert your edited code to site-packages?

[dingguo1996]

@jichilen I use python setup.py install again ,but it doesn't wok. Do you have the another way to fix it?

[dingguo1996]

AssertionError: Traceback (most recent call last): File "/home/siting/.conda/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop samples = collate_fn([dataset[i] for i in batch_indices]) File "/home/siting/.conda/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp> samples = collate_fn([dataset[i] for i in batch_indices]) File "/home/siting/files/DingGuo/mmdetection/mmdet/datasets/custom.py", line 169, in __getitem__ data = self.prepare_train_img(idx) File "/home/siting/files/DingGuo/mmdetection/mmdet/datasets/custom.py", line 242, in prepare_train_img scale_factor, flip) File "/home/siting/files/DingGuo/mmdetection/mmdet/datasets/transforms.py", line 109, in __call__ mmcv.impad(mask, pad_shape[:2], pad_val=0) for mask in masks File "/home/siting/files/DingGuo/mmdetection/mmdet/datasets/transforms.py", line 109, in <listcomp> mmcv.impad(mask, pad_shape[:2], pad_val=0) for mask in masks File "/home/siting/.conda/envs/maskrcnn_benchmark/lib/python3.7/site-packages/mmcv/image/transforms/geometry.py", line 183, in impad assert shape[i] >= img.shape[i]

Now the error shows me the above information.

[dingguo1996]

Do you know why the mmdet need to pad the img size to multiple of 32?

[jichilen]

Maybe you have an image with the shape(A,B,3), but in your annotation it has the shape of (B,A,3)

[jichilen]

If your data is in the form of coco_type, you may print the img_info and img.shape of it, once you find the exact image that with the confused annotation you can do some visualization on it, maybe this will help you find what is happening

[jichilen]

Do you know why the mmdet need to pad the img size to multiple of 32?

this is because the backbone may downsample the features to the size of [N,C,1/32H,1/32w] by stride=2 in convolution layers, this will avoid confusion in output_size calculation

[dingguo1996]

@jichilen OK, I see it. On the other hand, I try it and visualize them with error, but it seems to normal. For the same annotation, detectron and maskrcnn benchmark can accept it without any problems.

[dingguo1996]

@jichilen Could you use QQ or WeChat ? Could I add you to learn from you about this mmdet in detail?

[jichilen]

Here is an example in my dataset

In [1]: import cv2

In [2]: from PIL import Image

In [3]: im=cv2.imread('/data2/data/ART/vis/gt_1458.jpg')

In [4]: img=Image.open('/data2/data/ART/vis/gt_1458.jpg')

In [5]: im.shape
Out[5]: (1200, 1600, 3)

In [6]: img.size
Out[6]: (1600, 1200)

In [7]: im=cv2.imread('/data2/data/ART/train_images/gt_1458.jpg')

In [8]: img=Image.open('/data2/data/ART/train_images/gt_1458.jpg')

In [9]: im.shape
Out[9]: (1600, 1200, 3)

In [10]: img.size
Out[10]: (1600, 1200)

in cv2.img, it has the shape of (h,w,c), but in Image.img, it has the shape of (w,h)
the origin images are saved in dir train_images
different result are in (h,w) of the image

[dingguo1996]

Thank you for your patient reply. I have found the problem, after checking the shape between origin images and train_images. I will close the issue!

[lzhbrian]

@qq237942920 Hi, could you provide the way you solved this problem? I am having the same error.
Thanks

[dingguo1996]

I just check the annotations and images correspondence.

[my-hello-world]

@qq237942920 ，How to make comments and images correspond？

[dingguo1996]

@qq237942920 ，How to make comments and images correspond？

I just compare the images['img_shape'] in my annotations with the real images shape. Furthermore, maybe you should find out which images cause the problem in mmdet and then you can check it according the 'file_name'

[my-hello-world]

@qq237942920 ，How to make comments and images correspond？

I just compare the images['img_shape'] in my annotations with the real images shape. Furthermore, maybe you should find out which images cause the problem in mmdet and then you can check it according the 'file_name'

I use coco2014 datasets with instances_minival2014.json but I got the same error.
I just made the following changes：

data = dict(
    imgs_per_gpu=2,
    workers_per_gpu=2,
    train=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_minival2014.json',
        img_prefix=data_root + 'val2014/',
        # (long_edge, short_edge) or [(long1, short1), (long2, short2), ...]
        img_scale=(1333, 800),#
        img_norm_cfg=img_norm_cfg,
        size_divisor=32,
        flip_ratio=0.5,
        with_mask=True,
        with_crowd=True,
        with_label=True),
    val=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_minival2014.json',
        img_prefix=data_root + 'val2014/',
        img_scale=(1333, 800),
        img_norm_cfg=img_norm_cfg,
        size_divisor=32,
        flip_ratio=0,
        with_mask=True,
        with_crowd=True,
        with_label=True),
    test=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_minival2014.json',
        img_prefix=data_root + 'val2014/',
        img_scale=(1333, 800),
        img_norm_cfg=img_norm_cfg,
        size_divisor=32,
        flip_ratio=0,
        with_mask=False,
        with_label=False,
        test_mode=True))

[hellock]

This is most probable that the image shape in annotations is not the same as the actual shape of the loaded image. Some images may be taken by mobile devices and have different orientation. You may have a check.

[my-hello-world]

This is most probable that the image shape in annotations is not the same as the actual shape of the loaded image. Some images may be taken by mobile devices and have different orientation. You may have a check.

tks.
I checked it, but there is no progress, I think coco2014 is the same form as coco2017, right? For hardware reasons, I did this:
https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/configs/quick_schedules/e2e_faster_rcnn_R_50_FPN_quick.yaml

it just change the datasets like：

DATASETS:
  TRAIN: ("coco_2014_train", "coco_2014_valminusminival")
  TEST: ("coco_2014_minival",) 

to following：

DATASETS:
  TRAIN: ("coco_2014_minival",)
  TEST: ("coco_2014_minival",)

any advice？pls.

[dingguo1996]

This is most probable that the image shape in annotations is not the same as the actual shape of the loaded image. Some images may be taken by mobile devices and have different orientation. You may have a check.

tks.
I checked it, but there is no progress, I think coco2014 is the same form as coco2017, right? For hardware reasons, I did this:
https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/configs/quick_schedules/e2e_faster_rcnn_R_50_FPN_quick.yaml

it just change the datasets like：

DATASETS:
  TRAIN: ("coco_2014_train", "coco_2014_valminusminival")
  TEST: ("coco_2014_minival",) 

to following：

DATASETS:
  TRAIN: ("coco_2014_minival",)
  TEST: ("coco_2014_minival",)

any advice？pls.

I think you can try with coco2017 which I'm using. If you use the official dataset ,the annotations they open should right for the dataset. And you can try this format as follow:

DATASETS:
  TRAIN: ("coco_2017_train",)
  TEST: ("coco_2017_val",)

[shriramhr]

@qq237942920 ，How to make comments and images correspond？

I just compare the images['img_shape'] in my annotations with the real images shape. Furthermore, maybe you should find out which images cause the problem in mmdet and then you can check it according the 'file_name'

Nihao!

Can you please tell me how did you compate img_shape with real image shape? I am not able to correctly tell which image it failing on. Can you please tell me the exact debugging method you tried please?

[jichilen]

Maybe you can set batch size to 1, comment the code of the whole network(only the image processing
is left) ,and let the code works in a try block and throw an error with image name.