Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable support for RLE encoded segmentations #44

Open
mjkvaak opened this issue Jun 16, 2020 · 7 comments
Open

Enable support for RLE encoded segmentations #44

mjkvaak opened this issue Jun 16, 2020 · 7 comments

Comments

@mjkvaak
Copy link

mjkvaak commented Jun 16, 2020

Hi there! Thanks of the amazing repository.

I tried training my model with custom data (following detectron2 guidelines) with annotation masks encoded in RLE. I also used cfg.INPUT.MASK_FORMAT='bitmask'. All the detectron2 magic works - loading the data, building the model etc. - but the code crashes when the model first tries to evaluate the losses. Attached is (the relevant part of) the error message, from which I would deduce that somewhere in the code a conversion of masks from Bitmasks to Polygons would be needed.

Traceback (most recent call last):
 ...

  File "../centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "../centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "../centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'
@Schneey
Copy link

Schneey commented Jun 26, 2020

Hi there! Thanks of the amazing repository.

I tried training my model with custom data (following detectron2 guidelines) with annotation masks encoded in RLE. I also used cfg.INPUT.MASK_FORMAT='bitmask'. All the detectron2 magic works - loading the data, building the model etc. - but the code crashes when the model first tries to evaluate the losses. Attached is (the relevant part of) the error message, from which I would deduce that somewhere in the code a conversion of masks from Bitmasks to Polygons would be needed.

Traceback (most recent call last):
 ...

  File "../centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "../centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "../centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'

I met the same problem as you. Have you solved it now?

@mjkvaak
Copy link
Author

mjkvaak commented Jun 27, 2020

Unfortunately no: I had to switch from RLE to polygons. Here's a link, in case you need to make COCO style polygon annotations yourself out of segmentation masks: https://www.immersivelimit.com/tutorials/create-coco-annotations-from-scratch. However, I should mention that when simplifying the polygons' geometries with shapely, instead of using preserve_topology = False mentioned in the blog, one should prefer preserve_topology = True . The point is that in the former case a Polygon shape may get split into multiple polygons, a Multipolygon after the simplification. Then, a Multipolygon doesn't have the method for extracting the border points and the script will crash.

@fshamsafar
Copy link

Hi there! Thanks of the amazing repository.

I tried training my model with custom data (following detectron2 guidelines) with annotation masks encoded in RLE. I also used cfg.INPUT.MASK_FORMAT='bitmask'. All the detectron2 magic works - loading the data, building the model etc. - but the code crashes when the model first tries to evaluate the losses. Attached is (the relevant part of) the error message, from which I would deduce that somewhere in the code a conversion of masks from Bitmasks to Polygons would be needed.

Traceback (most recent call last):
 ...

  File "../centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "../centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "../centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'

I also want to use bitmasks; the polygon annotations are tricky when simplifying using shapely, and also occupy larger space. In order to use bitmasks, one needs to modify mask_head.py. Somewhere the class BitMasks should be used as mentioned in facebookresearch/detectron2#200 (comment) . However, I could not figure it out. Any help would be appreciated. Thanks.

@Paragjain10
Copy link

Paragjain10 commented Jan 13, 2021

Hello @miasch @mjkvaak @Schneey ,

I am facing the same problem. how did yall solve the problem?

I am trying to implement CenterMask2 https://github.com/youngwanLEE/centermask2 which makes use of Detectron2.
My annotations are in COCO RLE format, my data registration is successful. I visualized the data as well using the detectron2 visualizer, also I have set cfg.INPUT.MASk_FORMAT= bitmask, but somehow this error is being raised while I try to train the network.

Can somebody help me out here with what could be done?

/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/data/detection_utils.py:414: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/tensor_numpy.cpp:141.)
  torch.stack([torch.from_numpy(np.ascontiguousarray(x)) for x in masks])
/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/fcos/fcos_outputs.py:402: UserWarning: This overload of nonzero is deprecated:
	nonzero()
Consider using one of the following signatures instead:
	nonzero(*, bool as_tuple) (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/python_arg_parser.cpp:882.)
  per_candidate_nonzeros = per_candidate_inds.nonzero()
Traceback (most recent call last):
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 291, in <module>
    launch(
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/engine/launch.py", line 62, in launch
    main_func(*args)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 285, in main
    return trainer.train()
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 162, in train
    self.train_loop(self.start_iter, self.max_iter)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 151, in train_loop
    self.run_step()
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/engine/defaults.py", line 423, in run_step
    self._trainer.run_step()
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 228, in run_step
    loss_dict = self.model(data)
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 166, in forward
    _, detector_losses = self.roi_heads(images, features, proposals, gt_instances)
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'

@mjkvaak
Copy link
Author

mjkvaak commented Jan 13, 2021

If i remember correctly, I had the exact same error. I didn't ever put a lot of effort to try to solve problem, but switched to using conventional polygon annotations instead (see above).

@Paragjain10
Copy link

Hello @mjkvaak,
Unfortunately, the application I am working on is such that I cannot use polygons as a mask. I have to use Rle segmentation mask only. Is there any other solution to this problem?

Is it possible to train Cenetrmask2 with Rle segmentation annotations?

@15024287710Jackson
Copy link

If MASKIOU_ON = True, the error will come up. If MASKIOU_ON = False, the error will not take place.

cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)

The MASKIOU only can be trained with polygons instead of RLE format, but my dataset only has RLE format in segmentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants