[BUG] COCO export: Instance Segmentations OOB wrt their bounding box

[allenleetc]

Instructions

Thank you for submitting an issue. Please refer to our issue policy for information on what types of issues we address.

1. Please fill in this template to ensure a timely and thorough response
2. Place an "x" between the brackets next to an option if it applies. For example:
   • Selected option
3. Please delete everything above this line before submitting the issue

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
• Python version (python --version): 3.10.10
• FiftyOne version (fiftyone --version): 0.20.0
• FiftyOne installed from (pip or source): pip

Commands to reproduce

def check_ann(ann):
    bb = ann['bbox']
    segall = ann['segmentation']

    bbxyxy = bb.copy()
    bbxyxy[2] += bbxyxy[0]
    bbxyxy[3] += bbxyxy[1]

    for seg in segall:
        seg = np.array(seg,dtype=np.float64).reshape((len(seg)//2),2)

        #print(seg1)
        orx0 = seg[:,0]<bbxyxy[0]
        orx1 = seg[:,0]>bbxyxy[2]
        ory0 = seg[:,1]<bbxyxy[1]
        ory1 = seg[:,1]>bbxyxy[3]
        if np.any(orx0):
            print(f'x0, {seg[orx0,0]} against {bbxyxy[0]}')
        if np.any(orx1):
            print(f'x1, {seg[orx1,0]} against {bbxyxy[2]}')
        if np.any(ory0):
            print(f'y0, {seg[ory0,1]} against {bbxyxy[1]}')
        if np.any(ory1):
            print(f'y1, {seg[ory1,1]} against {bbxyxy[3]}')
    
    return seg, bbxyxy

PATH = '/home/allen/tmp/export_oiv7'

ds = foz.load_zoo_dataset('open-images-v7',max_samples=15,
                          split='validation',label_types=['segmentations'])

ds.export(export_dir=PATH,dataset_type=fo.types.COCODetectionDataset)

ds2 = fo.Dataset.from_dir(dataset_dir=PATH,dataset_type=fo.types.COCODetectionDataset)

JSON = os.path.join(PATH,'labels.json')
with open(JSON) as fh:
    j0 = json.load(fh)

for aidx,ann in enumerate(j0['annotations']):
    print(aidx)
    check_ann(ann)

Output:

0
x0, [752.5] against 753.651712
1
2
3
y0, [154.5 153.5] against 154.77248
4
5
y0, [350.5 351. ] against 351.99974399999996
6
x0, [266.5 266.5] against 267.327488
y0, [166.5 167.5] against 167.727725
7
y0, [86.5] against 87.560681
8
y0, [71.5 71.5] against 72.42499
9
y0, [468.5] against 469.25214600000004
10
x0, [102.5 103.5] against 103.999488
y0, [86.5 86.5] against 87.999744

... snip ...

Describe the problem

Exporting instance segmentations via COCODetectionDataset can result in segmentations that go out of bounds with respect to their bounding box by up to ~1.5px. This may cause an issue for some training pipelines and/or represent a small loss in accuracy.

I investigated one of the original COCO datasets downloaded from cocodataset.org and this issue was not present. So round-tripping through FO (importing and then re-exporting) likely causes the slight discrepancies.

What areas of FiftyOne does this bug affect?

• App: FiftyOne application issue
• Core: Core Python library issue
• Server: FiftyOne server issue

Willingness to contribute

The FiftyOne Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the FiftyOne codebase?

• Yes. I can contribute a fix for this bug independently
• Yes. I would be willing to contribute a fix for this bug with guidance from the FiftyOne community
• No. I cannot contribute a bug fix at this time

[DavidJimenez10]

Hi, I'm a newbie in OSS and CV and I would like to contribute to this project. I made a pull request and I'm open to any feedback

[swamitagupta]

Is this issue completed or still open for contribution?

[brimoor]

Hi @DavidJimenez10 and @swamitagupta, thanks for offering to help. Yes, a contribution would be welcome! 🙌

The relevant code for importing/exporting in COCO format is in this module:
https://github.com/voxel51/fiftyone/blob/develop/fiftyone/utils/coco.py

In particular, the code that converts to COCO masks and back into FiftyOne masks is here:

fiftyone/fiftyone/utils/coco.py

Lines 2154 to 2214 in 2f5e53e

	def _coco_segmentation_to_mask(segmentation, bbox, frame_size):
	x, y, w, h = bbox
	width, height = frame_size
	if isinstance(segmentation, list):
	# Polygon -- a single object might consist of multiple parts, so merge
	# all parts into one mask RLE code
	rle = mask_utils.merge(
	mask_utils.frPyObjects(segmentation, height, width)
	)
	elif isinstance(segmentation["counts"], list):
	# Uncompressed RLE
	rle = mask_utils.frPyObjects(segmentation, height, width)
	else:
	# RLE
	rle = segmentation
	mask = mask_utils.decode(rle).astype(bool)
	return mask[
	int(round(y)) : int(round(y + h)),
	int(round(x)) : int(round(x + w)),
	]
	def _polyline_to_coco_segmentation(polyline, frame_size, iscrowd="iscrowd"):
	if polyline.get_attribute_value(iscrowd, None):
	seg = polyline.to_segmentation(frame_size=frame_size, target=1)
	return _mask_to_rle(seg.mask)
	width, height = frame_size
	polygons = []
	for points in polyline.points:
	polygon = []
	for x, y in points:
	polygon.append(int(x * width))
	polygon.append(int(y * height))
	polygons.append(polygon)
	return polygons
	def _instance_to_coco_segmentation(
	detection, frame_size, iscrowd="iscrowd", tolerance=None
	):
	dobj = foue.to_detected_object(detection, extra_attrs=False)
	try:
	mask = etai.render_instance_image(
	dobj.mask, dobj.bounding_box, frame_size
	)
	except:
	# Either mask or bounding box is too small to render
	width, height = frame_size
	mask = np.zeros((height, width), dtype=bool)
	if detection.get_attribute_value(iscrowd, None):
	return _mask_to_rle(mask)
	return _mask_to_polygons(mask, tolerance)

[adv010]

Is this issue still open?