COCO RLE support?

[csbrown-noaa]

Describe the bug
VIAME COCO segmentation output not viewable in DIVE

To Reproduce
@brenda-tran-noaa do you have more info here?

Related data
This seems to be related to Run Length Encoding. It's not implemented in DIVE, but seems to be a valid output format from VIAME.

Desktop (please complete the following information):

• OS: any
• Browser: any

Additional context
NOAA NMFS is trying to coalesce around a common data format, so that all of our data will be interoperable. COCO/KWCOCO seems like a good fit for this, because COCO is already pretty widely supported in the ML community, and KWCOCO is an obvious extension of this format to video. To this end, we're trying to move all of our data input and model output into this format, no matter which model or annotation tool it comes from, so that we can mix, match, merge data and also easily use it in DIVE.

[csbrown-noaa]

The official implementation in kitware/kwcoco suggests that RLE is valid KWCOCO

Looks like the kwcoco python lib calls out to another library kwimage for RLE support

Not super related, but the original MSCOCO python libraries have RLE support as well

This page has an okay description of the distinction in the polygon/RLE format.

The original MSCOCO spec has some more info as well, but it's not very good IMHO

[BrendaTran]

Bug Description
When testing a model built using the viame_gpu_vertex_ai.docker image on an MP4 video file, the generated COCO output uses Run-Length Encoding (RLE) for the segmentation object.
Note: VIAME_PIPELINE=pipelines/detector_gmm_motion.pipe
Importing this generated file into DIVE throws an exception. The error originates here in the DIVE codebase: dive_utils/serializers/kwcoco.py#L92.
For example:
Actual behavior (Model Output)
"annotations": [
{
"segmentation": {
"counts": [1245, 5, 23, 10, 554, 8],
"size": [480, 640]
}
}
]
Expected Behavior (DIVE Expected Output)
DIVE expects the kwcoco object format, utilizing polygon coordinates rather than RLE:
"annotations": [
{
"segmentation": [[100, 200, 150, 200, 150, 230, 100, 230]]
}
]

cc: @kelseylmartin

[BryonLewis]

I didn't write in RLE support in DIVE because DIVE currently doesn't support segmentation masks. We have boxes/polygons/lines. I would have to write something that would attempt to convert the segmentation masks into polygons. Or I would have to import the functionality from DIVE-DSA (the other version of DIVE) for segmentation masks. While I've done it before it is definitely a bit of a lift to make sure the system can support masks.

Options:

1. Attempt to convert RLE masks into polygons for the annotations
   • Risks:
     • Polygons with holes aren't currently supported
     • Broken up polygons (segementations that aren't continuous) would need some logic to consider how to handle them
     • There would be an additional lag to import time because I would have to do these conversions
   • Benefits:
     • Could be done quicker than porting over the mask segmentation support in DIVE-DSA
2. Import Segmentation Masks from DIVE-DSA into DIVE
   • Risks:
     • This could take some time because the segmentation feature is a fairly major additional add
     • It's difficult to render back segmentations in real-time at full playback speed within the viewer. Rendering a couple dozen frames is easy, but playback at 30fps while generating the masks can be tricky.

• Benefits:
  • Would provide masks and mask editor UI to the system.

[csbrown-noaa]

It may also make sense to simply throw a warning instead of an error, for now. I think that sometimes COCO-format files have segmentations AND bounding boxes, so showing the bounding boxes even though the segmentation isn't working would be useful. I'm not actually familiar with the use case here, or the specific model we're using or who needs it.

[BryonLewis]

I created a PR: #1675 that will warn on RLE + bbox importing and will fail on RLE only importing

A demo video for web and desktop is included below:

DIVE-20260601-COCO-RLE-Web_small.mp4

DIVE-20260601-COCO-RLE-Desktop.mp4

[BryonLewis]

I've merged these changes into main, I'll do a deploy to viame.kitware.com later this week to update the server.

[BrendaTran]

Hi @BryonLewis, thanks for the update! We are looking forward to getting the new kitware/viame-web and kitware/viame-worker Docker images. Could you let us know when the updated image might be available to pull, and if there are any specific tags or steps we should look out for to grab this release?

[BryonLewis]

The updated images should be available to pull from Docker Hub. A docker compose pull girder_worker_training girder_worker_default girder should pull the new images for running the system.

This is still Girder-5 upgrade so if you haven't updated yet (I think there may have been some issues related to importing redis) it will require the upgrade. More details for Girder 5 Upgrade can be found in the Girder 5 Upgrade Documentation

There are 3 main images
kitware/viame-worker:cpu - Used for the ffmpeg transcoding and other non GPU based tasks. This is a fairly large container because it contains all of VIAME and CUDA support.
kitware/viame-worker:gpu - Used for Training/pipeline running
kitware/viame-web:latest - The Girder server and front-end client code

[BrendaTran]

@BryonLewis, quick question regarding the web demo: Will the RLE warning dialogs also trigger if we import via Girder Assetstore -> GCS Import? Our primary workflow relies on bulk-importing folders (media + JSON outputs) directly from Google Cloud Storage via Girder, rather than uploading through the DIVE viewer UI.

[BryonLewis]

Currently, they won't be shown because the postprocess is called within a worker manager. I could probably update it so that the main import script would get back warnings and show them

On my last sample import, this is the job that would start other jobs so I could possibly inject warnings inside of here: