ProMerge is a fundamental improvement over prior leading approaches to unsupervised instance segmentation and object detection. For the challenging SA1B benchmark, we observe a 41.8% improvement in AP and 20.6% improvement in AR compared to the CutLER SOTA. We observe increases in AR and AP across six benchmarks with the pseudolabel generation process and the downstream trained class agnostic detector.
Official PyTorch implementation for ProMerge (ECCV'24). Details can be found in the paper.
ProMerge: Prompt and Merge for Unsupervised Instance Segmentation
Dylan Li, Gyungin Shin
Meta Reality Labs, Oxford Visual Geometry Group
[paper], [project page]
- Prior leading unsupervised methods rely on repeatedly solving graph partitioning over the global context of the image in feature space. These methods miss smaller objects that are primarily local in context.
- We propose ProMerge (and the downstream ProMerge+), which generates a large number of masks per image by obviating the resolution of the generalized eigenvalue problem. ProMerge lifts precision/recall on diverse datasets while reducing pseudolabel generation time.
- We use the CutLER training recipe for a single round and show that training a MaskRCNN detector on the high-quality ProMerge pseudo-labels results in SOTA performance on six diverse benchmarks. ProMerge/Promerge+ and their attendant performance gains in AP/AR relative to CutLER/MaskCut are shown below.
| COCO2017 | COCO20K | LVIS | KITTI | Objects365 | SA-1B | |
|---|---|---|---|---|---|---|
| MaskCut [training-free] | 2.2 | 3.0 | 0.9 | 0.2 | 1.7 | 0.8 |
| ProMerge [training-free] | 2.4 | 3.0 | 1.3 | 0.3 | 2.2 | 1.2 |
| CutLER | 8.7 | 8.9 | 3.4 | 3.9 | 11.5 | 5.5 |
| ProMerge+ | 8.9 | 9.0 | 4.0 | 5.4 | 12.2 | 7.8 |
- ProMerge relies on a relatively simple procedure of prompting and merging masks. A brief procedural diagram is shown below.
To be updated.
Please download datasets and their annotation files:
- COCO2017 [
annotation file] - COCO-20K [
annotation file] - LVIS [
annotation file] - KITTI [
annotation file] - Objects365 [
annotation file] - SA-1B [
annotation file]
To be uploaded.
The majority of ProMerge, Detectron2 and DINO are licensed under the CC-BY-NC license. However portions of the project are available under separate license terms. CRF is licensed under the MIT license. If you later add other third party code, please keep this license info updated, and please let us know if that component is licensed under something other than CC-BY-NC, MIT, or CC0.
@inproceedings{li2024promerge,
title = {ProMerge: Prompt and Merge for Unsupervised Instance Segmentation},
author = {Li, Dylan and Shin, Gyungin},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2024}
}
If you have any questions about our code/implementation, please contact us at gyungin [at] robots [dot] ox [dot] ac [dot] uk.