U3HS finds unknowns as highly uncertain regions and clusters their corresponding instance-aware embeddings into individual objects. For the first time in panoptic segmentation with unknown objects, U3HS is trained without unknown categories, reducing assumptions, simplifying the training data collection, and leaving the settings as unconstrained as in real-life scenarios.
This is the Pytorch re-implementation of the paper Segmenting Known Objects and Unseen Unknowns without Prior Knowledge based on Detectron2.
- This is a re-implementation of U3HS framework, not the official code, it is not guaranteed to reproduce all numbers in the paper, please refer to the original numbers from Segmenting Known Objects and Unseen Unknowns without Prior Knowledge when making comparison.
See installation instructions.
See Preparing Datasets for U3HS.
See Getting Started with U3HS.
| Method | Lost&Found | open Cityscapes | closed Cityscapes | download | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PQ | RQ | SQ | PQ | RQ | SQ | PQ | RQ | SQ | ||
| U3HS-Paper | 7.94 | 12.37 | 64.24 | 41.21 | 51.67 | 79.77 | 46.53 | 58.99 | 78.87 | |
| U3HS-reimplemented | 10.000 | 14.676 | 68.141 | 40.671 | 51.509 | 74.708 | 49.489 | 62.387 | 76.973 | model | metrics |
If you find this code helpful in your research or wish to refer to the baseline results, please use the following BibTeX entry.
@inproceedings{gasperini2023segmenting,
title={Segmenting known objects and unseen unknowns without prior knowledge},
author={Gasperini, Stefano and Marcos-Ramiro, Alvaro and Schmidt, Michael and Navab, Nassir and Busam, Benjamin and Tombari, Federico},
booktitle={Proceedings of the IEEE/CVF international conference on computer vision},
pages={19321--19332},
year={2023}
}We have used utility functions from other wonderful open-source projects, we would espeicially thank the authors of: