RAAS is an anomaly segmentation research codebase built on top of Mask2Former and detectron2. It implements Maskomaly — three model variants that detect out-of-distribution objects in driving scenes using a Swin-L backbone trained on Cityscapes.
 |
 |
conda activate raas # Python 3.8, PyTorch 1.9.0, CUDA 11.1Do not pip install detectron2 separately. detectron2 must be built from source at raas/detectron2/:
cd /path/to/raas/detectron2 && pip install -e .OpenAI CLIP is required for maskomaly_id and maskomaly_ood:
pip install git+https://github.com/openai/CLIP.git
# If wrong clip package is installed: pip uninstall clip -y firstAll scripts must be run from Maskomaly/scripts/ with conda activate raas.
# Evaluation (inference + metrics in one pass)
python run_eval.py --model maskomaly --dataset fs_laf
python run_eval.py --model maskomaly_id --dataset smiyc_anomaly --debug
python run_eval.py --model maskomaly_ood --dataset smiyc_anomaly --strategy per_image
# Inference on an image folder (no ground truth)
python infer.py --model maskomaly --input /path/to/images --output /path/to/results
python infer.py --model maskomaly_id --input /path/to/images --output /path/to/results --debugOverride dataset paths at runtime to avoid editing source files:
python run_eval.py --model maskomaly --dataset fs_laf \
--input /path/to/LostAndFound \
--output /path/to/resultsThese constants appear near the top of scripts/run_eval.py and scripts/infer.py:
| Constant | Default |
|---|---|
_DETECTRON2_DIR |
/xxx/raas/detectron2 |
DEFAULT_CONFIG |
.../Mask2Former/configs/cityscapes/semantic-segmentation/swin/maskformer2_swin_large_IN21k_384_bs16_90k.yaml |
DEFAULT_WEIGHTS |
.../Maskomaly/maskomaly/ckpt/model_final_17c1ee.pkl |
DEFAULT_OUTPUT_BASE |
your_own_path |
DATASET_CONFIGS |
Per-dataset root paths |
Also sys.path.append('xxx/raas/Mask2Former') is hardcoded inside each model file (model_ori.py, model_id.py, model_ood.py) — update these too.
raas/
├── detectron2/ ← detectron2 source, built with pip install -e .
├── Mask2Former/ ← Mask2Former source
└── Maskomaly/
├── maskomaly/
│ ├── model_ori.py ← maskomaly (original, hardcoded mask indices)
│ ├── model_id.py ← maskomaly_id (road polygon + CLIP, ID prompts)
│ ├── model_ood.py ← maskomaly_ood (road polygon + CLIP, ID + OOD prompts)
│ └── datasets.py ← Dataset classes for all 5 benchmarks
├── detectron2_replacements/ ← patched DefaultPredictor
├── mask2former_replacements/ ← patched MaskFormer model
└── scripts/
├── run_eval.py ← end-to-end inference + evaluation
├── infer.py ← inference only (no ground truth)
└── eval.py ← metric functions (AUROC, AUPR, AP)
detectron2_replacements/ must appear before raas/detectron2/ in sys.path. The scripts insert it first automatically. This replaces detectron2.engine.defaults.DefaultPredictor with a patched version that returns 3 values instead of 1:
segmentation, mask_cls_result, mask_pred_result = self.model(image)
# segmentation: dict with "sem_seg" logits [C, H, W]
# mask_cls_result: [N_queries, N_classes+1] — raw logits, needs softmax
# mask_pred_result: [N_queries, H, W] — raw logits, needs sigmoidIf you see ValueError: not enough values to unpack (expected 3, got 1), the upstream detectron2 DefaultPredictor is being picked up instead of the patch.
All three models share the same BaseSegmentationModel.get_probs_and_seg() → softmax + sigmoid → numpy pipeline.
maskomaly (model_ori.py): Combines two signals: (1) high-entropy rejection — any query whose top class confidence > 0.7 suppresses its mask region; (2) anomaly promotion — queries at fixed indices [49, 31, 83, 32] contribute positively. Final score = 0.6 * rejection_mask + 0.4 * promotion_mask. Note: model_ori.py contains leftover debug writes (soft_mask.png, soft_mask2.png, all_soft_masks/ directory) that run unconditionally.
maskomaly_id (model_id.py): After computing the base soft mask, applies road-aware CLIP filtering: extracts road polygon from query mask #20, finds unmasked regions inside the polygon, crops each connected component, and classifies it with CLIP against 19 Cityscapes ID prompts. Components with ID confidence > 0.85 are suppressed (score → 0.05); others are marked anomalous (score → 1.0). CLIP model is loaded fresh per image call — there is no caching.
maskomaly_ood (model_ood.py): Same pipeline as maskomaly_id but the CLIP text prompts include additional OOD phrases ("something unusual in a driving scene", etc.). Decision logic uses combined ID+OOD probability instead of ID-only.
| Class | Dataset key | Image dir | Label dir |
|---|---|---|---|
FishyScapesLaF |
fs_laf |
original/ |
labels_masks/ |
FishyScapesStatic |
fs_static |
images/ |
labels_masks/ |
SMIYCANO |
smiyc_anomaly |
images_val/ (filtered to validation*) |
labels_masks/ |
SMIYCOBS |
smiyc_obstacle |
images_val/ (filtered to validation*) |
labels_masks/ |
RoadAnomaly |
roadanomaly |
original/ |
labels/ |
All datasets return (image, anomaly_gt, ignore, filename). Anomaly ground truth uses label == 1; ignore/void uses label == 255 (dataset-specific).
get_scores(ground_truths, anomaly_probs, ignores, mode) returns (AP, AUROC, FPR@95, AUPR). mode="accumulate" (default) flattens all images together before scoring; mode="image" averages per-image scores.
We thank the authors of the following codebases, which this repository builds upon:
- Maskomaly — anomaly segmentation with Mask2Former
- Mask2Former — universal image segmentation
- detectron2 — object detection and segmentation framework
If you find our work useful please cite our paper:
@misc{yan2026roadawareanomalysegmentationqueryguided,
title={Road-Aware Anomaly Segmentation with Query-Guided Polygons and CLIP in Autonomous Driving},
author={Zhiran Yan and Gordon Elger},
year={2026},
eprint={2607.04304},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2607.04304},
}