Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
This branch is 14 commits ahead, 2 commits behind adynathos:master.


Evaluation procedure


  • Place the datasets in ./datasets (or override with env var DIR_DATASETS)

    • dataset_ObstacleTrack
    • dataset_AnomalyTrack
    • dataset_LostAndFound (or provide location in env DIR_LAF)
    • dataset_FishyLAF (or provide location in env DIR_FISHY_LAF)
  • Run inference and store results in files. Run inference for the following splits, as the other splits are subsets of those:

    • AnomalyTrack-all
    • ObstacleTrack-all
    • LostAndFound-test
    • LostAndFound-train
import numpy as np
from tqdm import tqdm
import cv2 as cv
from road_anomaly_benchmark.evaluation import Evaluation

def method_dummy(image, **_):
	""" Very naive method: return color saturation """
	image_hsv = cv.cvtColor(image, cv.COLOR_RGB2HSV_FULL)
	anomaly_p = image_hsv[:, :, 1].astype(np.float32) * (1./255.)
	return anomaly_p

def main():

	ev = Evaluation(
		method_name = 'Dummy', 
		dataset_name = 'ObstacleTrack-all',
		# dataset_name = 'AnomalyTrack-test',

	for frame in tqdm(ev.get_frames()):
		# run method here
		result = method_dummy(frame.image)
		# provide the output for saving
		ev.save_output(frame, result)

	# wait for the background threads which are saving

The files will be stored in ./outputs/anomaly_p/.... The storage directory can be overriden with env var DIR_OUTPUTS. There are also some methods already implemented and available in

Tip: to set environment variables export ENV_VAR=VALUE.


This step will also create plots in ./outputs/{metric} (override with env var DIR_OUTPUTS)

  • Metrics for anomaly track, splits AnomalyTrack-validation, FishyLAFAnomaly-val

python -m road_anomaly_benchmark metric PixBinaryClass $methods AnomalyTrack-validation,FishyLAFAnomaly-val
python -m road_anomaly_benchmark metric SegEval-AnomalyTrack $methods AnomalyTrack-validation,FishyLAFAnomaly-val
  • Metrics for obstacle track, splits ObstacleTrack-validation, LostAndFound-testNoKnown

python -m road_anomaly_benchmark metric PixBinaryClass $methods ObstacleTrack-validation,LostAndFound-testNoKnown
python -m road_anomaly_benchmark metric SegEval-ObstacleTrack $methods ObstacleTrack-validation,LostAndFound-testNoKnown


The anomaly scores can be visualized without ground truth by running with --only-frame-vis flag.

python -m road_anomaly_benchmark metric PixBinaryClass $methods $dsets --only-frame-vis

If ground truths are available, the --frame-vis option both evaluates the metric and generates visualizations with the ROI region marked. The flag could be used for LostAndFound for example:

python -m road_anomaly_benchmark metric PixBinaryClass $methods LostAndFound-testNoKnown --frame-vis

Plots and Tables

The following code snippets for comparisons of methods with plots and tables can only be used if ground truths are available. The general syntax for comparing methods is as follows:

python -m road_anomaly_benchmark comparison MyComparison metric1,metric2 method1,method2 dset1,dset2
  • Anomaly splits: AnomalyTrack-validation, FishyLAFAnomaly-val
# Anomaly track tables
python -m road_anomaly_benchmark comparison TableAnomaly1 PixBinaryClass,SegEval-AnomalyTrack $methods_ano AnomalyTrack-validation --names names.json
python -m road_anomaly_benchmark comparison TableAnomaly2 PixBinaryClass,SegEval-AnomalyTrack $methods_ano FishyLAFAnomaly-val --names names.json
  • Obstacle splits: ObstacleTrack-validation, LostAndFound-testNoKnown
# Obstacle track tables
python -m road_anomaly_benchmark comparison TableObstacle1 PixBinaryClass,SegEval-ObstacleTrack $methods_obs ObstacleTrack-validation --names names.json
python -m road_anomaly_benchmark comparison TableObstacle2 PixBinaryClass,SegEval-ObstacleTrack $methods_obs LostAndFound-testNoKnown --names names.json


If you use this repository, please consider citing our paper:

	title={SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation},
	author={Robin Chan and Krzysztof Lis and Svenja Uhlemeyer and Hermann Blum and Sina Honari and Roland Siegwart and Pascal Fua and Mathieu Salzmann and Matthias Rottmann},
	booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},


Benchmark of detection methods for anomalies and obstacles in traffic images.






No releases published


No packages published


  • Python 79.2%
  • Jupyter Notebook 20.6%
  • Shell 0.2%