Threshold Calculation for Image and Pixel Anomalies

[weristeddy]

Describe the bug

Hello everyone,

I've been working with the anomalib recently and have encountered a question regarding the threshold calculation for image and pixel anomalies.

Specifically, I'm curious about how the anomalib determines the thresholds for image and pixel anomalies. Does it follow a specific method, such as choosing the point at the most left corner of the ROC curve, or does it employ another approach?

Furthermore, I'm interested in understanding whether the thresholds for image and pixel anomalies differ in their calculation and results.

Any insights or information about the internal workings of Anomalib's thresholding mechanism would be greatly appreciated. I want to ensure that I have a clear understanding of how these thresholds are determined to make informed decisions in my anomaly detection tasks.

Thank you in advance for your assistance

Best regards

Dataset

MVTec

Model

PatchCore

Steps to reproduce the behavior

see config file

OS information

/

Expected behavior

/

Screenshots

No response

Pip/GitHub

GitHub

What version/branch did you use?

main

Configuration YAML

dataset:
  name: toothbrush
  format: folder
  path: ./datasets/toothbrush
  normal_dir: ./train/good # name of the folder containing normal images.
  abnormal_dir: ./test/defective # name of the folder containing abnormal images.
  normal_test_dir: ./test/good # name of the folder containing normal test images.
  task: segmentation # classification or segmentation
  mask: ./ground_truth/defective #optional
  extensions: null
  split_ratio: 0.2 # ratio of the normal images that will be used to create a test split
  image_size: 256
  normalization: imagenet # data distribution to which the images will be normalized: [none, imagenet]
  train_batch_size: 4
  test_batch_size: 4
  num_workers: 8
  transform_config:
    train: null
    val: null
  test_split_mode: from_dir # options: [from_dir, synthetic]
  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)
  val_split_mode: same_as_test # options: [same_as_test, from_test, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
  create_validation_set: true
  tiling:
    apply: False
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

model:
  name: patchcore
  backbone: wide_resnet50_2
  pre_trained: true
  layers:
    - layer2
    - layer3
  coreset_sampling_ratio: 0.1
  num_neighbors: 9
  normalization_method: min_max # options: [null, min_max, cdf]

metrics:
  image:
    - F1Score
    - AUROC
    - Precision
    - Recall
  pixel:
    - F1Score
    - AUROC
    - Precision
    - Recall
  threshold:
    method: adaptive #options: [adaptive, manual]
    manual_image: null
    manual_pixel: null

visualization:
  show_images: False # show images on the screen
  save_images: True # save images to the file system
  log_images: True # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 0
  path: ./results/patchcore/metrics

logging:
  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: false # Logs the model graph to respective logger.

optimization:
  export_mode: null # options: onnx, openvino

# PL Trainer Args. Don't add extra parameter here.
trainer:
  enable_checkpointing: true
  default_root_dir: null
  gradient_clip_val: 0
  gradient_clip_algorithm: norm
  num_nodes: 1
  devices: 1
  enable_progress_bar: true
  overfit_batches: 0.0
  track_grad_norm: -1
  check_val_every_n_epoch: 1 # Don't validate before extracting features.
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 1
  min_epochs: null
  max_steps: -1
  min_steps: null
  max_time: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  limit_predict_batches: 1.0
  val_check_interval: 1.0 # Don't validate before extracting features.
  log_every_n_steps: 50
  accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  strategy: null
  sync_batchnorm: false
  precision: 32
  enable_model_summary: true
  num_sanity_val_steps: 0
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_n_epochs: 0
  auto_lr_find: false
  replace_sampler_ddp: true
  detect_anomaly: false
  auto_scale_batch_size: false
  plugins: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle

Logs

/

Code of Conduct

• I agree to follow this project's Code of Conduct

[blaz-r]

Hello. Adaptive threshold is calculated using the following class:
https://github.com/openvinotoolkit/anomalib/blob/main/src/anomalib/utils/metrics/anomaly_score_threshold.py#L15
It works by finding a threshold that optimizes the F1 score.

When it comes to image and anomaly thresholds, they are calculated separately:

anomalib/src/anomalib/models/components/base/anomaly_module.py

Lines 48 to 50 in d7e7d86

	self.threshold_method: ThresholdMethod
	self.image_threshold = AnomalyScoreThreshold().cpu()
	self.pixel_threshold = AnomalyScoreThreshold().cpu()

The method is the same, but they differ in data they work on. In case of image threshold, it is calculated on image level predictions (indicating if image is anomalous or not) trying to maximize image level F1. In case of pixel threshold, the calculation is done on the entire anomaly map produced by the model:

anomalib/src/anomalib/models/components/base/anomaly_module.py

Lines 159 to 183 in d7e7d86

	def _compute_adaptive_threshold(self, outputs: EPOCH_OUTPUT) -> None:
	self.image_threshold.reset()
	self.pixel_threshold.reset()
	self._collect_outputs(self.image_threshold, self.pixel_threshold, outputs)
	self.image_threshold.compute()
	if "mask" in outputs[0].keys() and "anomaly_maps" in outputs[0].keys():
	self.pixel_threshold.compute()
	else:
	self.pixel_threshold.value = self.image_threshold.value
	self.image_metrics.set_threshold(self.image_threshold.value.item())
	self.pixel_metrics.set_threshold(self.pixel_threshold.value.item())
	@staticmethod
	def _collect_outputs(
	image_metric: AnomalibMetricCollection,
	pixel_metric: AnomalibMetricCollection,
	outputs: EPOCH_OUTPUT,
	) -> None:
	for output in outputs:
	image_metric.cpu()
	image_metric.update(output["pred_scores"], output["label"].int())
	if "mask" in output.keys() and "anomaly_maps" in output.keys():
	pixel_metric.cpu()
	pixel_metric.update(torch.squeeze(output["anomaly_maps"]), torch.squeeze(output["mask"].int()))

[weristeddy]

thank you very much for your help!

[Husnir24]

Hi, may i know if the adaptive threshold calculated based on images in valid dataset? or is it based on whole test dataset? is it possible to calculate threshold for each images in test dataset?

[samet-akcay]

Hi @Husnir24, it is calculated based on the validation set. However, note that current anomaly detection benchmarks such as MVTec does not have a validation set. In such cases, we use the same tests set as the validation set.

The existing threshold methods in anomalib do not support image-specific threshold calculation. However, we are working on some methods to enable this in later anomalib versions.