[Bug]: ValueError: The .compute() return of the metric logged as 'pixel_BinaryROC' must be a tensor.

[jamal-saeedi]

Describe the bug

File "/mnt/d/projects/pytorch/torch_dataset/mvtec_test.py", line 69, in
trainer.fit(model=model, datamodule=datamodule)
ValueError: The .compute() return of the metric logged as 'pixel_BinaryROC' must be a tensor. Found (tensor([0.0000, 0.1008, 0.1138, ..., 1.0000, 1.0000, 1.0000], device='cuda:0'), tensor([0.0000, 0.9915, 0.9953, ..., 1.0000, 1.0000, 1.0000], device='cuda:0'), tensor([1.0000, 1.0000, 1.0000, ..., 0.6186, 0.6186, 0.6185], device='cuda:0'))

Dataset

MVTec

Model

PADiM

Steps to reproduce the behavior

from future import annotations

import os
from pathlib import Path
from typing import Any

from git.repo import Repo

current_directory = Path.cwd()
root_directory = current_directory
os.chdir(current_directory)

import numpy as np
from matplotlib import pyplot as plt
from PIL import Image
from pytorch_lightning import Trainer
from torchvision.transforms import ToPILImage

from anomalib.config import get_configurable_parameters
from anomalib.data import get_datamodule
from anomalib.data.utils import read_image
from anomalib.deploy import OpenVINOInferencer
from anomalib.models import get_model
from anomalib.pre_processing.transforms import Denormalize
from anomalib.utils.callbacks import LoadModelCallback, get_callbacks

MODEL = "padim" # 'padim', 'cflow', 'stfpm', 'ganomaly', 'dfkde', 'patchcore'
CONFIG_PATH = root_directory / f"anomalib/models/{MODEL}/config.yaml"
with open(file=CONFIG_PATH, mode="r", encoding="utf-8") as file:
print(file.read())

config = get_configurable_parameters(config_path=CONFIG_PATH)

datamodule = get_datamodule(config)
datamodule.prepare_data() # Downloads the dataset if it's not in the specified root directory
datamodule.setup() # Create train/val/test/prediction sets.

i, data = next(enumerate(datamodule.val_dataloader()))
print(data.keys())

print(data["image"].shape, data["mask"].shape)

from anomalib.pre_processing.transforms import Denormalize

def show_image_and_mask(sample: dict[str, Any], index: int) -> Image:
img = ToPILImage()(Denormalize()(sample["image"][index].clone()))
msk = ToPILImage()(sample["mask"][index]).convert("RGB")

return Image.fromarray(np.hstack((np.array(img), np.array(msk))))

Visualize an image with a mask

show_image_and_mask(data, index=0)

Set the export-mode to OpenVINO to create the OpenVINO IR model.

config.optimization.export_mode = "openvino"

config.optimization.export_mode = "torch"

Get the model and callbacks

model = get_model(config)
callbacks = get_callbacks(config)

trainer = Trainer(**config.trainer, callbacks=callbacks) #
trainer.fit(model=model, datamodule=datamodule)

OS information

OS information:

• OS: [e.g. Ubuntu 20.04]
• Python version: [e.g. 3.8.10]
• Anomalib version: [e.g. 0.3.6]
• PyTorch version: [e.g. 1.9.0]
• CUDA/cuDNN version: [e.g. 11.1]
• GPU models and configuration: [e.g. 2x GeForce RTX 3090]
• Any other relevant information: [e.g. I'm using a custom dataset]

Expected behavior

.

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

No response

Configuration YAML

dataset:
  name: mvtec
  format: mvtec
  path: ./datasets/MVTec
  category: bottle
  task: segmentation
  train_batch_size: 8
  eval_batch_size: 8
  num_workers: 8
  image_size: 256 # dimensions to which images are resized (mandatory)
  center_crop: null # dimensions to which images are center-cropped after resizing (optional)
  normalization: imagenet # data distribution to which the images will be normalized: [none, imagenet]
  transform_config:
    train: null
    eval: null
  test_split_mode: from_dir # options: [from_dir, synthetic]
  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)
  val_split_mode: same_as_test # options: [same_as_test, from_test, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
  tiling:
    apply: false
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

model:
  name: padim
  backbone: resnet18
  pre_trained: true
  layers:
    - layer1
    - layer2
    - layer3
  normalization_method: min_max # options: [none, min_max, cdf]

metrics:
  image:
    - F1Score
    - AUROC
  pixel:
    - F1Score
    - AUROC
  threshold:
    method: adaptive #options: [adaptive, manual]
    manual_image: null
    manual_pixel: null

visualization:
  show_images: False # show images on the screen
  save_images: True # save images to the file system
  log_images: True # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 42
  path: ./results

logging:
  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: false # Logs the model graph to respective logger.

optimization:
  export_mode: torch # options: torch, onnx, openvino

# PL Trainer Args. Don't add extra parameter here.
trainer:
  enable_checkpointing: true
  default_root_dir: null
  gradient_clip_val: 0
  gradient_clip_algorithm: norm
  num_nodes: 1
  devices: 1
  enable_progress_bar: true
  overfit_batches: 0.0
  # track_grad_norm: -1
  check_val_every_n_epoch: 1 # Don't validate before extracting features.
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 25
  min_epochs: null
  max_steps: -1
  min_steps: null
  max_time: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  limit_predict_batches: 1.0
  val_check_interval: 1.0 # Don't validate before extracting features.
  log_every_n_steps: 10
  accelerator: gpu # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  # strategy: auto
  sync_batchnorm: false
  precision: 32
  enable_model_summary: true
  num_sanity_val_steps: 0
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_n_epochs: 0
  # auto_lr_find: false
  # replace_sampler_ddp: true
  detect_anomaly: false
  # auto_scale_batch_size: false
  plugins: null
  # move_metrics_to_cpu: false
  # multiple_trainloader_mode: max_size_cycle

Logs

Exception has occurred: ValueError
The `.compute()` return of the metric logged as 'pixel_BinaryROC' must be a tensor. Found (tensor([0.0000, 0.1008, 0.1138,  ..., 1.0000, 1.0000, 1.0000], device='cuda:0'), tensor([0.0000, 0.9915, 0.9953,  ..., 1.0000, 1.0000, 1.0000], device='cuda:0'), tensor([1.0000, 1.0000, 1.0000,  ..., 0.6186, 0.6186, 0.6185], device='cuda:0'))
  File "/mnt/d/projects/pytorch/torch_dataset/mvtec_test.py", line 69, in <module>
    trainer.fit(model=model, datamodule=datamodule)
ValueError: The `.compute()` return of the metric logged as 'pixel_BinaryROC' must be a tensor. Found (tensor([0.0000, 0.1008, 0.1138,  ..., 1.0000, 1.0000, 1.0000], device='cuda:0'), tensor([0.0000, 0.9915, 0.9953,  ..., 1.0000, 1.0000, 1.0000], device='cuda:0'), tensor([1.0000, 1.0000, 1.0000,  ..., 0.6186, 0.6186, 0.6185], device='cuda:0'))

Code of Conduct

• I agree to follow this project's Code of Conduct

[jamal-saeedi]

when I add task to the metric_cls in the following function, this bug generates, (using torchmetrics v1.2)

def metric_collection_from_names(metric_names: list[str], prefix: str | None) -> AnomalibMetricCollection:
"""Create a metric collection from a list of metric names.

The function will first try to retrieve the metric from the metrics defined in Anomalib metrics module,
then in TorchMetrics package.

Args:
    metric_names (list[str]): List of metric names to be included in the collection.
    prefix (str | None): prefix to assign to the metrics in the collection.

Returns:
    AnomalibMetricCollection: Collection of metrics.
"""
metrics_module = importlib.import_module("anomalib.utils.metrics")
metrics = AnomalibMetricCollection([], prefix=prefix)
for metric_name in metric_names:
    if hasattr(metrics_module, metric_name):
        metric_cls = getattr(metrics_module, metric_name)
        metrics.add_metrics(metric_cls(task='binary'))
    elif hasattr(torchmetrics, metric_name):
        try:
            metric_cls = getattr(torchmetrics, metric_name)
            metrics.add_metrics(metric_cls(task='binary'))
        except TypeError:
            warnings.warn(f"Incorrect constructor arguments for {metric_name} metric from TorchMetrics package.")
    else:
        warnings.warn(f"No metric with name {metric_name} found in Anomalib metrics or TorchMetrics.")
return metrics

[samet-akcay]

Hi @jamal-saeedi, if you use a recent torchmetrics such 1.2, you could get some issues. Please refer to this issue and the PR. Once this PR is merged on the torchmetrics side, we will bump the version

[jamal-saeedi]

hi, the issue is related to the class rapper being used in new version of torch metric without using init, and it can be resolved by directly using the class like BinaryF1Score or BinaryROC, but I would like to know about the possibility of using the rapper as base class.