🚀 Add MVTec LOCO dataset and sPRO metric #1967

djdameln · 2024-04-09T10:46:57Z

📝 Description

This PR is based on the closed PR #1686 , which adds the implementation of MVTec LOCO AD dataset [Paper] along with the sPRO metric.

🛠️ Fixes #574 #471 #1341

✨ Changes

Select what type of change your PR is:

🐞 Bug fix (non-breaking change which fixes an issue)
🔨 Refactor (non-breaking change which refactors the code base)
🚀 New feature (non-breaking change which adds functionality)
💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
📚 Documentation update
🔒 Security update

✅ Checklist

Before you submit your pull request, please make sure you have completed the following steps:

📋 I have summarized my changes in the CHANGELOG and followed the guidelines for my type of change (skip for minor changes, documentation updates, and test enhancements).
📚 I have made the necessary updates to the documentation (if applicable).
🧪 I have written tests that support my changes and prove that my fix is effective or my feature works (if applicable).

For more information about code review checklists, see the Code Review Checklist.

…ectory Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

…'masks' item Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

…g from the dataset Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

…cCollection Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

…to None Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Co-authored-by: Samet Akcay <samet.akcay@intel.com> Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

…lib into feature/mvtec-loco

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

…lib into feature/mvtec-loco

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

Co-authored-by: Ashwin Vaidya <ashwin.vaidya@intel.com>

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

…lib into feature/mvtec-loco

codecov · 2024-04-09T15:17:48Z

Codecov Report

Attention: Patch coverage is 85.71429% with 30 lines in your changes are missing coverage. Please review.

Project coverage is 81.39%. Comparing base (14ab566) to head (1cc19d7).

Files	Patch %	Lines
src/anomalib/callbacks/metrics.py	68.75%	10 Missing ⚠️
src/anomalib/metrics/spro.py	85.71%	9 Missing ⚠️
src/anomalib/data/image/mvtec_loco.py	91.57%	8 Missing ⚠️
src/anomalib/data/base/datamodule.py	66.66%	2 Missing ⚠️
src/anomalib/metrics/collection.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1967      +/-   ##
==========================================
+ Coverage   81.32%   81.39%   +0.07%     
==========================================
  Files         229      231       +2     
  Lines       10348    10541     +193     
==========================================
+ Hits         8415     8580     +165     
- Misses       1933     1961      +28

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/anomalib/callbacks/metrics.py

ashwinvaidya17

Thanks for polishing this. I have one comment.

src/anomalib/data/base/datamodule.py

ashwinvaidya17

Thanks

…re/mvtec-loco

willyfh and others added 30 commits February 9, 2024 12:46

add FROM_DIR option to val split mode to support a provided val dir…

c5c6bc7

…ectory Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

add a conditional check for the FROM_DIR option of val split mode

4c76092

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

add the mvtec loco ad dataset classes

ecc7169

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

add the default config file for mvtec loco ad dataset

9d89a68

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

update initialization files to include MVTec LOCO dataset

a312be0

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

remove unnecessary Path conversion

37a6f6b

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

add mvtec_loco.yaml to the readme documentation of configs

a733bf7

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

add dummy image generation for mvtec loco dataset

38ad1f4

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

add unit test for mvtec loco dataset

05e3558

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

update changelog to include the addition of mvtec loco dataset

519780a

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

add mvtec loco dataset to the sphinx-based docs

c3c6a39

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

fix the malformed table

ca91b8e

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

binarize the masks and avoid the possibility of the merge_mask is None

37f4dbc

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Merge the masks using sum operation without binarization

3559a7c

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

override getitem method to handle binarization and to add additional …

83539b4

…'masks' item Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Add saturation config to the datamodule

ddd33c5

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Update the saturation config on the metrics based on the loaded confi…

3af6eeb

…g from the dataset Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

add masks as a keyword args to the update method of the AnomalibMetri…

53f2297

…cCollection Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Shorten the comments to solve ruff issues

3d00d69

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Add sPro metric implementation

f024e91

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Change the saturation threshold to tensor

9b8ca3b

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Handle case with only background/normal images in scoring

29aaf46

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

rename spro metric and change the default value of saturation_config …

f432753

…to None Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

add unit test for spro metric

7d348d8

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

fix pre-commit issues

5597bfc

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

handle file not found error when loading saturation config

7b02863

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

validate path before processing

6048a08

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

update changelog with new PR

e237347

Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Update src/anomalib/data/image/mvtec_loco.py

f9b67b8

Co-authored-by: Samet Akcay <samet.akcay@intel.com> Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

Update src/anomalib/data/image/mvtec_loco.py

63bf8de

Co-authored-by: Samet Akcay <samet.akcay@intel.com> Signed-off-by: Willy Fitra Hendria <willyfitrahendria@gmail.com>

djdameln and others added 13 commits April 9, 2024 14:13

Merge branch 'feature/mvtec-loco' of github.com:openvinotoolkit/anoma…

4a8e487

…lib into feature/mvtec-loco

Update src/anomalib/data/base/datamodule.py

934f753

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

move and update loco config

63f210e

Merge branch 'feature/mvtec-loco' of github.com:openvinotoolkit/anoma…

98cd7f6

…lib into feature/mvtec-loco

Update tests/unit/data/image/test_mvtec_loco.py

ba371c4

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

Update tests/unit/metrics/test_spro.py

114f8e6

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

Update src/anomalib/data/image/mvtec_loco.py

7dbb983

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

Update src/anomalib/cli/cli.py

6587cac

Co-authored-by: Ashwin Vaidya <ashwin.vaidya@intel.com>

Update src/anomalib/metrics/spro.py

36ebf1c

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

Update src/anomalib/metrics/spro.py

b998823

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

fix mask loading

c844c1d

Merge branch 'feature/mvtec-loco' of github.com:openvinotoolkit/anoma…

501c116

…lib into feature/mvtec-loco

ruff

58c3b3f

djdameln added 3 commits April 9, 2024 17:58

fix multiple mask loading

4028b9a

masks -> semantic_mask

c87f742

Merge branch 'main' into feature/mvtec-loco

eabb755

djdameln commented May 1, 2024

View reviewed changes

src/anomalib/callbacks/metrics.py Outdated Show resolved Hide resolved

add metric collection for semantic pixel metrics

15e1e74

samet-akcay reviewed May 2, 2024

View reviewed changes

src/anomalib/callbacks/metrics.py Outdated Show resolved Hide resolved

add comment

69953ed

djdameln requested a review from samet-akcay May 3, 2024 06:02

ashwinvaidya17 reviewed May 14, 2024

View reviewed changes

src/anomalib/data/base/datamodule.py Show resolved Hide resolved

djdameln added 2 commits May 16, 2024 09:58

merge main

38d98c2

check if val_data is assigned for val_split_mode from_dir

3ffee16

ashwinvaidya17 approved these changes May 16, 2024

View reviewed changes

djdameln and others added 3 commits May 16, 2024 12:09

initialize semantic pixel metric names as list

b7c0d94

Merge branch 'main' of github.com:openvinotoolkit/anomalib into featu…

9da8a68

…re/mvtec-loco

Merge branch 'main' into feature/mvtec-loco

1cc19d7

samet-akcay approved these changes May 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🚀 Add MVTec LOCO dataset and sPRO metric #1967

🚀 Add MVTec LOCO dataset and sPRO metric #1967

djdameln commented Apr 9, 2024 •

edited by samet-akcay

codecov bot commented Apr 9, 2024 •

edited

ashwinvaidya17 left a comment

ashwinvaidya17 left a comment

🚀 Add MVTec LOCO dataset and sPRO metric #1967

Are you sure you want to change the base?

🚀 Add MVTec LOCO dataset and sPRO metric #1967

Conversation

djdameln commented Apr 9, 2024 • edited by samet-akcay

📝 Description

✨ Changes

✅ Checklist

codecov bot commented Apr 9, 2024 • edited

Codecov Report

ashwinvaidya17 left a comment

Choose a reason for hiding this comment

ashwinvaidya17 left a comment

Choose a reason for hiding this comment

djdameln commented Apr 9, 2024 •

edited by samet-akcay

codecov bot commented Apr 9, 2024 •

edited