Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AUC is lower than PADIM #1

Open
XiaoJiNu opened this issue Jun 29, 2021 · 9 comments
Open

AUC is lower than PADIM #1

XiaoJiNu opened this issue Jun 29, 2021 · 9 comments

Comments

@XiaoJiNu
Copy link

XiaoJiNu commented Jun 29, 2021

As following, auc of some classes is much less than PADIM. I think better AUC is more important than better PRO.

image

@Pangoraw
Copy link
Owner

Pangoraw commented Jun 29, 2021

It seems that https://github.com/xiahaifeng1995/PaDiM-Anomaly-Detection-Localization-master is using a center crop on the images. Do you know if this is common practice on the MVTec AD dataset ?

If you add it to the example mvtec file you get similar image ROCAUC scores (see a1b80ad for an example).

@XiaoJiNu
Copy link
Author

@Pangoraw Sorry, I didn't know the difference. So, you meant that https://github.com/xiahaifeng1995/PaDiM-Anomaly-Detection-Localization-master uses transforms.CenterCrop(224) but you didn't, that's why your ROCAUC is lower ? On the other hand, I think it's not good to crop the center of images because anomaly is not always in the center.

@Pangoraw
Copy link
Owner

Pangoraw commented Jun 30, 2021

Also which backbone and number of dimensions (k) are you using for both PaDiM and this method ?

On the other hand, I think it's not good to crop the center of images because anomaly is not always in the center.

The main goal is still to compare the methods in the same setting to assess the performance. You can chose to remove or add transformations for your own application. Also note that the pixel ROCAUC (which favors big anomalies) you have on the left column is different from the PRO score metric (see this paper for more details).

@XiaoJiNu
Copy link
Author

XiaoJiNu commented Jul 2, 2021

@Pangoraw Thanks for your reply. Another question is that the paper uses semiOrthogonal to accelerate computing speed, but I didn't find the speed comparasion. How much it is faster than not use semiOrthogonal ?

@Pangoraw
Copy link
Owner

Pangoraw commented Jul 2, 2021

Well, compared to methods learnt using gradient descent like a VAE or AnoGAN it is faster because you only need a single forward pass. But it will be marginally slower than PaDiM I guess.

@XiaoJiNu
Copy link
Author

XiaoJiNu commented Jul 3, 2021

@Pangoraw Ok, I will test it more. Your job is so great !

@XiaoJiNu
Copy link
Author

XiaoJiNu commented Jul 5, 2021

@Pangoraw Hello, I have modified the code and the result is as following picture, average auc is higher than before adding center crop but still lower than PADIM. What should I do to improve auc ? Whether auc of Semiorthogonal is indeed lower than PADIM ?

image

@Pangoraw
Copy link
Owner

Pangoraw commented Jul 6, 2021

hmm interesting.. I was able to get a better ROC AUC by computing the image score after the gaussian blur but it is still not as high as PaDiM. However, i found that changing the random seed in the PaDiM implementation yields totally different results (see xiahaifeng1995/PaDiM-Anomaly-Detection-Localization-master#15). I think this particular seed selects relevant dimensions for discriminating the MVTec dataset but may not be as good for other problems where you would need to search for a new seed whereas with the semi-orthogonal you should have a more consistant performance because all dimensions are blended.

@XiaoJiNu
Copy link
Author

XiaoJiNu commented Jul 8, 2021

@Pangoraw Maybe seed selets is the keypoint, I will test it on other datasets in the future and discuss the result with you. Thank you !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants