Getting NaN loss values while training PatchCore model on custom dataset

[RahaviSelvarajan]

Describe the bug

• I am training the patchcore model on a custom dataset. During training, loss value in the progress bar is shown as NaN. Why would this happen?

Screenshots
Epoch 0: 2%|█▋ Aggregating the embedding extracted from the training set.5/218 [00:06<04:30, 1.27s/it, loss=nan]
Creating CoreSet Sampler via k-Center Greedy
Getting the coreset from the main embedding.
Assigning the coreset as the memory bank.
Epoch 1: 2%|███▏ Aggregating the embedding extracted from the training set. | 5/218 [01:34<1:07:23, 18.98s/it, loss=nan]
Creating CoreSet Sampler via k-Center Greedy
Getting the coreset from the main embedding.
Assigning the coreset as the memory bank.
Epoch 2: 2%|███▏ Aggregating the embedding extracted from the training set. | 5/218 [06:39<4:43:55, 79.98s/it, loss=nan]

[alexriedel1]

This is because you do not update any weights during the "training" of patchcore but only save features while inferencing your training data (https://arxiv.org/pdf/2106.08265.pdf). Doing this for multiple epochs will not improve the model unless you apply random image augmentations in each epoch (which is why the default number of training epochs is 1 and you shouldn't change this parameter unless for any obvious reasons)

[samet-akcay]

@RahaviSelvarajan, as @alexriedel1 pointed out, the reason is because the CNN is only used for feature extraction, which doesn't produce a loss value. If this looks confusing, we could consider removing it.

[nguyenanhtuan1008]

@samet-akcay so in this situation. Train 1 epoch is the same result compared to multiple epochs?

[samet-akcay]

@nguyenanhtuan1008, yes. Training single vs multiple epoch would yield the same result for all models that use CNN for feature extraction.

The exception would be to apply random image augmentations, as @alexriedel1 pointed out above.

[nixczhou]

hi regarding this, how to do augmentation in patchcore. Seems like patchcore need heavy augmentation right?