Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NaN error in validation phase #7

Closed
hvy1seok opened this issue May 24, 2023 · 1 comment
Closed

NaN error in validation phase #7

hvy1seok opened this issue May 24, 2023 · 1 comment

Comments

@hvy1seok
Copy link

I am trying to run this code by using maxillo dataset. But it occurs a error below .

Traceback (most recent call last):
File "/home/syu/Documents/maison/alveolar_canal/main.py", line 170, in
val_iou, val_dice = experiment.test(phase="Validation")
File "/home/syu/Documents/maison/alveolar_canal/experiments/experiment.py", line 276, in test
loss = self.loss(output.unsqueeze(0), gt.unsqueeze(0), partition_weights)
File "/home/syu/Documents/maison/alveolar_canal/losses/LossFactory.py", line 60, in call
raise ValueError(f'Loss {loss_name} has some NaN')
ValueError: Loss DiceLoss has some NaN

I'm wondering if this code and dataset work as written on the readme or if there are no corrupted files in this dataset.
Big congratulation ti your acivements in cvpr!

@LucaLumetti
Copy link
Collaborator

Hi @hvy1seok,
I can assure you that there are no corrupted files, as long as they didn't get corrupted while you were downloading them, but it's quite unlikely. Looks like you used the DiceLoss, while we always employed the JaccardLoss. Thus, a bug might be slipped in the code of the DiceLoss and never get noticed, I'm gonna look deeply into it in the following days. if you would like to spend some time debugging it, I can give you some guess on which is the guilty part that produces NaNs:

  • Assure that in this call, both pred and gt has no NaNs.
  • Assure that also gt_onehot doesn't contain NaNs.
  • Assure that cardinality has no zero values.

@prittt prittt closed this as completed Jul 31, 2023
LucaLumetti added a commit that referenced this issue Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants