-
Notifications
You must be signed in to change notification settings - Fork 150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
fastai efficientdet fails on learn.validate() with AttributeError: 'NoneType' object has no attribute 'shape' #638
Comments
I found that applying a following patch: from fastai.learner import AvgLoss
from fastai.torch_core import find_bs
@patch
def accumulate(self:AvgLoss, learn):
#bs = find_bs(learn.yb)
bs = find_bs(learn.xb)
self.total += learn.to_detach(learn.loss.mean())*bs
self.count += bs fixes the issue. Im not sure though how should we store that patch in icevision. Anny suggestions? |
I too had this issue which got solved with this patch and online help from @potipot . Thanks a lot. Hopefully, this is pushed as a PR. |
@lgvaz this is still an issue. Just wanted to ask for your opinion on where to put this fastai patch? |
@potipot i get access denied on the colab notebook, can you change the permissions? |
Wow, this error is really subtle, specially the why it was not happening before. The error was introduced by #630 but the bug was there since the beginning. To first understand it we have to take a look at how def find_bs(b):
"Recursively search the batch size of `b`."
return item_find(b).shape[0]
def item_find(x, idx=0):
"Recursively takes the `idx`-th element of `x`"
if is_listy(x): return item_find(x[idx])
if isinstance(x,dict):
key = list(x.keys())[idx] if isinstance(idx, int) else idx
return item_find(x[key])
return x In our case And this is what #630 changed, before the first key in the dict was And calling find_bs gives returns After it started being Now The solution is to do what @potipot proposed here but without patching, instead we should do the same thing that we do for torchvision models, when creating the learner ( class RCNNAvgLoss(fastai.AvgLoss):
def accumulate(self, learn):
bs = len(learn.yb)
self.total += fastai.to_detach(learn.loss.mean()) * bs
self.count += bs
recorder = [cb for cb in learn.cbs if isinstance(cb, fastai.Recorder)][0]
recorder.loss = RCNNAvgLoss() We also have to check if the same happens for the mmdet models. @potipot would you like to do a PR for this one? |
@lgvaz im not sure whats the idea behind |
Wait, I got a bit confused, what solution actually gives the correct batch size? For |
Problem is, I'm not sure what this is trying to do. I wasn't able to get values other than 1 there. So I guess the idea is to take an average of the loss and weigh it with the batch size. We could actually ignore this and just put a blank 1, so no weighing would happen. When training via tests with bs=2 using RCNN and EfficientDet both I was always getting a single element (not a batch) from |
The problem is that the test dataset (fridge_ds) has only 1 element in the validation set. Therefore I cannot test which list corresponds to the actual batch size, as it is always 1. :D I will increase the size of validation set to say 3 and then test again. |
馃悰 Bug
when trying to simply validate metrics for an efficientdet model with fastai
it fails when trying to read the batch size automatically: in
accumulate, find_bs
To Reproduce
Steps to reproduce the behavior:
The text was updated successfully, but these errors were encountered: