Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address icevision 3-band training accuracy #70

Open
rbavery opened this issue May 6, 2022 · 2 comments
Open

Address icevision 3-band training accuracy #70

rbavery opened this issue May 6, 2022 · 2 comments
Assignees

Comments

@rbavery
Copy link
Collaborator

rbavery commented May 6, 2022

On a first pass through 10 epochs of the 3 band dataset, accuracy of the IOU metric is low, with no predictions being generated for any example results. This is in contrast to the 1 band dataset which does generate some valid predictions on a smaller partial dataset.

I'll be looking into the trainer more closely to see why no valid predictions are being generated. First thing I'll do is to increase the resolution of the train and loss curves and try to substantially overfit on the training set, since currently training loss decreases at about the same rate as the validation set.

@rbavery rbavery self-assigned this May 6, 2022
@rbavery
Copy link
Collaborator Author

rbavery commented May 9, 2022

List of priorities for improving Mask RCNN accuracy:

  • correctly set the anchor aspect ratios and bounding boxes
  • only train on prominent classes (coincident vessel first), add in more to see what breaks
  • replace aux channels with all black

first thing I'll try is comparing the simpler fastai unet 1 band to 3 band to see if this is an issue with the aux data or the icevision trainer.

@rbavery
Copy link
Collaborator Author

rbavery commented May 10, 2022

Update, the fastai unet Dice loss goes up to .3 when training on the first progressive resize of 64. this is slightly above the Dice loss of .2 after 5 epochs for CV1. so it would appear that there's something specific to the icevision implementation that is causing problems. @lillythomas has also discovered a new error where with the single band model, training cuases kernel death after about 4 epochs. We're tabling these issues while we focus on the Unet baseline.

@rbavery rbavery removed the Sprint 8 label May 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant