Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training Error #7

Closed
jainnipun11 opened this issue Jun 9, 2022 · 4 comments
Closed

Training Error #7

jainnipun11 opened this issue Jun 9, 2022 · 4 comments

Comments

@jainnipun11
Copy link

Hey! I unzipped the images in the suggested path, but still I keep getting the:

FileNotFoundError: [Errno 2] No such file or directory: '/home/data_storage/mimic-cxr/dataset/image_preprocessing/re_512_3ch/Train/s50328096.jpg'

Can you elaborate why this error is coming?

Thanks.

@ttumyche
Copy link
Collaborator

ttumyche commented Jun 9, 2022

Hi, jainnipun

I guess that error occurred in this line
How did you define the suggested path? or you can just change 'fixed_path' to yours.

@jainnipun11
Copy link
Author

Thanks! Resolved the "fixed_path" issue, but now it's giving CUDA out of memory error:

CUDA out of memory. Tried to allocate 314.00 MiB (GPU 0; 15.78 GiB total capacity; 13.97 GiB already allocated; 240.75 MiB free; 14.33 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

What to do? Should I decrease the batch size?

@ttumyche
Copy link
Collaborator

Yup, reduce the batch size to fit your GPU VRAM first.
If that does not solve the error, let me know again

@jainnipun11
Copy link
Author

Yes! Training started after I reduced the "batch_size" from 36 to 15. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants