-
Notifications
You must be signed in to change notification settings - Fork 140
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AnalogConv2d fails when using TT-v2 #642
Comments
Hi @Zhaoxian-Wu, Thanks for reporting this issue. what GPU were you using when you encountered this issue? |
Hi @Zhaoxian-Wu for the CUDA memory problem, it looked like the problem had to do with how to set the DEVICE. If I set it by |
It tried the same technique with torch.empty and I did not see the hanging(looping) issue either. So this is torch problem. Let us know if you have any questions. |
@Zhaoxian-Wu do you still have this issue. if not, we can close this. Please let us know |
I tried this solution and it worked! It seems to be the issue from Pytorch. Thank you for your help @kkvtran! It is weird for the Pytorch community to leave this issue for such a long time. |
Description
When I tried to use the TT-v2 algorithm to train the convolutional network, I got a Cuda error.
How to reproduce
After running the following
main.py
file, I got an errorRuntimeError: CUDA_CALL Error 'an illegal memory access was encountered' at cuda_util.cu:653
Besides, if I create
torch.empty()
instead oftorch.one()
, the forward clausemodel(images)
never stops, I guess there could be some endless loop happening.Other information
The text was updated successfully, but these errors were encountered: