-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error when saving Flux models/CuArrays from GPU #55
Comments
Right, you can't save CuArrays with BSON.jl. Doing |
|
Agreed. If you have time, it would be great if you can submit a Flux PR to make this very clear in the docs. |
Agreed, just had the same problem and it's written nowhere in the Flux documentation |
Sounds like this issue has been resolved? Out of curiosity, it is possible to overload some method so that when someone tries to save CuArray to BSON, it copies the data into an Array and saves that? And maybe it's possible it's possible to save a tiny bit of metadata so that when loading a "CuArray" from disk, it creates an Array and copies the data over into a newly created CuArray? |
I would say |
@jpsamaroo a fellow student hit this bug last week, I'm thinking throw an Where should this be, unfortunately there's no interface package defining this type, so I'm thinking adding a |
Saved Flux models or CuArrays while they are in GPU Memory can only be loaded again in the same julia session. Once this session is terminated and a new session is started, loading this data will either result in random values or in a CUDA error.
This can make trained and saved models useless since most of the time they will be loaded in a new julia session...
MWE (for CuArrays):
this gives me the correct output:
Loading the data in a new session
will result in an error:
The error occurs during the show command and not during loading!
I experienced the same issue when I tried to save Flux models. Saving and loading worked without errors but the loaded model had not the trained weights but random values.
The documentation of Flux only says that GPU support needs to be available when loading models which where in GPU memory when saved.
The text was updated successfully, but these errors were encountered: