Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot load network snapshot for restarting: corrupted pickles? #17

Closed
blackcat84 opened this issue Jul 25, 2019 · 1 comment
Closed

Comments

@blackcat84
Copy link

blackcat84 commented Jul 25, 2019

I'm trying to restart a job by loading a network snapshot pickle (called network-snapshot-001283.pkl). The pickle is written during training of the previous run and loaded at the beginning of the restart by the functions at the beginning of training/misc.py.

I get the following error when trying to load the snapshot:

_pickle.UnpicklingError: invalid load key, '<'.

This error appears for all my snapshots produced training with different data on different runs, as well as when trying to use generate_figures.py.
Either the pickle written during training is corrupted, or I'm doing something wrong, although I haven't change any line corresponding to this part.
Any idea?

Thanks

@blackcat84
Copy link
Author

blackcat84 commented Jul 25, 2019

Nevermind, I've just found out that indeed the files were corrupted due to a bad file transfer between two different machines. Nothing wrong with the code.

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant