Enable some sort of caching - improve hyperparameter tuning

[egillax]

When running a deep learning model that takes a long time it's very frustrating if it errors out and you have to start from scratch. It would be useful to implement some sort of caching. So if you use a saveLoc that has data in it, it would check in those and just load data and continue with the hyperparameter tuning from where it left off. This requires saving the data just before starting training.

I would also like to refactor the hyperparameter search, wrap in a try - catch and in case of cuda memory errors not stop but continue with next iteration.