Saving caffemodel and solverstate

[jaewoosong]

Hi! Always thank you for this great program. May I ask about saving caffemodel and solverstate?

I tried to save the trained model as .caffemodel and .solverstate files. It seemed that the snapshot of cifar10_quick_solver.prototxt did not work in SparkNet.

I found commented lines in CifarDBApp.scala:

if (i % 10 == 0) {
  net.setWeights(netWeights)
  net.saveWeightsToFile("/root/weights/" + i.toString + ".caffemodel")
}

But I could not understand how to use those lines. At the same time, it seems that many things have been changed with recent update so I guess even if I had found out how to use those lines, it would not be applicable to the new version of SparkNet.

I feel sorry to ask a question because I can see that you are very busy. A lot of code update! Thank you for always responding to my issue writings and questions.

[robertnishihara]

Good point, saving models is critical, so we need to reintroduce that code (it's no longer in the master).

In the code snippet above, we first move the weights from the WeightCollection object netWeights into the Net object net. Then we call the net's method saveWeightsToFile which calls a caffe method that stores a snapshot of the weights.

We could save the WeightCollection object directly, but we probably want to provide an interface for reading/writing .caffemodel files so that SparkNet interfaces nicely with Caffe.

[jaewoosong]

Thank you!