Support hyperparameters tuning

[groar]

Being able to automatically optimize hyperparameters directly in digits would be a great feature. The two obvious methods would be random search or bayesian optimization.

I've been playing for the past week with using spearmint (bayesian optimization) with digits. Here is how I do it for now:

• I added to the json_dict method of ModelJob the best accuracy computed during training on the validation set.
• I setup a spearmint experiment as a function that uses the digits API to create a model from a dataset, with some variables (such as the learning rate, the step size and lr decay) being parameters of the experiment. It waits for the task to be completed and then return the val accuracy.

We end up with a bunch of model jobs in digits, and the optimized values for the parameters outputted by spearmint.

I'm now wondering what would be the best way to integrate random search or spearmint support into digits. I imagine it could be in the form of a special ModelJob with training tasks generated by spearmint for instance.

What do you think ?

[GiuliaP]

I have just found this code: https://github.com/kuz/caffe-with-spearmint
I though it might be useful for this task. @groar @lukeyeager did you know it?
I am starting to play with it to optimize fine-tuning parameters. Indeed I also would like to automatize the parameter search instead of launching each experiment manually... Having this integrated in DIGITS would be great!

[lukeyeager]

/cc @jmancewicz - you've been looking into parameter-sweep stuff, right?

[jmancewicz]

Yes, but won't be able to look at it for little while more.

[gheinrich]

With #708 now you can sweep through many values of learning rate and batch size.

[GiuliaP]

Thank you for the follow up. I will try to use it.

[gheinrich]

Closing (enhancement implemented in #708)