Automating hyper-parameter tuning

[sophia-wright-blue]

Hello,

I'm learning a lot from the book, and am beginning to write my own code now. I have a question about a statement in the section "Hyperparameter Optimization Algorithms" - in the box titled "CAN’T HYPERPARAMETER OPTIMIZATION BE AUTOMATED?" you mention that

"In recent years, there has been a surge of work focused on improving the algorithmic foundations of model tuning. Gaussian processes, evolutionary algorithms, and reinforcement learning have all been used to learn model hyperparameters and architectures with very limited human input"

Could you point me to some papers/references that use Gaussian processes or evolutionary algorithms to automate the tuning process?

Thank you,

[rbharath]

It's tricky to run Gaussian process optimization, but can add a lot of value if done right. I'd recommend checking out Spearmint to get started:

https://github.com/HIPS/Spearmint

I'm not familiar with a good evolutionary algorithm library though.

[sophia-wright-blue]

Thank you for your reply. I'll check out Spearmint. I'm familiar with Differential Evolution for parameter estimation, so I'm planning to use DE on my first attempt. Please let me know if you do come across any recent references that compare the GP and DE methods. Thank you again.