Tutorial on Efficient Hyperparameter Optimization for Machine Learning #9
Description
Title
Tutorial on Efficient Hyperparameter Optimization for Machine Learning
Responsible person(s)
Marcel Aach (m.aach@fz-juelich.de), JSC and University of Iceland
Xin Liu (xi.liu@fz-juelich.de), JSC
Format
Tutorial with presentations and hands-on sessions
Timeframe
Half day (3-4 hours)
Description
The performance of machine learning models is highly dependent on their hyperparameters that are set by the user before the training. The hyperparameters define the general architecture of the model (e.g., via the number of layers or the neurons per layer in a neural network) and control the learning process (e.g., via the learning rate of the optimizer or the weight decay). However, searching for optimal hyperparameter values is a long and resource-intensive process, as many different combinations need to be evaluated and the final performance of a combination can usually only be measured after a machine learning model is fully trained.
This tutorial presents a systematic introduction to the field of Hyperparameter Optimization (HPO) and demonstrates how to make use of resource-efficient methods to reduce the runtime of HPO in small and large settings on High-Performance Computing systems. Two HPO optimization libraries (Ray Tune and DeepHyper) are introduced, making use of evolutionary, Bayesian, and early stopping-based algorithms. As HPO is a general method and can be adapted to any machine learning model, it is useful for scientists from many different domains.
Requirements
Room with presentation equipment and enough power outlets for Laptops
Participants should bring their own Laptops and be familiar with running code on a cluster
Compute time reservation and accounts on a GPU-based system at JSC for the participants to run code during the hands-on sessions
5-20 participants
Agenda:
13:30 - 14:15 General Introduction to Hyperparameter Optimization and Algorithms
14:15 - 15:45 Hands-On with Ray Tune Library
15:45-16:00 Break
16:00 - 16:30 Hands-On with Weights and Biases Library for Logging and Visualization of HPO
16:30-16:45 Discussion of Results
Participants should have access to Google Colab (https://colab.research.google.com/), as well as an account with Weights and Biases (https://wandb.ai/) for the hands-on part.