Hyper-Parameter Optimization

[L1aoXingyu]

This guide explains how to train your model with hyperparameters optimization.

Before You Start

Following Getting Started to setup the environment and install requirements.txt dependencies.

We provide two search algorithms, HyperOptSearch and TuneBOHB. You can refer here to learn more information about search algorithms in ray[tune]. In addition, we provide four trial schedulers, ASHA, HyperBand, PBT, and BOHB. More information about trial schedulers can be found here.

Design Hyperparameters Search Space

There are many hyperparameters used for various training settings, such as batch size, learning rate, weight decay, and so on. Better initial guess will produce better results, so it's important to initialize these values range properly before tuning.

Each search algorithm has a specific way of defining the search space, for example, you can define bohb hyperparameters space as below

fast-reid/projects/HPOReID/train_hpo.py

Lines 178 to 190 in a5dac37

	search_space.add_hyperparameters([
	# CS.UniformFloatHyperparameter(name="lr", lower=1e-6, upper=1e-2, log=True),
	# CS.UniformIntegerHyperparameter(name="delay_iters", lower=20, upper=60),
	# CS.UniformFloatHyperparameter(name="ce_scale", lower=0.1, upper=1.0),
	# CS.UniformIntegerHyperparameter(name="circle_scale", lower=8, upper=256),
	# CS.UniformFloatHyperparameter(name="circle_margin", lower=0.1, upper=0.5),
	# CS.UniformFloatHyperparameter(name="wd", lower=0, upper=1e-3),
	# CS.UniformFloatHyperparameter(name="wd_bias", lower=0, upper=1e-3),
	CS.CategoricalHyperparameter(name="bsz", choices=[64, 96, 128, 160, 224, 256]),
	CS.CategoricalHyperparameter(name="num_inst", choices=[2, 4, 8, 16, 32]),
	# CS.CategoricalHyperparameter(name="autoaug_enabled", choices=[True, False]),
	# CS.CategoricalHyperparameter(name="cj_enabled", choices=[True, False]),
	])

Here, we just want to search batch_size and num_instance. You can define your own hyperparameters search space.

Then, you need to modify function update_config to make the hyperparameter valid.

fast-reid/projects/HPOReID/train_hpo.py

Lines 96 to 118 in a5dac37

	def update_config(cfg, config):
	cfg.defrost()
	# lr, weight decay
	# cfg.SOLVER.BASE_LR = config["lr"]
	# cfg.SOLVER.ETA_MIN_LR = config["lr"] * 0.0022
	# cfg.SOLVER.DELAY_ITERS = config["delay_iters"]
	# cfg.SOLVER.WEIGHT_DECAY = config["wd"]
	# cfg.SOLVER.WEIGHT_DECAY_BIAS = config["wd_bias"]
	# batch size, number of instance
	cfg.SOLVER.IMS_PER_BATCH = config["bsz"]
	cfg.DATALOADER.NUM_INSTANCE = config["num_inst"]
	# loss related
	# cfg.MODEL.LOSSES.CE.SCALE = config["ce_scale"]
	# cfg.MODEL.HEADS.SCALE = config["circle_scale"]
	# cfg.MODEL.HEADS.MARGIN = config["circle_margin"]
	# data augmentation
	# cfg.INPUT.DO_AUTOAUG = config["autoaug_enabled"]
	# cfg.INPUT.CJ.ENABLED = config["cj_enabled"]
	return cfg

At last, you can decide which hyperparameters can be displayed on the console output.

fast-reid/projects/HPOReID/train_hpo.py

Line 206 in a5dac37

parameter_columns=["bsz", "num_inst"],

Define Fitness Score

Fitness is the value we seek to maximize. In fastreid, we have defined a default fitness score as a weighted combination of metrics: Rank@1 and mAP.

fast-reid/projects/HPOReID/hporeid/tune_hooks.py

Line 56 in a5dac37

metrics = dict(r1=results['Rank-1'], map=results['mAP'], score=(results['Rank-1'] + results['mAP']) / 2)

Then just pass the name score to metric, and max to mode.

fast-reid/projects/HPOReID/train_hpo.py

Line 138 in a5dac37

exp_metrics = dict(metric="score", mode="max")

HPO Train

Then you can train hyperparameter optimization with BOHB(Bayesian Optimization with HyperBand) search algorithm with 20 trials like the following command line

python3 projects/HPOReID/train_hpo.py --config-file projects/HPOReID/configs/baseline.yml --srch-algo "bohb" --num-trials 20

If you don't have enough resource, you can allocate the specific GPU and CPU from resources_per_trial

fast-reid/projects/HPOReID/train_hpo.py

Line 213 in a5dac37

resources_per_trial={"cpu": 12, "gpu": 1},

---

Frequently asked questions about ray[tune] can be found here.

[huangzhenjie]

What is the method of hyper-parameter optimization and are there any papers about it?

[L1aoXingyu]

This project supports Hyperband, bohb and population-based training.

[huangzhenjie]

when i try to train, the error is "train_hpo.py: error: unrecognized arguments: --srch-algo", What should I do?

[L1aoXingyu]

Use the latest code and try again.

[huangzhenjie]

I would like to ask what configuration is required for the machine to run Hyper-Parameter Optimization, thanks.

[L1aoXingyu]

Nothing special for machine configuration.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.