Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
181 lines (142 sloc) 6.06 KB
description
Syntax to set the hyperparameter ranges, search strategy, and other aspects of your sweep.

Configuration

Use these configuration fields to customize your sweep. There are two ways to specify your configuration:

  1. YAML file: best for distributed sweeps
  2. Python data structure: best for running a sweep from a Jupyter Notebook
Top-level key Meaning
name The name of the sweep, displayed in the W&B UI
description Text description of the sweep (notes)
program Training script to run (required)
metric Specify the metric to optimize (used by some search strategies and stopping criteria)
method Specify the search strategy (required)
early_terminate Specify the stopping criteria
parameters Specify parameters bounds to search (required)

Metric

Specify the metric to optimize. This metric should be logged explicitly to W&B by your training script. For example, if you want to minimize the validation loss of your model:

# [model training code that returns validation loss as valid_loss]
wandb.log({"val_loss" : valid_loss})
metric sub-key Meaning
name Name of the metric to optimize
goal minimize or maximize (Default is minimize)

Examples

{% tabs %} {% tab title="Maximize" %}

metric:
  name: val_loss
  goal: maximize

{% endtab %}

{% tab title="Minimize" %}

metric:
  name: val_loss

{% endtab %} {% endtabs %}

Search Strategy

Specify the search strategy with the method key in the sweep configuration.

method Meaning
grid Grid search iterates over all possible combinations of parameter values.
random Random search chooses random sets of values.
bayes Bayesian Optimization uses a gaussian process to model the function and then chooses parameters to optimize probability of improvement. This strategy requires a metric key to be specified.

Examples

{% tabs %} {% tab title="Random search" %}

method: random

{% endtab %}

{% tab title="Grid search" %}

method: grid

{% endtab %}

{% tab title="Bayes search" %}

method: bayes
metric:
  name: val_loss
  goal: minimize

{% endtab %} {% endtabs %}

Stopping Criteria

Early Termination speeds up hyperparameter search by stopping any poorly-performing runs.

early_terminate sub-key Meaning
type specify the stopping algorithm

We support the following stopping algorithm(s):

type Meaning
hyperband Use the hyperband method

Hyperband stopping evaluates whether a program should be stopped or permitted to continue at one or more brackets during the execution of the program. Brackets are configured at static iterations for a specified metric (where an iteration is the number of times a metric has been logged -- if the metric is logged every epoch, then there are epoch iterations).

In order to specify the bracket schedule, eithermin_iter or max_iter needs to be defined.

early_terminate sub-key Meaning
min_iter specify the iteration for the first bracket
max_iter specify the maximum number of iterations for the program
s specify the total number of brackets (required for max_iter)
eta specify the bracket multiplier schedule (default: 3)

Examples

{% tabs %} {% tab title="Hyperband (min_iter)" %}

early_terminate:
  type: hyperband
  min_iter: 3

Brackets: 3, 9 (3*eta), 27 (9 * eta), 81 (27 * eta) {% endtab %}

{% tab title="Hyperband (max_iter)" %}

early_terminate:
  type: hyperband
  max_iter: 27
  s: 2

Brackets: 9 (27/eta), 3 (9/eta) {% endtab %} {% endtabs %}

Parameters

Describe the hyperparameters to explore. For each hyperparameter, specify the name and the possible values as a list of constants or a range with a certain distribution.

Values Meaning
distribution: (distribution) A distribution from the distribution table below. If not specified, the sweep will set to uniform if max and min are set, categorical if values are set and constant if value is set.
min: (float) max: (float) Continuous values between min and max
min: (int) max: (int) Integers between min and max
values: [(float), (float), ...] Discrete values
value: (float) A constant
mu: (float) Mean for normal or lognormal distributions
sigma: (float) Standard deviation for normal or lognormal distributions
q: (float) Quantization parameter for quantized distributions

Distributions

Name Meaning
constant Constant distribution. Must specify value.
categorical Categorical distribution. Must specify values.
int_uniform Uniform integer. Must specify max and min as integers.
uniform Uniform continuous. Must specify max and min as floats.
q_uniform Quantized uniform. Returns round(X / q) * q where X is uniform. Q defaults to 1.
log_uniform Log uniform. Number between exp(min) and exp(max) so that the logarithm of the return value is uniformly distributed.
q_log_uniform Quantized log uniform. Returns round(X / q) * q where X is log_uniform. Q defaults to 1.
normal Normal distribution. Value is chosen from normal distribution. Can set mean mu (default 0) and std dev sigma (default 1).
q_normal Quantized normal distribution. Returns round(X / q) * q where X is normal. Q defaults to 1.
log_normal Log normal distribution. Value is chosen from log normal distribution. Can set mean mu (default 0) and std dev sigma (default 1).
q_log_normal Quantized log normal distribution. Returns round(X / q) * q where X is log_normal. Q defaults to 1.

Example

parameters:
  my-parameter:
    min: 1
    max: 20
You can’t perform that action at this time.