This is the source repository for Guild AI.
Guild AI is an open source toolkit that automates and optimizes machine learning experiments.
- Run unmodified training scripts, capturing each run result as a unique experiment
- Automate trials using grid search, random search, and Bayesian optimization
- Compare and analyze runs to understand and improve models
- Backup training related operations such as data preparation and test
- Archive runs to S3 or other remote systems
- Run operations remotely on cloud accelerators
- Package and distribute models for easy reproducibility
For more on features, see Guild AI - Features.
- Latest release on PyPI
- Guild AI website
- Guild AI Slack (for questions/support)
- Open issues
- Linux, macOS, Windows (Windows requires Docker)
- Python 2.7 or 3 with
Install Guild AI
To install Guild AI with pip, run:
pip install guildai
If you already have Guild AI installed, you can upgrade it to the latest release using:
pip install guildai --upgrade
To install pre-release versions (e.g. you want the latest features or
fixes), include the
--pre command line option:
pip install guildai --upgrade --pre
NOTE: You may need to run
pip as a privileged user (e.g. run using
sudo) or else specify the
--user command line option if you file
permission errors when running
Ensure that you have TensorFlow installed.
NOTE: As of Guild 0.6, all major computational and ML frameworks are supported including PyTorch, Keras, MLKit, scikit-learn, and XGBoost. TensorFlow is required for TensorBoard integration. Your models do not have to use TensorFlow to work with Guild AI. The TensorFlow requirement will be removed in future releases of Guild.
For detailed install instructions, see Install Guild AI.
In this Quick Start guide, we create a mock training script and run it to illustrate the following features:
- Run, capture and compare experiments
- Use grid search and random search to explore hyperparameter space
- Use Bayesian optimization to achieve the best performance
Links to more advanced topics are provided at the end of this section.
Mock training script
In a new project directory, create a file named
import numpy as np x = 0.1 noise = 0.1 loss = (np.sin(5 * x) * (1 - np.tanh(x ** 2)) + np.random.randn() * noise) print("x: %f" % x) print("noise: %f" % noise) print("loss: %f" % loss)
NOTE: This is a mock (fake) training script — it doesn't train anything! However, it illustrates the basics of training, where an operation is used to minimize loss given a set of inputs. In this example, we calculate loss using a noisy function and a single input x, which represents our hyperparameter.
Run an experiment
Open a command console and change to the project directory.
Use Guild to run the mock training script:
guild run train.py
Guild prompts you with the default values as defined in
Enter to run the script.
NOTE: Guild automatically detects the flags defined in
uses the default values. Later we run
train.py with different values
x and even use Guild to find values that minimize the function
When Guild runs a script, it captures the run as a unique experiment. You can list runs by running:
View information for this run using:
guild runs info
By default, Guild shows information for the latest run.
NOTE: For information on any Guild command, use:
guild COMMAND --help. For a list of commands, use
guild --help. Online help is
Experiments are saved on disk within file system directories. List files associated with the latest run using:
In the case of our mock training script, the files list is empty because the script doesn't generate any files. However, you can see the path where the run is located.
Guild stores information associated with each run in files located in
.guild subdirectory of each run. You can list all of the files
associated with the latest run, including Guild files, by specifying
--all command line option:
guild ls --all
If you want to export an experiment, use the
guild export /tmp/my-experiments
This is useful for creating archives of runs that can be imported by
you and others using the
import command. For a more advanced example
of backing runs up to the cloud, see Get Started - Backup and
You can list runs located in an archive directory by specifying the
--archive command line option:
guild runs --archive /tmp/my-experiments
Run a second experiment
In the command console, run:
guild run train.py x=0.2
Enter to confirm the operation.
train.py a second time using a new value for
Compare the two runs using
Guild Compare is spreadsheet-like application that lets you view
experiment results. Use the cursor keys to navigate to various
columns. Sort a column in ascending order by press
1 and in
descending order by pressing
? to view a list of key
bindings for Guild Compare.
When you're done comparing the runs, press
q to return to the
Run multiple trials using grid and random search
Guild supports running multiple trials using various methods.
First, use Guild to run three trials over a discrete search space for
x (i.e. a grid search, or parameter sweep) run:
guild run train.py x=[-2.0,0.0,2.0]
Enter to confirm. Guild generates three trials, one for each
specified value of
Next, run three trials using random search over a uniform
guild run train.py x=uniform[-1.0:1.0] --max-trials 3
Enter to run the trials. Guild generates another three trials,
according to the command line option
Compare the runs again:
guild compare --table
This time we use the
--table command line option for
tells Guild to print the results as a table rather than run
Sort the results by
loss in ascending order by running:
guild compare --table --min loss
Limit the results to the top three results (i.e. the three results
with the lowest loss) by using the
guild compare --table --min loss --top 3
Optimize loss using Bayesian methods
As you can see from the previous sections, Guild specializes in running, capturing, and comparing experiments. In this section, we demonstrate how this facility can be further used to optimize hyperparameters.
Let's try to find values for
x that minimize
loss. Because our
mock training script is noisy (to simulate machine learning processes
that are inherently noisy) we cannot find a single value for
loss is always lowest. Nonetheless, we can find a range of values
that are tend to produce better results.
In the command console, run:
guild run train.py x=uniform[-2.0:2.0] --optimizer bayesian --max-trials 20
Enter to continue. Guild runs 20 trials, trying values for
with the goal of minimizing
loss. By default, uild uses a Bayesian
method with Gaussian processes to explore values for
x that have a
higher likelihood of producing lower values of
loss. You can specify
other optimizers including
forest (decision trees) and
(gradient boosted trees).
NOTE: Guild's built-in Bayesian optimization support is implemented using the excellent skopt library. In future releases, Guild will support additional optimization libraries, including optimizers you define yourself.
For more information on using Bayesian optimizers in Guild, see Get Started - Bayesian Optimization .
Compare all of the runs:
In this case we run
compare in interactive mode. To quickly find
runs with the lowest loss, use the cursor keys to navigate to the
loss column and press
1 to sort in ascending order.
x that are close to
-0.3 should be listed toward the
The function defined in
train.py (again, a mock function used for
illustration purposes — it doesn't actually train anything) can be
used to plot the relationship between
We indeed see that loss is lowest where
x is around
q to exit Guild Compare.
If you want to delete a run, use:
guild runs rm RUN_ID_OR_INDEX
You can get the
guild runs. You can also specify a
INDEX, which is included in the runs list.
To delete all runs, use:
guild runs rm
You can restore deleted runs using the
For more complete coverage of managing runs with Guild, see Get Started - Manage Runs.
Refer to the Guild AI website for more information on Guild.
For more step-by-step tutorials, see the other Get Started guides:
For a complete list of commands supported by Guild, see: