User acceptance tests
These are run in order as a part of Guild AI's user acceptance test.
This file is used to run the tests. Any modifications to this file will modify the acceptance test.
These tests expect Guild to be installed.
Env check and unit tests
Check the environment and run unit tests.
Run commands with missing arguments and other errors.
Guild behavior without packages, models, or runs.
Before we start using Guild to train models, we use it to example an empty environment. This includes demonstrating all of its list commands and helpfulness to the user for operations on missing models.
Training Guild package models
Installing and training a model from a Guild package.
One of Guild's main features is the ability to install and train models from packages. We start here with various packages that exercise core Guild functionality.
mnist package contains two models that roughly correspond to the
TensorFlow expert and intro tutorial examples. These are named
First we install the package.
Then we confirm we have the expected models and operations.
Next we train the
softmax model and view the results.
Finally we'll uninstall the
mnist package and verify that its
associated models and operations removed.
hello package is a distributed version of the
This package provide a single model with several operations that exercise core Guild functionality. For simplicity all of the tests are contained in a single file file.
Guild provides Keras packages under the
namespace. These tests demonstrate installing and training the
keras-mlp model provided in the
First we'll install the package and confirm the availbility of models and operations.
Next we'll train the
Training Guild example models
Installing and training Guild examples.
Guild examples are similar to package source, but they are indepedent and may have different features and behavior. We test them here as a matter of coverage and to take advantage of any differences.
mnist example is similar to the package of the same name. It
maintains its lineage from privious Guild releases by providing
intro models, which correspond to the TensorFlow
We start by listing the example models and operations.
Next we train
Once we have a trained model we can run the
evaluate operation on
As a final check, we want to ensure that the example operations used the data provided by the model file resource.
The mnist example defines two models containing 'mnist', which lets us test Guild's handling of an ambiguous model spec.
hello example is a non ML model that simply prints messages to
the console. While trivial in this respect, its various operation
demonstrate important Guild features.
For simplicity we maintain a single test file.
keras example demonstrates how plugins can enumerate models in a
directory and provide operations for compatible models.
For our tests of the example model, we'll first uninstall some packages that were installed in pervious tests.
And test the
Creating and installing packages.
Guild packages are generated from source directories that contain guildfiles with a package definition. They may contain models and their required scripts as well as package resoures.
For our tests, we'll build and install packages using sources from the
guild-index GitHub repository.
Here we build and install the
mnist package, which is otherwise
identical to the distributed package we tested earlier.
Tests covering miscellaneous behavior.
Guild behavior with various error cases.