Permalink
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
230 lines (149 sloc) 7.58 KB

User acceptance tests

These are run in order as a part of Guild AI's user acceptance test.

This file is used to run the tests. Any modifications to this file will modify the acceptance test.

These tests expect Guild to be installed.

Env check and unit tests

Check the environment and run unit tests.

Command errors

Run commands with missing arguments and other errors.

Empty environment

Guild behavior without packages, models, or runs.

Before we start using Guild to train models, we use it to example an empty environment. This includes demonstrating all of its list commands and helpfulness to the user for operations on missing models.

Training Guild package models

Installing and training a model from a Guild package.

One of Guild's main features is the ability to install and train models from packages. We start here with various packages that exercise core Guild functionality.

mnist package

The mnist package contains two models that roughly correspond to the TensorFlow expert and intro tutorial examples. These are named cnn and softmax respectively.

First we install the package.

Then we confirm we have the expected models and operations.

Next we train the softmax model and view the results.

Finally we'll uninstall the mnist package and verify that its associated models and operations removed.

hello

The hello package is a distributed version of the hello example.

This package provide a single model with several operations that exercise core Guild functionality. For simplicity all of the tests are contained in a single file file.

keras.mnist

Guild provides Keras packages under the keras package namespace. These tests demonstrate installing and training the keras-mlp model provided in the keras.mnist package.

First we'll install the package and confirm the availbility of models and operations.

Next we'll train the mnist-mlp model.

Training Guild example models

Installing and training Guild examples.

Guild examples are similar to package source, but they are indepedent and may have different features and behavior. We test them here as a matter of coverage and to take advantage of any differences.

mnist example

The mnist example is similar to the package of the same name. It maintains its lineage from privious Guild releases by providing expert and intro models, which correspond to the TensorFlow tutorial examples.

We start by listing the example models and operations.

Next we train intro.

Once we have a trained model we can run the evaluate operation on it.

As a final check, we want to ensure that the example operations used the data provided by the model file resource.

The mnist example defines two models containing 'mnist', which lets us test Guild's handling of an ambiguous model spec.

hello example

The hello example is a non ML model that simply prints messages to the console. While trivial in this respect, its various operation demonstrate important Guild features.

For simplicity we maintain a single test file.

keras example

The keras example demonstrates how plugins can enumerate models in a directory and provide operations for compatible models.

For our tests of the example model, we'll first uninstall some packages that were installed in pervious tests.

And test the keras example.

Packaging

Creating and installing packages.

Guild packages are generated from source directories that contain guildfiles with a package definition. They may contain models and their required scripts as well as package resoures.

For our tests, we'll build and install packages using sources from the guild-index GitHub repository.

Here we build and install the mnist package, which is otherwise identical to the distributed package we tested earlier.

Other tests

Tests covering miscellaneous behavior.

alt-run-dir

Error cases

Guild behavior with various error cases.

Packages

slim.resnet