COMP9444 Neural Networks and Deep Learning
Session 2, 2017
Project 1 - Digit Recognition
In this assignment, you will be implementing a single layer network, a two layer network and a convolutional network to classify handwritten digits. you will work with the MNIST dataset, a common dataset used to evaluate Machine Learning models. Preliminaries
Before commencing this assignment, you should download and install TensorFlow, and the appropriate python version. It is also helpful to complete the 'MNIST for beginners' tutorial located on the TensorFlow website https://www.tensorflow.org/get_started/mnist/beginners TensorFlow (TF) is an opensource library primarily used to construct, train and evaluate machine learning models. TF allows rapid development and supports automatic differentiation - meaning backprop is able to be done automatically for any model adequately defined. TF also abstracts away much of the low-level code required to set up training on GPU's; in many cases TF will automatically detect and utilize your computer's GPU if it has one. Central to the design of TF is the concept of a 'graph' - a low level representation of a model consisting of nodes and tensors. Broadly, implementing a TF model can be broken down into two sections; creating the graph, and training/testing it. This assignment is mainly concerned with graph creation. You can read more about the general structure of TensorFlow here.
Getting Started
You will see two files: train.py and hw1.py Now run train.py by typing
python3 train.py When run for the first time, train.py should create a new folder called data and download a copy of the MNIST dataset into this folder. All subsequent runs of train.py will use this local data. (Don't worry about the ValueError at this stage.) The file train.py contains the TensorFlow code required to create a session, build the graph, and run training and test iterations. It has been provided to assist you with the testing and evaluation of your model.
The file train.py calls functions defined in hw1.py and should not be modified during the course of the assignment. The only situation where you should modify train.py is when you need to switch between different network architectures. This can be done by setting the global variable on line 7:
network = "none" to any of the following values: network = "onelayer" network = "twolayer" network = "conv" The file hw1.py contains function definitions for the three networks to be created. You may also define helper functions in this file if necessary, as long as the original function names and arguments are not modified. Changing the function name, argument list, or return value will cause all tests to fail for that function. Your marks will be automatically generated by a test script, which will evaluate the correctness of the implemented networks. For this reason, it is important that you stick to the specification exactly.
The functions input_placeholder() and target_placeholder() specify the inputs and outpus of your networks in the TensorFlow graph. They have been implemented for you. In addition, there is a function train_step() that passes batches of images to the constructed TensorFlow Graph during training. It's implementation should help you understand the shape and structure of the actual data that is being provided to the model.
Unless otherwise specified, the underlying type (dtype) for each TF object should be float32. INPUT_SIZE, where it appears in comments, refers to the length of a flattened single image; in this case 784. OUTPUT_SIZE, where it appears in comments, refers to the length of a one-hot output vector; in this case 10.
In the provided file hw1.py, detailed specifications are provided in the comments for each function.
Write a function onelayer(X, Y, layersize=10) which creates a TensorFlow model for a one layer neural network (sometimes also called logistic regression). Your model should consist of one fully connected layer with weights w and biases b, using softmax activation. Your function should take two parameters X and Y that are TensorFlow placeholders as defined in input_placeholder() and target_placeholder(). It should return varibles w, b, logits, preds, batch_xentropy and batch_loss, where:
w and b are TensorFlow variables representing the weights and biases, respectively logits and preds are the input to the activation function and its output xentropy_loss is the cross-entropy loss for each image in the batch batch_loss is the average of the cross-entropy loss for all images in the batch Change line 7 of train.py to network = "onelayer" and test your network on the MNIST dataset by typing python3 train.py It should achieve about 92% accuracy after 5 epochs of training.
Create a TensorFlow model for a Neural Network with two fully connected layers of weights w1, w2 and biases b1, b2, with ReLU activation functions on the first layer, and softmax on the second. Your function should take two parameters X and Y that are TensorFlow placeholders as defined in input_placeholder() and target_placeholder(). It should return varibles w1, b1, w2, b2, logits, preds, batch_xentropy and batch_loss, where: w1 and b1 are TensorFlow variables representing the weights and biases of the first layer w2 and b2 are TensorFlow variables representing the weights and biases of the second layer logits and preds are the inputs to the final activation functions and their output xentropy_loss is the cross-entropy loss for each image in the batch batch_loss is the average of the cross-entropy loss for all images in the batch Change line 7 of train.py to network = "twolayer" and test your network on the MNIST dataset by typing python3 train.py
Create a TensorFlow model for a Convolutional Neural Network. This network should consist of two convolutional layers followed by a fully connected layer of the form: conv_layer1 → conv_layer2 → fully-connected → output Your function should take two parameters X and Y that are TensorFlow placeholders as defined in input_placeholder() and target_placeholder(). It should return varibles conv1, conv2, w, b, logits, preds, batch_xentropy and batch_loss, where: conv1 is a convolutional layer of convlayer_sizes[0] filters of shape filter_shape conv2 is a convolutional layer of convlayer_sizes[1] filters of shape filter_shape w and b are TensorFlow variables representing the weights and biases of the final fully connected layer logits and preds are the inputs to the final activation functions and their output xentropy_loss is the cross-entropy loss for each image in the batch batch_loss is the average of the cross-entropy loss for all images in the batch
Hints:
use tf.layer.conv2d the final layer is very similar to the onelayer network, except that the input will be from the conv2 layer. If you reshape the conv2 output using tf.reshape, you should be able to call onelayer() to get the final layer of your network Change line 7 of train.py to network = "conv" and test your network on the MNIST dataset by typing python3 train.py It may take several minutes to run, depending on your processor.
All TensorFlow objects, if not otherwise specified, should be explicity created with tf.float32 datatypes. Not specifying this datatype for variables and placeholders will cause your code to fail some tests. TensorFlow provides multiple API's, at various levels of abstraction. For the specified functionality in this assignment, there are generally high level TensorFlow library calls that can be used. As we are assessing TensorFlow, functionality that is technically correct but implemented manually, using a library such as numpy, will fail tests. If you find yourself writing 50+ line methods, it may be a good idea to look for a simpler solution.
In addition to the output of train.py, you can view the progress of your models and the created TensorFlow graph using the TensorFlow visualization platform, TensorBoard. After beginning training, run the following command from the src directory: python3 -m tensorflow.tensorboard --logdir=./summaries Depending on your installation, the following command might also work: tensorboard --logdir=./summaries open a Web browser and navigate to http://localhost:6006 you should be able to see a plot of the train and test accuracies in TensorBoard if you click on the histogram tab you'll also see some histograms of your weights, biases and the pre-activation inputs to the softmax in the final layer Make sure you are in the same directory from which train.py is running. Don't worry if you are unable to get TensorBoard working; it is not required to complete the assignment, but it can be a useful tool to monitor training, so it is probably worth your while becoming familiar with it. Click here for more information:
Single Layer
Two Layer Network
Conv Layer
