3D CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3D Convolutional Neural Network with tflearn\n",
    "----------------------------------------------------------------\n",
    "\n",
    "This template does not include pre-processing. Our dimensions will be height, width, and time.\n",
    "We will be using tflearn (abstraction of machine learning library tensorflow)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing libraries \n",
    "import numpy as np\n",
    "import tflearn\n",
    "from tflearn.layers.conv import conv_3d, max_pool_3d\n",
    "from tflearn.layers.core import input_data, dropout, fully_connected\n",
    "from tflearn.layers.estimator import regression\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining constants\n",
    "\n",
    "Across all of our data, we will need to have the same dimensions of image size and time. These can be modified during preprocessing, but at this step, the input dimensions will be fixed. \n",
    "\n",
    "We also have constants that we can adjust in our model - learning rate, number of convolutional layers, and number of pooling layers. If there are fewer pooling layers than convolutional layers, the pooling layers will be inserted periodically between to convolutional layers. For example, if there is a 3:1 ratio of convolutional layers to pooling layers, there will be one pooling layer inserted at the same position per three convolutional layers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining constants\n",
    "IMG_SIZE = 50 # 50 x 50 image, for example\n",
    "TIME_LEN = 5 # 5 time points, for example\n",
    "LR = 1e-3 # learning rate, for example\n",
    "CONV_LAYERS = 1\n",
    "POOL_LAYERS = 1\n",
    "NO_CHANNELS = 1 # how many input streams i.e. a network with 3 channels could have RGB, depth, and body skeleton streams\n",
    "VERSION = 1 # to keep track of changes in a particular model\n",
    "NO_SIGNS = 26 # number of signs to learn - i.e. the alphabet\n",
    "N_EPOCH = 10 # number of epochs\n",
    "MODEL_NAME = '3dconv-{}--{}--{}--{}--{}-.model'.format(LR, CONV_LAYERS, POOL_LAYERS, NO_CHANNELS, VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building our neural network\n",
    "#### Input layer\n",
    "\n",
    "Our first layer will be the input layer. The tflearn function to create the input layer is:\n",
    "```\n",
    "input_data(shape = [batch, depth, height, width, number_of_channels], name = 'optional_name_of_layer')\n",
    "\n",
    "```\n",
    "The batch parameter should be `None`.\n",
    "\n",
    "#### Convolutional layer\n",
    "\n",
    "We then need to create our first 3D convolutional layer. The tflearn funtion to create the 3D convolutional layer is:\n",
    "```\n",
    "conv_3d(input, number_of_filters, filter_size, strides = number_of_strides, padding = 'padding', activation = 'activation_function', bias = True or False, weights_init = 'initial_weights', bias_init = 'initial bias', regularizer = None, weight_decay = weight_decay, trainable = True or False, restore = True or False, reuse = True or False, scope = 'optional_layer_scope', name = 'optional_name_of_layer')\n",
    "```\n",
    "\n",
    "Parameter definitions\n",
    "* **input**: 5D tensor with dimensions \\[batch, depth, height, width, number_of_channels\\]\n",
    "    * takes the output from a previous layer - for example the input layer\n",
    "* **number_of_filters**: an integer specifying the number of convolutional filters\n",
    "    * the depth of the output is given by the number of filters\n",
    "* **filter_size**: an integer or list of integers indicating the size of the filter\n",
    "    * for a filter size of 2x2x2, one could enter 2, or \\[2 2 2\\]\n",
    "    * in 2D convolutional neural networks, common filter sizes are 2x2 and 3x3\n",
    "    * the best 3D CNN size is 3x3x3\n",
    "* **strides**: an integer or list of integers indicating the number of strides of the filter\n",
    "    * default = \\[1 1 1 1 1\\]: 1x1x1 stride, can also be written as 1\n",
    "    * for a stride of 2x2x2, one can write \\[2 2 2 2 2\\], or simply 2\n",
    "    * the most common stride is 1\n",
    "* **padding**: either 'same' or 'valid'\n",
    "    * default = 'same'\n",
    "    * 'same' padding: pads the input with zeros so that the width and height dimensions of the input and output are the same\n",
    "    * 'valid' padding: there is no padding, and the width and height dimensions of the output are smaller\n",
    "    * 'same' is most commonly used\n",
    "* **activation**: either 'linear', 'tanh', 'sigmoid', 'softmax', 'softplus', 'softsign', 'relu', 'relu6', 'leaky_relu', 'prelu', 'elu', 'crelu', or 'selu'\n",
    "    * 'linear' by default\n",
    "    * the most common activation used in a convolutional layer is 'relu'\n",
    "    * 'linear': computes f(x) = x\n",
    "        * used when a straight line is a good approximation of the relationship between input and output\n",
    "        * works for linear regression analysis, but has very poor performance in convolutional neural networks\n",
    "    * 'tanh': computes the hyperbolic tangent tanh(x) element-wise \n",
    "    * 'sigmoid': computes the sigmoid function sigmoid(x) element-wise\n",
    "    * 'softmax': computes softmax\\[i, j\\] = exp(logits\\[i, j\\]) / sum(exp(logits\\[i\\])) for each batch i and class j\n",
    "        * a useful activation function for the fully connected layer \n",
    "    * 'softplus': computes log(exp(features) + 1)\n",
    "    * 'softsign': computes features / (abs(features) + 1)\n",
    "    * 'relu': computes the rectified linear function max(features, 0)\n",
    "    * 'relu6': computes min(relu, 6) = min(max(features, 0), 6)\n",
    "    * other activation functions:\n",
    "        * 'leaky_relu'\n",
    "        * 'prelu'\n",
    "        * 'elu'\n",
    "        * 'crelu'\n",
    "        * 'selu'\n",
    "* **bias**: either True or False\n",
    "    * indicates whether to use a bias or not\n",
    "    * default = True\n",
    "* **weights_init**: either 'zeros', 'uniform', 'uniform_scaling', 'normal', 'truncated_normal', 'xavier', or 'variance_scaling' \n",
    "    * default = 'truncated_normal'\n",
    "    * 'zeros': sets all weights to zero\n",
    "    * 'uniform': sets all weights to random values from a uniform distribution\n",
    "    * 'uniform_scaling': sets all weights to random values from a uniform distribution without scaling variance\n",
    "        * it is better to initialise the weights with uniform_scaling rather than uniform to keep the scale of the input variance constant\n",
    "    * 'normal': sets all weights to random values from a normal distribution\n",
    "    * 'truncated_normal': sets all weights to random values from a normal distribution with a specified mean and standard deviation\n",
    "        * values more than 2 standard deviations from the mean are dropped and re-picked\n",
    "        * the default mean is 0.0, and the default standard deviation is 0.02\n",
    "    * 'xavier': performs \"Xavier\" initialisation for weights\n",
    "        * designed to keep the scale of the gradients roughly the same in all layers\n",
    "        * http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf\n",
    "    * 'variance_scaling': initialises weights without scaling variance, has different modes\n",
    "* **bias_init**: either 'zeros', 'uniform', 'uniform_scaling', 'normal', 'truncated_normal', 'xavier', or 'variance_scaling'\n",
    "    * the initialisers (zeros, uniform, etc.) work the same as they do for the weights\n",
    "    * the default is zeros - this is a good starting point for the biases\n",
    "* **regularizer**: adds a regulariser to the weights\n",
    "    * by default regularizer = None \n",
    "* **weight_decay**: specifies weight decay of regulariser\n",
    "    * can be ignored unless we decide to modify the regulariser\n",
    "    * by default 0.001\n",
    "* **trainable**: indicates whether the weights are trainable - either True or False\n",
    "    * by default True, which is what we want\n",
    "* **restore**: indicates whether the layer weights should be restored when loading the model - either True or False\n",
    "    * by default True, which is what we want\n",
    "* **reuse**: indiates whether the variables in the layer should be shared with other layers in the scope\n",
    "    * by default False, we can ignore this unless we would like to share variables across layers\n",
    "* **scope**: optionally defines the layer's scope - used for sharing variables across layers\n",
    "* **name**: an optional name for the layer\n",
    "    * default = 'Conv3D'\n",
    "\n",
    "The main parameters we will be interested in changing are input, number of filters, filter size, strides, and activation. \n",
    "\n",
    "In 2 dimensions, there are some important equations to keep in mind when changing parameters:\n",
    "* output width = (input width - filter size + 2 * padding)/strides + 1\n",
    "* output height = (input height - filter size + 2 * padding)/strides + 1\n",
    "* output depth = number of filters\n",
    "\n",
    "By setting padding to 'same', it ensures that the output and input width and height dimensions are the same by adjusting the padding parameter above.\n",
    "\n",
    "#### Pooling layer\n",
    "\n",
    "After one or more convolutional layers, we can add a pooling layer. In some cases, reducing pooling can improve results. Pooling does, however, have the advantage of reducing size. It is still generally convention to use pooling layers.\n",
    "\n",
    "The tflearn function for pooling in 3D is:\n",
    "```\n",
    "max_pool_3d(input, kernel_size, strides = number_of_strides, padding = 'padding', name = 'optional_name_of_layer')\n",
    "```\n",
    "\n",
    "Parameter definitions:\n",
    "* **input**: 5D tensor with dimensions \\[batch, depth, height, width, number_of_channels\\]\n",
    "    * takes the output from a previous layer - for example a convolutional layer\n",
    "* **kernel_size**: an integer or list of integers indicating the pooling kernel size \n",
    "    * default = 1, also written as \\[1 1 1\\] for a 1x1 kernel\n",
    "    * the most common kernel size is 2 (2x2x2 kernel)\n",
    "* **strides**: an integer or list of integers indicating the number of strides of the kernel\n",
    "    * default = \\[1 1 1 1 1\\]: 1x1 stride, can also be written as 1\n",
    "    * for a stride of 2x2, one can write \\[2 2 2 2 2\\], or simply 2\n",
    "    * the most common stride is 2\n",
    "* **padding**: either 'same' or 'valid'\n",
    "    * default = 'same'\n",
    "* **name**: an optional name for the layer\n",
    "    * default = 'MaxPool3D'\n",
    "\n",
    "In two dimensions there are some important equations to keep in mind when changing parameters:\n",
    "* output width = (input width - kernel size)/strides + 1\n",
    "* output height = (input height - kernel size)/strides + 1\n",
    "* output depth = input depth\n",
    "\n",
    "Two common settings for the parameters are (in 2 dimensions)\n",
    "* most common: filter size = 2, strides = 2\n",
    "* overlapping pooling: filter size = 3, strides = 2\n",
    "\n",
    "#### Fully connected layer\n",
    "\n",
    "We will have two or three fully connected layers. The first fully connected layer will connect to the last convolutional or pooling layer. The number of neurons (units) in the first fully connected layer is generally the number of filters in the previous convolutional layer multiplied by 8. The first layer generally has an rectified linear ('relu') or hyberbolic tangent ('tanh') activation function. If there are three fully connected layers, the second fully connected layer will have the same number of units and activation function as the first. The last layer will have the same number of neurons as the different classifications - i.e. if we are classifying the letters of the alphabet, there will be 26 neurons. The activation function of the last layer is 'softmax' in all of the examples I have seen. There is sometimes dropout between the fully connected layers. This will be discussed further in the dropout section.\n",
    "\n",
    "The tflearn function for the fully connected layer is:\n",
    "```\n",
    "fully_connected(input, number_of_units, activation = 'activation_function', bias = True or False, weights_init = 'initial_weights', bias_init = 'initial bias', regularizer = None, weight_decay = weight_decay, trainable = True or False, restore = True or False, reuse = True or False, scope = 'optional_layer_scope', name = 'optional_name_of_layer')\n",
    "```\n",
    "\n",
    "Parameter definitions:\n",
    "* **input**: 5D tensor with dimensions \\[batch, depth, height, width, number_of_channels\\] flattened into a 2D tensor\n",
    "    * * takes and flattens the output from a previous layer - for example a pooling layer\n",
    "* **number_of_units**: integer indicating the number of neurons in the fully connected layer\n",
    "    * fully connected layers before the last layer have an arbitrary number of neurons - but it is good to follow some standards (i.e. number of neurons = number of filters in previous conv layer * 8)\n",
    "    * last fully connected layer has as many neurons as classes (i.e. 26 - see above)\n",
    "* **activation**: see activation definition above under 'Convolutional Layer'\n",
    "    * for the fully connected layers excluding the last layer, the activation function is generally the same as the activation function used in the convolutional layers\n",
    "    * for the last fully connected layer, the activation function is usually 'softmax'\n",
    "* **bias**: either True or False\n",
    "    * indicates whether to use a bias or not\n",
    "    * default = True\n",
    "* **weights_init**: see weights_init definition above under 'Convolutional Layer'\n",
    "* **bias_init**: see bias_init definition above under 'Convolutional Layer'\n",
    "* **regularizer**: adds a regulariser to the weights\n",
    "    * by default regularizer = None \n",
    "* **weight_decay**: specifies weight decay of regulariser\n",
    "    * can be ignored unless we decide to modify the regulariser\n",
    "    * by default 0.001\n",
    "* **trainable**: indicates whether the weights are trainable - either True or False\n",
    "    * by default True, which is what we want\n",
    "* **restore**: indicates whether the layer weights should be restored when loading the model - either True or False\n",
    "    * by default True, which is what we want\n",
    "* **reuse**: indiates whether the variables in the layer should be shared with other layers in the scope\n",
    "    * by default False, we can ignore this unless we would like to share variables across layers\n",
    "* **scope**: optionally defines the layer's scope - used for sharing variables across layers\n",
    "* **name**: an optional name for the layer\n",
    "    * default = 'FullyConnected'\n",
    "    \n",
    "#### Dropout layer\n",
    "The dropout layer is an optional layer inserted between fully connected layers. A dropout layer ignores some randomly selected neurons, meaning that they will not be considered in a particular backward or forward pass. At each training stage, the randomly selected neurons are dropped out of the network. Dropout is used to prevent overfitting. Overfitting is when the neurons in a layer become dependent on each other during training, reducing their individual power. The number of neurons dropped is determined by the keep probability. In many cases 0.8 is the best keep probability, but many networks also use a keep probability of 0.5. \n",
    "\n",
    "The tflearn function for the dropout layer is:\n",
    "```\n",
    "dropout(input, keep_probability, noise_shape=None, name='optional_layer_name')\n",
    "```\n",
    "\n",
    "Parameter definitions:\n",
    "* **input**: a tensor\n",
    "    * the 2D tensor output of a fully connected layer (dimensions = \\[samples, number_of_units\\])\n",
    "* **keep_probability**: a float representing the probability that each neuron is kept\n",
    "* **noise_shape**: a one dimensional tensor, representing the shape of the neurons kept and dropped\n",
    "    * by default, each element is kept or dropped independently\n",
    "    * if noise shape is specified, only dimensions with noise_shape\\[i\\] = shape(input)\\[i\\] will be kept or dropped independently\n",
    "    * the documentation's example: if shape(input) = \\[k, l, m, n\\] and noise_shape = \\[k, 1, 1, n\\], each batch and channel component will be kept independently and each row and column will be kept or not kept together\n",
    "    * at this point we will ignore noise shape\n",
    "* **name**: an optional name for the layer\n",
    "    * default = 'Dropout'\n",
    "    \n",
    "#### Regression layer\n",
    "\n",
    "The regression layer is used to apply linear or logistic regression to the output of the convolutional neural network (the final fully connected layer). A gradient descent optimiser needs to be specified that will minimise the loss function (also needs to be specified). The regression layer works in specifying *how* the convolutional network is trained, rather than being considered a layer of the convolutional neural network.\n",
    "\n",
    "The tflearn function for the regression layer is:\n",
    "```\n",
    "regression(input, optimizer = 'gradient_descent_optimiser', loss='loss_function', metric = 'metric_used', learning_rate = learning_rate, dtype=tf.float32, batch_size = batch_size, shuffle_batches = True or False, to_one_hot = True or False, n_classes = None, trainable_vars = None, restore = True or False, op_name = None, validation_monitors = None, validation_batch_size = None, name = 'optional_name_of_layer')\n",
    "```\n",
    "\n",
    "Parameter definitions:\n",
    "* **input**: 2-dimensional tensor from the last fully connected layer\n",
    "* **optimizer**: either 'sgd', 'rmsprop', 'adam', 'momentum', 'adagrad', 'ftrl', 'adadelta', 'proxi_adagrad', or 'nesterov'\n",
    "    * 'sgd': uses stochastic gradient descent\n",
    "        * the basic optimiser for neural networks\n",
    "        * can specify learning rate decay - by default none, but it's often recommended to lower the learning rate as training progresses\n",
    "        * to specify learning rate decay: \n",
    "            * `sgd = tflearn.optimizers.SGD(learning_rate=LR, lr_decay=0.5, decay_step=100, staircase=False, use_locking=False, name='SGD') `\n",
    "            * `conv_net = regression(conv_net, optimizer = sgd) `\n",
    "    * 'rmsprop': maintain a moving average of the square of gradients, and divide the gradient by the root of this average\n",
    "    * 'adam': calculates an exponential moving average of the gradient and the squared gradient\n",
    "        * parameters beta1 and beta2 control the decay rates of these moving averages\n",
    "        * combines the advantages of the adaptive gradient algorithm (adagrad) and root mean square propagation (rmsprop)\n",
    "        * most popular optimiser for CNNs\n",
    "    * 'momentum': helps accelerate stochastic gradient descent in the relevant direction and dampens oscillations\n",
    "        * can also specify learning rate decay - by default none\n",
    "    * 'adagrad': improves stochastic gradient descent by adapting the learning rate to the parameters\n",
    "        * decreases learning rates for parameters associated with commonly occurring features\n",
    "        * increases learning rates for parameters associated with infrequently occurring features\n",
    "    * 'ftrl': implements the Ftrl-proximal algorithm (Follow-the-regularized-leader)\n",
    "        * uses its own global base learning rate and can behave like Adagrad with learning_rate_power=-0.5, or like gradient descent with learning_rate_power=0.0.\n",
    "        * how to change learning power:\n",
    "            * `ftrl = tflearn.optimizers.Ftrl(learning_rate=0.01, learning_rate_power=-0.1)`\n",
    "            * `conv_net = regression(conv_net, optimizer = ftrl)`\n",
    "    * 'adadelta': extension of adagrad that attempts to reduce the aggressiveness of adagrad decrease in learning rate\n",
    "        * instead of storing all past squared gradients, adadelta restricts the window of stored past squared gradients to some fixed size w\n",
    "    * 'proxi_adagrad': see http://papers.nips.cc/paper/3793-efficient-learning-using-forward-backward-splitting.pdf\n",
    "    * 'nesterov': very similar to momentum optimiser, but calculates the gradient at the approximated new position rather than the current position\n",
    "* **loss**: either 'softmax_categorical_crossentropy', 'categorical_crossentropy', 'binary_crossentropy', 'weighted_crossentropy', 'mean_square', 'hinge_loss', 'roc_auc_score', or 'weak_cross_entropy_2d'\n",
    "    * note: in the tflearn the loss functions are under objectives, and NOT losses\n",
    "    * default: 'categorical_crossentropy'\n",
    "    * 'softmax_categorical_crossentropy': computes softmax cross entropy between the predicted output value and the actual output value\n",
    "    * 'categorical_crossentropy': computes cross entropy between the predicted output value and the actual output value\n",
    "        * we will be using this for now\n",
    "    * 'binary_crossentropy': computes sigmoid cross entropy between the predicted output value and the actual output value\n",
    "    * 'weighted_crossentropy': computes a weighted sigmoid cross entropy between the predicted output value and the actual output value\n",
    "    * 'mean_square': computes the mean square error between the predicted output value and the true output value\n",
    "    * 'hinge_loss': computes the loss using the hinge loss function \n",
    "    * 'roc_auc_score': approximates the area under the curve score, measuring overall performance for a full range of threshold levels\n",
    "    * 'weak_cross_entropy_2d': calculates the semantic segmentation using weak softmax cross entropy loss\n",
    "* **metric**: either 'acc', 'top5', 'r2', 'weighted_r2'\n",
    "    * default is 'acc'\n",
    "    * 'acc': computes the accuracy of the model - this is what we will use\n",
    "    * 'top5': computes top-k mean accuracy of the model\n",
    "    * 'r2': computes the coefficient of determination - for linear regression\n",
    "    * 'weighted_r2': also computes the coefficient of determination - for linear regression\n",
    "* **learning_rate**: the optimiser's learning rate (the constant LR)\n",
    "    * default is 0.001\n",
    "    * we will be adjusting learning rate\n",
    "    * if the learning rate is too small, gradient descent is very slow\n",
    "    * if the learning rate is too high, gradient descent can overshoot the minimum\n",
    "        * may fail to converge\n",
    "    * a good range is 0.001 to 0.01\n",
    "* **dtype**: layer's placeholder type\n",
    "    * default = tf.float32\n",
    "    * we can ignore this\n",
    "* **batch_size**: batch size of data for training \n",
    "    * default = 64\n",
    "    * we should be able to ignore this\n",
    "* **shuffle_batches**: whether to shuffle the batches or not at each epoch\n",
    "    * default = True\n",
    "    * we will always want this to be true\n",
    "* **to_one_hot**: if True, labels will be encoded to one hot vectors\n",
    "    * default = False\n",
    "    * we should be able to ignore this\n",
    "* **n_classes**: only applicable if to_one_hot = True\n",
    "    * default = None\n",
    "    * we should be able to ignore this \n",
    "* **trainable_vars**: list of variables, that if specified, will be the only variables that have updated weights\n",
    "    * otherwise all variables will be updated - what we want\n",
    "    * leave this\n",
    "* **restore**: determines whether variables related to the optimiser will be restored when loading a model\n",
    "    * default = True - what we want\n",
    "    * leave this\n",
    "* **op_name**: optional name for layer's optimiser\n",
    "* **validation_monitors**: list of variables to compute during validation\n",
    "    * default = None\n",
    "    * we should be able to ignore this\n",
    "* **validation_batch_size**: specifies batch size for validation\n",
    "    * default = None\n",
    "    * we should be able to ignore this\n",
    "* **name**: optional name for layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv_net = input_data(shape = [None, TIME_LEN, IMG_SIZE, IMG_SIZE, NO_CHANNELS], name = 'input')\n",
    "\n",
    "conv_net = conv_3d(conv_net, 32, 3, strides = 1, padding = 'same', activation = 'relu')\n",
    "\n",
    "conv_net = max_pool_3d(conv_net, 2, strides = 2)\n",
    "\n",
    "conv_net = fully_connected(conv_net, 256, activation = 'relu')\n",
    "\n",
    "conv_net = dropout(conv_net, 0.8)\n",
    "\n",
    "conv_net = fully_connected(conv_net, NO_SIGNS, activation = 'softmax')\n",
    "\n",
    "conv_net = regression(conv_net, optimizer = 'adam', learning_rate = LR, loss = 'categorical_crossentropy', name = 'targets')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating and training our model\n",
    "#### Creating the model\n",
    "```\n",
    "tflearn.DNN(network, clip_gradients = gradient_clipping_float, tensorboard_verbose = tensorboard_verbose_level_integer, tensorboard_dir = 'directory_to_store_tensorboard_logs', checkpoint_path = None, best_checkpoint_path = None, max_checkpoints = None, session = None, best_val_accuracy = minimum_accuracy_for_best_checkpoint_path)\n",
    "```\n",
    "Parameter definitions:\n",
    "\n",
    "* **network**: our fully built neural network \n",
    "* **clip_gradients**: determines the threshold that gradients should be clipped to\n",
    "    * default = 5.0\n",
    "    * this prevents the gradient value from getting to large and causing overflow or causing the model to overshoot the minima\n",
    "    * documentation is farily scarce on this, but I think it's safe to just ignore this\n",
    "* **tensorboard_verbose**: tensorboard summary verbose level (0, 1, 2, or 3)\n",
    "    * 0: Loss, Accuracy\n",
    "    * 1: Loss, Accuracy, Gradients\n",
    "    * 2: Loss, Accuracy, Gradients, Weights\n",
    "    * 3: Loss, Accuracy, Gradients, Weights, Activations, Sparsity\n",
    "    * default = 0\n",
    "* **tensorboard_dir**: file directory to store tensorboard logs in \n",
    "* **checkpoint_path**: path to store model checkpoints in\n",
    "    * if None, no model checkpoints will be saved\n",
    "    * default = None\n",
    "* **best_checkpoint_path**: path to store the checkpoint where the validation rate reaches the highest point in the session\n",
    "    * validation rate must also be above the best_val_accuracy\n",
    "    * default = None\n",
    "* **max_checkpoints**: maximum number of checkpoints\n",
    "    * if None, no limit\n",
    "    * default = None\n",
    "* **session**: a session for running operations\n",
    "    * if None, a new session will be created\n",
    "    * when providing a session, variables must have already been initialised\n",
    "    * default = None (what we want)\n",
    "* **best_val_accuracy**: the minimum validation accuracy needed to save a checkpoint in best_checkpoint_path\n",
    "    * default = 0.0\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tflearn.DNN(conv_net, tensorboard_dir = 'log')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you have already saved a checkpoint - network already trained\n",
    "if os.path.exists('{}.meta'.format(MODEL_NAME)):\n",
    "    model.load(MODEL_NAME)\n",
    "    print('model loaded')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to reset tensorboard\n",
    "\n",
    "# import tensorflow as tf\n",
    "# tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fitting the model\n",
    "\n",
    "This trains the model using inputs (X) and targets (Y).\n",
    "\n",
    "```\n",
    "model.fit(X_inputs, Y_targets n_epoch = number_of_epochs, validation_set = ({'name_of_input_layer':test_inputs}, {'name_of_estimator_layer':test_targets}), show_metric = True or False, batch_size = None, shuffle = None, snapshot_epoch = True or False, snapshot_step = steps_between_snapshots, excl_trainops = None, validation_batch_size = None, run_id = None, callbacks = [])\n",
    "```\n",
    "Parameter definitions:\n",
    "\n",
    "* **X_inputs**: the features (i.e. videos) in an array, list of arrays, or dictionary\n",
    "    * our data will be in a dictionary - {'name_of_input_layer':X}\n",
    "        * the name of the input layer is the keys \n",
    "        * X is a numpy array of features (i.e. videos) \n",
    "* **Y_targets**: the labels (i.e. which letter the video shows) in an array, list of arrays, or dictionary\n",
    "    * our data will be in a dictionary - {'name_of_estimator_layer':Y}\n",
    "        * the name of the estimator layer (for us the estimator layer is the regression layer) is the keys \n",
    "        * Y is a numpy array of labels \n",
    "* **n_epoch**: the number of epochs to run\n",
    "    * default = 10\n",
    "* **validation_set**: test features and labels to determine the accuracy of the model\n",
    "    * input and targets are shown in the dictionary format, but can also be in the other formats that X_inputs and Y_targets are in\n",
    "    * default = None\n",
    "* **show_metric**: whether to display accuracy at each step\n",
    "    * True = display accuracy\n",
    "    * default = False\n",
    "* **batch_size**: if an integer (not None), overrides the estimator's batch size by the integer value\n",
    "    * default = None\n",
    "    * we can ignore this\n",
    "* **shuffle**: True or False or None - if True or False, overrides the estimator's shuffle parameter by True or False\n",
    "    * default = None\n",
    "    * we can ignore this\n",
    "* **snapshot_epoch**: True or False\n",
    "    * if True, snapshots the model (evaluating it with the validation set) at the end of every epoch\n",
    "    * default = True\n",
    "* **snapshot_step**: if an integer (not None), the model takes a snapshot every integer value steps\n",
    "    * default = None\n",
    "    * it is useful to set this to an integer value if the epochs take a long time to run or the batches are large\n",
    "* **excl_trainops**: a list of training operations to exclude from the training process\n",
    "    * default = None\n",
    "    * we can ignore this\n",
    "* **validation_batch_size**: if an integer (not None), overrides the estimator's validation batch size by the integer value\n",
    "    * default = None\n",
    "    * we can ignore this\n",
    "* **run_id**: name for the run\n",
    "    * default = None\n",
    "    * it may be useful to change this when we're using tensorboard\n",
    "* **callbacks**: custom callbacks to use in training\n",
    "    * ignore this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit({'input':X}, {'targets':Y}, n_epoch = N_EPOCH, validation_set = ({'input':test_X}, {'targets':test_Y}), \n",
    "          snapshot_step = 50, show_metric = True, run_id = MODEL_NAME) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using tensorboard\n",
    "Copy and paste this into command prompt:\n",
    "```\n",
    "tensorboard --logdir = foo:C:/Users/gabri/Documents/Python Scripts/Project/3D CNN/log\n",
    "```\n",
    "Then go to the link tensorboard displays in command prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}