Pooya

TensorFlow:

TensorFlow is an open source software library for numerical computation using data flow graphs in which nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them.

TensorFlow is Google Brain's second generation machine learning system, with a reference implementation released as open source software last November. It provides a Python API, as well as a less documented C/C++ API.

TensorFlow can run on multiple CPUs and GPUs (with optional CUDA extensions for general-purpose computing on graphics processing units). It runs on 64-bit Linux or Mac OS X desktop or server systems, as well as on mobile computing platforms, including Android and Apple's iOS.

The purpose is to train neural networks to detect and decipher patterns and correlations. There are some other libraries for numerical computations in python. One of these libraries is Theano, which is one of the strong python libraries for machine learning tasks. However, Theano's applicability in different GPU's is not experimentally verified yet, which was one of the major reasons for pushing us through TensorFlow, which gives strong computational functions to use in neural networks like activation functions, calculating loss and gradient descent.

Evaluation:

In our project we want to implement different architectures of neural networks with different complexities on a large dataset to spot the effect of complexity of neural networks on its accuracy on large datasets. There is an assumption in machine learning which says simple machine architectures don't learn from extra provided data. However, if we increase the complexity of machine then the performance would increase by the increase in amount of data. It is very important to consider that it is not applicable to train very complex neural networks on very large datasets, since it needs a lot of time. Therefore, it is very important to take advantage of some high performance computing techniques like using multiple GPUs/CPUs to split the dataset in different batches and load these batches to the designed complex neural network while it is training on some other data batches.

In this project we aim to measure the strong/weak scaling and the speed up of different neural network architectures to evaluate the effect of using hpc techniques in complex neural networks and measure the accuracy of different architectures on large datasets to evaluate how complexity affects neural network's learning and accuracy. To measure the accuracy of the network we will use logloss function which is generally used to measure the logarithm of the likelihood function for a Bernouli random distribution.

Time Table:

The time scheduling for this project is as follows:

1. Data Clean Up

2. Setting up and learning TensorFlow

3. Setting up a small Neural Network (NN) in TensorFlow

4. Running a full experimental NN on the dataset

5. Probable debuggings

6. Setting up Amazon Cloud

7. Deploying the Architecture to Amazon

8. Fine-Tuning

9. Evaluation of the results

Reference:

[1] https://www.tensorflow.org/

[2] https://en.wikipedia.org/wiki/TensorFlow

=============================================================== Here is the brief description for works which are done:

1. TensorFlow is installed and setup.

2. TensorFlow classes and functions which are useful in developing the network are developed and learned.

3. Tools for providing visual results in TensorFlow are learned.

4. To get familiar with TensorFlow a simple classification model with softmax layer is implemented and trained.

Following is the architecture of this simple classifier:

image credit: www.tensorflow.com

In which y = softmax(wx + b)

To update the weights and biases in this model we used gradient descent optimizer to minimize the cross entropy as the cost function of the model.

=============================================================== The system which we are doing our simulations has two GPUs, so to take the advantage of these available GPUs we have taken the following strategy:

Our strategy in taking the advantage of two GPUs:

1. We are splitting the data in different batches. Each GPU will do the training in one of the two batches which is loaded to each of them each time. It is very important to consider that the hyper-parameters of the Neural Network are updated in each step of training, so by considering that we have two different updates resulted from each GPU we will average the results of two GPUs to get the new hyper-parameters.

2. There is a high latency in transferring the data from/to GPU. Therefore, we are making some queues in which the input data is putted in this queue each time the GPUs are training the batches. This will decrease the negative effect of the latency associated with GPUs' data transfer.