07 3rd February, Saturday

I've been experimenting in tensor flow on a much smaller network that recognises the MNIST dataset. I have been looking to find some correlation between activations in each of the layers caused by training images and the weights between layers.

My first idea was to try to get a weight signature for each of the training images, by this I mean get their activation at each layer and multiply this by each connection. This gives us a matrix with the same dimensions as the weight matrix between the layers, that tells us the activation in each of the connections caused by the input. The idea here was that for each image I could get a representation of how each of the connections in the network were activated by that image. Then I looked to average these connection activations over all the training images that were actually used to train the network. My hope was that I would be able to get a representation of which connections were most used for a set of input images and that if these images were the whole set then the representation would be very similar to the actual weights of the connections in the network.

Having actually tried this I was fairly unsuccessful. I suspected that because these calculations usually happen before bias/activation functions are applied I would be able to get away with this. I'm not completely sure yet as to whether this is just not going to work, or whether incorporating bias/activation function in my calculations of the activations of connections for each image will result in more similarity between connection activation and weights.

I've also been struggling to compare the matrices relating to connection activation and weights, and finding a good way other than finding the normals between each vector of connections might help here.