Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Numbers recognition #90

Open
JosselinSomerville opened this issue Mar 3, 2018 · 18 comments
Open

Numbers recognition #90

JosselinSomerville opened this issue Mar 3, 2018 · 18 comments

Comments

@JosselinSomerville
Copy link

Hi !
I've tried to implement the same neural network library as in the videos
However I have some problems

I am coding in Python, in Processing. Just to warn you, I am not really familiar with Python and Processing, that's why i have everything in one file ( I wasn't able to separate it in defferent files and then import them , if you know how to do it please tell me)

So the problem i am getting, is that the neural network is getting stuck. In fact I rarely go over 85% accuracy with is quite bad I think. And it doesn't matter how long i let it run, arround 85% , the accuracy just randomly changes. I know that my code is not really clear and messy , but could someone try to find what's going on ? Just so you know, I've stored, all the images in 60 000 files (1 per image ) , each containing a number between 0 and 1 for the pixel color

I've also added the possibility to have several hidden layers in the neural network
However, iam not sure if I am doing the backpropagation and the gradient descent correctly, could anyone check and tell me ?

Thanks a lot, I am new to GitHub so sorry if I did things wrong
( and by the way, I am French, that's why my english might be bad )

@JosselinSomerville
Copy link
Author

JosselinSomerville commented Mar 3, 2018

import math

class Matrix:
    
    def __init__(self, nb_lignes = 1, nb_colonnes = 1, value = 0):
        self.n = nb_lignes
        self.m = nb_colonnes
        self.mat = []
        
        for i in range(0,self.n):
            self.mat.append([])
            for j in range(0,self.m):
                self.mat[-1].append(value)
    
        
        
    def show(self):
        maxLongueur = 1
        for ligne in self.mat:
            for elt in ligne:
                if(len(str(elt)) > maxLongueur):
                    maxLongueur = len(str(elt))
    
        
        for ligne in self.mat:
            string = "( "
    
            for elt in ligne:
                string += str(elt) + " "*(maxLongueur-len(str(elt)) + 1) + "| "
    
            string += ")"
            print(string)
            
    def __add__(self, B):
        """Additionne 2 matrices A et B de meme dimensions ( n x m )
        A une complexite en O(n x m) operations elementaires"""
        
        if(type(B) == type(self)):
            n = len(self.mat)
            m = len(self.mat[0])
        
            R = Matrix(n, m) # Matrice que l'on va renvoyer
        
            for i in range(n): # Pour chaque ligne
                for j in range(m): # Pour chaque case de la ligne
                    R.mat[i][j] = self.mat[i][j] + B.mat[i][j]
        
            return R
        elif(type(B) == type(1)):
            n = len(self.mat)
            m = len(self.mat[0])
            
            R = Matrix(n, m, 0) # Matrice que l'on va renvoyer
            
            for i in range(n): # Pour chaque ligne
                for j in range(m): # Pour chaque case de la ligne
                    R.mat[i][j] = self.mat[i][j] + B
                    
            return R
        else:
            return NotImplemented
        
        
    def __sub__(self, B):
        return self + B * (-1)
    
    
    def __mul__(self, B):
        """Calcule le produit A x B avec A de taille n x m et B de taille m x p ( AB de taille n x p )
        A une complexite en O(n x m * p) operations elementaires"""
        
        if(type(B) == type(self)):
            if(self.m == B.n):
                n = self.n
                m = B.n
                p = B.m
                #print("matrix(" + str(self.n) + "," + str(self.m) + ") * matrix(" + str(B.n) + "," + str(B.m) + ")")
            
                R = Matrix(n, p, 0) # Matrice que l'on va renvoyer
            
                for i in range(n): # Pour chaque ligne
                    for j in range(p): # Pour chaque case de la ligne
                        somme = 0
                        for k in range(m):
                            somme += self.mat[i][k] * B.mat[k][j]
                        R.mat[i][j] = somme
            
                return R
            elif((self.n == B.n) and (self.m == B.m)):
                #print("matrix x matrix (same dimensions)")
                R = Matrix(self.n, self.m, 0) # Matrice que l'on va renvoyer
            
                for i in range(self.n): # Pour chaque ligne
                    for j in range(self.m): # Pour chaque case de la ligne
                        R.mat[i][j] = self.mat[i][j] * B.mat[i][j]
            
                return R
            
        elif((type(B) == type(1.0)) or (type(B) == type(1))):
            #print("matrix * number")
            n = len(self.mat)
            m = len(self.mat[0])
            
            R = Matrix(n, m, 0) # Matrice que l'on va renvoyer
            
            for i in range(n): # Pour chaque ligne
                for j in range(m): # Pour chaque case de la ligne
                    R.mat[i][j] = self.mat[i][j] * B
                    
            return R
        else:
            print("not implemented")
            print(type(B), type(1))
            return NotImplemented
        
    
    def map_function(A, f):
        n = len(A.mat)
        m = len(A.mat[0])
            
        R = Matrix(n, m, 0) # Matrice que l'on va renvoyer
            
        for i in range(n): # Pour chaque ligne
            for j in range(m): # Pour chaque case de la ligne
                R.mat[i][j] = f(A.mat[i][j])
                    
        return R
    
    def randomize(self):
        n = len(self.mat)
        m = len(self.mat[0])
        
        for i in range(n): # Pour chaque ligne
            for j in range(m): # Pour chaque case de la ligne
                self.mat[i][j] = random(2) - 1
                
    def fromArray(self, arr):
        n = len(arr)
        m = 1
        A = Matrix(n,m)
        
        for i in range(n):
            A.mat[i][0] = arr[i]
        
        return A
            
    def toArray(self):
        arr = []
        for i in range(self.n): # Pour chaque ligne
            for j in range(self.m): # Pour chaque case de la ligne
                arr.append(self.mat[i][j])
        return arr
    
    
    def transpose(A):
        n = len(A.mat)
        m = len(A.mat[0])
            
        R = Matrix(m, n, 0) # Matrice que l'on va renvoyer
        for i in range(n): # Pour chaque ligne
            for j in range(m): # Pour chaque case de la ligne
                R.mat[j][i] = A.mat[i][j]
                
        return R


class NeuralNetwork:
    
    def sigmoid(x):
        return 1 / (1 + exp(-x))
    
    def derivative_sigmoid(y):
        return y * ( 1 - y)
    
    
    def __init__(self, nb_inputs, nb_layers, nb_nodes_per_hidden, nb_outputs, function = sigmoid, derivative = derivative_sigmoid):
        self.input_nodes = nb_inputs
        self.layers = nb_layers
        self.hidden_nodes = nb_nodes_per_hidden
        self.output_nodes = nb_outputs
        
        self.activation = function
        self.derivative_activation = derivative
        self.nb_train = 0
        self.learningRate = 0.1
        
        
        self.weights = []
        self.bias = []
        self.nb_char_weights = 15
        
        # On ajoute l'entree
        self.weights.append(Matrix(self.hidden_nodes, self.input_nodes))
        self.weights[-1].randomize()
        self.bias.append(Matrix(self.hidden_nodes, 1))
        self.bias[-1].randomize()
        
        # On ajoute les couches cachees
        for i in range(max(0, self.layers - 2)):
            self.weights.append(Matrix(self.hidden_nodes, self.hidden_nodes))
            self.weights[-1].randomize()
            self.bias.append(Matrix(self.hidden_nodes, 1))
            self.bias[-1].randomize()
        
        # On ajoute la sortie
        self.weights.append(Matrix(self.output_nodes, self.hidden_nodes))
        self.weights[-1].randomize()
        self.bias.append(Matrix(self.output_nodes, 1))
        self.bias[-1].randomize()

        
        
    def feed(self, input_array):
        results = []
        for i in range(self.layers + 1):
            results.append(Matrix())
            
        # On 'rentre' l entree
        results[0] = Matrix().fromArray(input_array)
        
        # On genere les resultats
        for i in range(self.layers):
            results[i+1] = self.weights[i] * results[i]
            results[i+1] = results[i+1] + self.bias[i]
            results[i+1] = Matrix.map_function(results[i+1], self.activation)
        
        return results[-1].toArray()
    
    
    def train(self, input_array, target_array):
        self.nb_train += 1
        
        # ---------- Feed function ------------- #
        results = []
        for i in range(self.layers + 1):
            results.append(Matrix())
            
        # On 'rentre' l entree
        results[0] = Matrix().fromArray(input_array)
        
        # On genere les resultats
        for i in range(self.layers):
            results[i+1] = self.weights[i] * results[i]
            results[i+1] = results[i+1] + self.bias[i]
            results[i+1] = Matrix.map_function(results[i+1], self.activation)
        
        
        
        
        # ---------- Computing errors ----------#
        targets = Matrix().fromArray(target_array)
        
        errors = []
        for i in range(self.layers):
            errors.append(Matrix())
            
        # On connait la dernière erreur
        errors[-1] = targets - results[-1]
        
        # On calcule les autres erreurs
        for i in range(self.layers - 2, -1, -1):
            weights_t = Matrix.transpose(self.weights[i+1])
            errors[i] = weights_t * errors[i+1]
        
    
        
        
        # ---------- Computing gradients --------#
        gradients = []
        for i in range(self.layers):
            gradients.append(Matrix())
            gradients[i] = Matrix.map_function(results[i+1], self.derivative_activation)
            gradients[i] = errors[i] * gradients[i]
            gradients[i] = gradients[i] * self.learningRate
            
        
        # ---------- Computing deltas ----------#
        deltas = []
        for i in range(self.layers):
            deltas.append(Matrix())
            results_t = Matrix.transpose(results[i])
            deltas[i] = gradients[i] * results_t
            
        
        # ----------- Adjusting weights ---------#
        for i in range(self.layers):
            self.weights[i] = self.weights[i] + deltas[i]
            self.bias[i] = self.bias[i] + gradients[i]
            
            
            
            
    
    
    def saveWeights(self, name_file, folder):
        file = open("C:/Users/josse/Documents/Processing/" + folder + "/data/weights/" + name_file, "w")
        string = str(self.nb_char_weights)+ "/"
        for matrix in self.weights:
            for ligne in matrix.mat:
                for elt in ligne:
                    l_round = self.nb_char_weights -(1 + len(str(int(elt))))
                    if(elt < 0):
                        l_round -= 1
                    l_zeros = self.nb_char_weights - len(str(round(elt, l_round)))
                    string += str(round(elt, l_round)) + "0"*l_zeros + "," 
                    
                    
        for matrix in self.bias:
            for ligne in matrix.mat:
                for elt in ligne:
                    l_round = self.nb_char_weights -(1 + len(str(int(elt))))
                    if(elt < 0):
                        l_round -= 1
                    l_zeros = self.nb_char_weights - len(str(round(elt, l_round)))
                    string += str(round(elt, l_round)) + "0"*l_zeros + "," 
                    
        #print(string)
        file.write(string)
        file.close()
        
    
    def loadWeights(self, name_file):
        file = open("C:/Users/josse/Documents/Processing/" + folder + "/data/weights/" + name_file, "rb")
        nb_char = int(file.read(2))
        file.read(1) # Skip "/"
        
        weights = []
        for matrix in self.weights:
            weights.append(Matrix(matrix.n, matrix.m))
            for i in range(matrix.n):
                for j in range(matrix.m):
                    weights[-1].mat[i][j] = float(file.read(nb_char))
                    file.read(1) # Skip ","
        
        bias = []
        for matrix in self.bias:
            bias.append(Matrix(matrix.n, matrix.m))
            for i in range(matrix.n):
                for j in range(matrix.m):
                    bias[-1].mat[i][j] = float(file.read(nb_char))
                    file.read(1) # Skip ","
                    
        self.weights = weights
        self.bias = bias
                    
            
        file.close()
            
        
        
        
            
               

    
    
    
file_weights = "weights_4_16.txt"
#file_weights = "weights_2_16.txt"
folder = "Neural_chiffres"
loadWeights = True
   
nn = NeuralNetwork(784,3,50,10)

nb_train = 60000
nb_test = 10000

nb_feed_par_test = 100
decrement_learning_rate = 0.05
save_tick = 1000
test_tick = 250


last_test = 0
last_saved = [hour(), minute(), second()]
last_predicted = 0
last_probabilite = 0


canvas = 0
zoom = 10

def setup():
    global  canvas
    canvas = []
    for i in range(784):
        canvas.append(0)
        
    if(loadWeights):
        nn.loadWeights(file_weights)
        
    size(84*zoom,28*zoom)
    print("Done")
    frameRate(30)
    nn.learningRate = 0.0065
    
    
def draw():
    global nb_train, nb_test, nn, canvas, last_saved, last_test, last_predicted, file_weights, test_tick, save_tick, folder
    
    clear()
    noStroke()
    textAlign(CENTER)
    

    # ============ Canvas ============= #
    for i in range(784):
        fill(canvas[i]*255)
        rect(i%28 * zoom, i // 28 * zoom, zoom, zoom)
    
    if(mousePressed):
        if(mouseX < 28*zoom):
            x = int(mouseX / zoom)
            y = int(mouseY / zoom)
            canvas[y*28 + x] = 1
            
            if(x > 0):
                canvas[y*28 + x -1] = 1
            if(x < 27):
                canvas[y*28 + x +1] = 1
            if(y > 0):
                canvas[(y-1)*28 + x] = 1
            if(y <27):
                canvas[(y+1)*28 + x] = 1
                
    
    # ============= Training =========== #
    index = int(random(nb_train))
    n,liste = read(index, "train")
    
    result = []
    for i in range(10):
        if(i == n):
            result.append(1)
        else:
            result.append(0)
                
    
    
    if((nn.nb_train % save_tick == 0) and (nn.nb_train >0)):
        nn.saveWeights(file_weights, folder)
        print("Saved")
        last_saved = [hour(), minute(), second()]

    if((nn.nb_train % test_tick == 0)):
        pourcentage = test(nb_feed_par_test)
        last_test = pourcentage
        #nn.learningRate = round((100 - pourcentage)**1 / 100.0,7) * decrement_learning_rate
        print("LearningRate = " + str(nn.learningRate))
        
    nn.train(liste, result)
    drawImage(liste)
            
                

        
        
    
    # Affichage infos
    fill(255)
    text("Nombre d'entrainements : " + str(nn.nb_train),42*zoom, 2*zoom)
    text("Vitesse : " + str(int(frameRate + 0.5)) + " entrainements / s", 42*zoom, 3.5*zoom)
    text("Derniere sauvegarde : ",42*zoom, 8.5*zoom)
    text(str(last_saved[0]) + "h " + str(last_saved[1]) + "m " + str(last_saved[2]) + "s",42*zoom, 10*zoom)
    text("Dernier test de " + str(nb_feed_par_test) + " : " ,42*zoom, 13.5*zoom)
    text("Taux de reussite : " + str(last_test) + " %",42*zoom, 15*zoom)
    text("Learning rate : " + str(nn.learningRate),42*zoom, 18.5*zoom)
    text("Prediction du dessin :",42*zoom, 25*zoom)
    text("C'est un : " + str(last_predicted) + " ( " + str(last_probabilite) + "% )",42*zoom, 26.5*zoom)
   
            

def drawImage(liste):
    global zoom
    img = createImage(28,28,RGB)
    
    for i in range(784):
        img.pixels[i] = color(liste[i]*255)
    
    img.resize(28*zoom, 28*zoom)
    image(img, 56*zoom,0)
        
    
            
    

def read(index, type):
    global folder
    file = open("C:/Users/josse/Documents/Processing/" + folder + "/data/rc/" + type + "/" + str(index) + ".txt", "rb")
    nb_char = int(file.read(1))
    num = int(file.read(1))
    liste = []
    
    for i in range(784):
        liste.append(float(file.read(nb_char)))
        file.read(1) # La virgule
        
    return num, liste
    

def keyPressed():
    global canvas, nn, last_saved, last_test, last_predicted, last_probabilite, file_weights, folder
    #print(key)
    if(key == " "):
        for i in range(784):
            canvas[i] = 0
    elif(key == "\n"):
        liste = []
        for i in range(len(canvas)):
            liste.append(canvas[i] * 2.0 - 1)
            
        outputs = nn.feed(liste)
        maximum = 0
        for i in range(len(outputs)):
            if(outputs[i] > outputs[maximum]):
                maximum = i
                
        print("C'est un : " + str(maximum))
        last_predicted = maximum
        last_probabilite = int(100 * outputs[maximum])
        print(outputs)
    elif(key == "t"):
        pourcentage = test(nb_feed_par_test)
        last_test = pourcentage
        #nn.learningRate = round((100 - pourcentage)**1 / 100.0,7) * decrement_learning_rate
        print("LearningRate = " + str(nn.learningRate))
        print("\nPourcentage = " + str(pourcentage) + "\n")
    elif(key == "g"):
        loop()
    elif(key == "s"):
        nn.saveWeights(file_weights, folder)
        print("Saved")
        last_saved = [hour(), minute(), second()]




def test(nb):
    global  nn, nb_test, nb_train        
        
    reussite = 0.0
    
    for i in range(nb):
        index = int(random(nb_train))
        n,liste = read(index, "train")
                
        outputs = nn.feed(liste)
        maximum = 0
        for i in range(len(outputs)):
            if(outputs[i] > outputs[maximum]):
                maximum = i
                
        if(maximum == n):
            reussite += 1
            
    return float(reussite/nb) * 100

@Versatilus
Copy link
Collaborator

Because Python uses formatting and white space for structure, I need you to edit your comment on GitHub website and put three backtick characters (`) and the word "python" on the line above your code and another three backtick characters on the line after your code. That will make GitHub do syntax highlighting for your code and preserve all of the white space.

```python
print "Countdown..."

for i in [3, 2, 1, 0]:
if i > 0:
print str(i) + "..."
else:
print "BLAST OFF!"
```

becomes

print "Countdown..."

for i in [3, 2, 1, 0]:
    if i > 0:
        print str(i) + "..."
    else:
        print "BLAST OFF!"

@JosselinSomerville
Copy link
Author

Of course, I didn't know how to do it !
Here you go, I hope you find what's wrong

@JosselinSomerville
Copy link
Author

By the way tell me if you need the code that creates the files for the input data

@Versatilus
Copy link
Collaborator

I'm still not done yet. I haven't forgotten about this. Every time I started working on this over the weekend I would fall asleep. It wasn't boring. I was just very sleepy all weekend!

@JosselinSomerville
Copy link
Author

Oh thanks, I was a bit worried ! No problem, tell me if you need any precisions !

@tobiasvonarx
Copy link

tobiasvonarx commented Mar 10, 2018

I'm not sure about the error, you can find my version of a similar neural network here if it helps you. It's for number recognition as well and has a similar structure.

@tobiasvonarx
Copy link

To import python scripts (e.g. nn.py) from the same directory into another python file (e.g. main.py) you can do:

# import relevant classes from nn.py into main.py
from nn import Matrix, NeuralNetwork

with the folder structure looking like this:

/ nn.py
/ main.py

If you want the file in a different directory you will have to reference the folder:

from folder.nn import Matrix, NeuralNetwork

where the folder structure looks like this

/ main.py
    / folder
        / nn.py

Note that you might need to have a __init__.py file for this to work (to be included as a package), so you can create an empty one in the directory of the file you want to import (nn.py in this case). I hope I could help you 😉

@JosselinSomerville
Copy link
Author

Thanks a lot ! I'm a bit tired right now, but I'll definitely check it out tomorrow.
I'll make sure to send you my feed back
Thanks !

@JosselinSomerville
Copy link
Author

Hi ! I wanted to try your code but I couldn't because I don't know which files were needed and where to place them, could you tell me please or upload them on GitHub ?
Thanks !

@tobiasvonarx
Copy link

tobiasvonarx commented Mar 12, 2018

Hi, you can find my code by clicking here. You can either download the repository in its current state or fork it, whichever you prefer. To run it, you can run
python NeuralNetwork.py
from the root directory inside the repository folder. Make sure to use Python 3.x to run the script, otherwise errors will occur.

The python script NeuralNetwork.py contains both the NeuralNetwork class definition and the code for the MNIST classification in one file (i'll seperate them into seperate files some day). When running NeuralNetwork.py, it will look for pre-trained weight matrices in the data/ subdirectory. Make sure not to delete them, as the training process has 10 epochs and is very long-winded. When run, the code will query the NeuralNetwork about your own pictures, found in the my_own_images subdirectory, and output what it thinks in the console output. If you want to add your own, create a 28x28 png (without transparency and b/w works best) and rename it 2828_my_own_label, where label is the number your image is supposed to represent.

If you want to look into the code, I'd give you a tip and tell you that all code before line 90 belongs to the Neural Network itself, whereas everything after line 90 is specific to MNIST and number recognition.

@JosselinSomerville
Copy link
Author

Yeah sure I'll try, it's just that I do not have the mnist.csv file. I have one mnist file but it's not à .csv file. So where did you download it ? Could you upload it maybe ?
Thanks again for your help !

@tobiasvonarx
Copy link

The one that I use can be found here, it isn't the full length version, as that exceeded GitHub's file size limit of 100MB. The original CSV can be downloaded from here

@JosselinSomerville
Copy link
Author

Thanks ! And how accurate did you get with this setup ?
I'll try it tonight

@JosselinSomerville
Copy link
Author

Hi ! I've tried your code and the results are amazing !
I've mosty understood you code, however, they are some parts that I don't really get :

self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)),
                                        numpy.transpose(hidden_outputs))

What is : (1.0 - final_outputs) ?

And now in the training process, you're doing some image treatment but, I don't really understand what you are doing with +10.0 and -1.0 ( could you explain please ? )

inputs = (numpy.asfarray(all_values[1:]) / 255.0 * 0.99) + 0.01
inputs_plus10 = rotate(inputs.reshape(28,28), 10.0, cval=0.01, order=1, reshape=False).reshape(784)
inputs_minus10 = rotate(inputs.reshape(28,28), -10.0, cval=0.01, order=1, reshape=False).reshape(784)
# create the target output values (all 0.01, except the desired label which is 0.99)
targets = numpy.zeros(output_nodes) + 0.01
# all_values[0] is the target label for this record
targets[int(all_values[0])] = 0.99
neuralNetwork.train(inputs, targets)
neuralNetwork.train(inputs_plus10, targets)
neuralNetwork.train(inputs_minus10, targets)

Anyway thanks a lot for sharing your code ! You're awesome !

@Versatilus
Copy link
Collaborator

final_outputs * (1.0 - final_outputs) is the derivative of the cost function, as the first step in back propagation. I haven't read his full source code, but I think that's to change the data up a little before training. Some research has shown improved accuracy on this dataset by skewing the images.

I still haven't given up on looking at your code, but every time I get settled down to go over it someone or something distracts me.

@tobiasvonarx
Copy link

tobiasvonarx commented Mar 13, 2018

@Versatilus summarized it perfectly, thank you for that. In addition, i'd like to clarify that for every image in the mnist dataset the code skews the picture, effectively rotating the picture 10 degrees in either direction (+10 and 10) and resulting in much more training data. Thank you for the positive feedback!

@Versatilus
Copy link
Collaborator

@JosselinSomerville I'm sorry that it's been almost a month and I still haven't finished looking at your code. I haven't forgotten about it. I've just been very busy, which is no real excuse. If you're still interested, I'll try to get to it soon. Assuming I don't nap too much, I'll try to get to it this weekend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants