-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
164 lines (149 loc) · 8.61 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
<!DOCTYPE html>
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Overpass&display=swap" rel="stylesheet">
<link rel="stylesheet" href="styles.css">
<script src = "https://cdn.jsdelivr.net/npm/chart.js@3.7.0/dist/chart.min.js"></script>
</head>
<div class = "titleText">
boolean algebra neural network
<div id = "subtitle">
<h5>Aaron Lee, 2021 </h5>
<a href = "https://github.com/Fungeey/perceptronxor">
<img src = "img/git.png" style = "width:1rem;height:1rem;" id="gitlogo"><img src = "img/arrow.webp" class="arrow">
</a>
</div>
</div>
<div id="main">
<h1>the xor classification problem</h1>
<p class = "problemblock">When given two binary inputs, a neural network should return 1 if the two inputs are not equal, and a 0 if the two inputs are equal.</p>
<p>
Making a neural network learn the XOR function is a classic ai problem. (there are dozens of articles explaining this exact problem).
XOR and XNOR are more complicated than any other logic gate because they are not <mark>linearly separable</mark>.
</p>
<br>
<h1>interactive simulation</h1>
<p>This is a neural network that uses a truth table as training data. After it has trained, it will converge on an
algorithm which can accurately implement the desired boolean algebra function.
</p>
<br>
<div class = "simulationHolder">
<table class = "truthtable">
<tr>
<th>A</th>
<th>B</th>
<th>f</th>
<th>NN</th>
</tr>
<tr>
<td>0</td>
<td>0</td>
<td><button id = "b0" class = "tableButton" onclick="toggleValue(this, 0)"><div class = 'buttonDot'>0</div></td>
<td id = "nn0">. . .</td>
</tr>
<tr>
<td>0</td>
<td><div class = "tableDot">1</div></td>
<td><button id = "b1" class = "tableButton" onclick = "toggleValue(this, 1)"><div class = 'buttonDot'>1</div></td>
<td id = "nn1">. . .</td>
</tr>
<tr>
<td><div class = "tableDot">1</div></td>
<td>0</td>
<td><button id = "b2" class = "tableButton" onclick="toggleValue(this, 2)"><div class = 'buttonDot'>1</div></button></td>
<td id = "nn2">. . .</td>
</tr>
<tr>
<td><div class = "tableDot">1</div></td>
<td><div class = "tableDot">1</div></td>
<td><button id = "b3" class = "tableButton" onclick="toggleValue(this, 3)"><div class = 'buttonDot'>0</div></button></td>
<td id = "nn3">. . .</td>
</tr>
</table>
<div>
<br>
<p style = "width: 300px; text-align: center">
Click on the outputs in the 'f' column to change them.
</p>
<br>
<button id = "startNN" onclick = "startTrain()">Start!</button>
<p id = "runStats" style = "text-align:center;margin-top:0.5rem;"></p>
</div>
</div>
<br>
<canvas id="errorchart"></canvas>
<br>
<p>After clicking start, the network will output values that (should) closely match the desired 'f' column.
The accuracy of the neural network is measured by a <mark>cost function</mark>, which is minimized over the many iterations.
</p>
<br>
<h1>a basic neural network</h1>
<p>A neural network consists of many <mark>nodes</mark>, which are all interconnected.
Here's the neural network structure that I used above: </p>
<br>
<img src="img/mlp.png" style = "width:70%;margin:auto;opacity:0.8;">
<a href ="https://www.researchgate.net/figure/Topology-of-ANN-used-to-solve-logic-XOR-problem_fig1_228939274" class = "imgSource">Photo by researchgate.com</a>
<br>
<p>The output of each node is a weighted sum of its inputs, plus a bias value:</p>
<p class = "focusp">a = Σwx + b</p>
<p>where a is the activation, w is the weight, x is the input, and b is the bias.
With only 3 neurons, the network already has 6 weights and 3 biases. These values are initially randomized, meaning the network
produces very inaccurate results to begin with.
<br><br>
Adjusting all of these values slightly
with a process called <mark>backpropagation</mark> allows the network to find weights and biases that minimize it's cost (how wrong its prediction is).
By repeating this process thousands of times, the network learns to makes better predictions.
</p>
<br>
<h1>problems i ran into</h1>
<b style="margin:0 0 0.5rem;">reliability</b>
<div class = "problemblock">
<p>I struggled to make the network reliable: sometimes it would be completely wrong
despite how many iterations I trained it for.
<a href = "https://stats.stackexchange.com/questions/213666/is-it-normal-that-a-neural-network-sometimes-doesnt-learn-xor">Apparently</a>
this is because there are multiple local minimums that the network
can get stuck in without being able to improve further.
<br><br>
I solved this by initializing the weights randomly with values from <mark>0.5 to 1</mark>, instead of <mark>0 to 1</mark>. Doing this greatly increased the
chances that it would end up in the correct local minimum. <b style="font-weight:normal;color:var(--darkerergrey);font-size:smaller;">(is that cheating ??)</b>
</p>
</div>
<b style="margin:0 0 0.5rem;">performance</b>
<div class = "problemblock">
<p>
The first implementation I made would require 10 or 20 thousand iterations before becoming accurate.
<br><br>
I read <a href="https://stats.stackexchange.com/questions/330559/why-is-tanh-almost-always-better-than-sigmoid-as-an-activation-function">here</a>
that using the <mark>tanh</mark> activation function rather than the <mark>sigmoid</mark> function lets the network learn faster.
I also used <mark>web workers</mark> to put all of the computation on a separate thread and increased the number of hidden nodes from 2 to 3.
After making those changes, my network can reach an acceptable accuracy within 2 to 4 thousand iterations.
</p>
</div>
<br>
<h1>thoughts</h1>
<p>
Throughout this project I learned some things:
<li>how to design a responsive, mobile-friendly website that looks good</li>
<li>how to use javascript <b style="font-weight:normal;color:var(--darkerergrey);font-size:smaller;">(its nice but can get messy sometimes?)</b></li>
<li>the basics of neural networks, especially backpropagation calculus which took forever to understand</li>
</p>
<br>
<h1>resources i used</h1>
<li><a href="https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi">3blue1brown's youtube series on neural networks</a></li>
<li><a href="https://towardsdatascience.com/implementing-the-xor-gate-using-backpropagation-in-neural-networks-c1f255b4f20d">An article by Siddhartha Dutta about backpropagation</a></li>
<li><a href="https://www.youtube.com/playlist?list=PLRqwX-V7Uu6aCibgK1PTWWu9by6XFdCfh">Daniel Shiffman's youtube series on neural networks</a></li>
<li><a href="http://yen.cs.stir.ac.uk/~kjt/techreps/pdf/TR148.pdf">This paper by Richard Bland about the XOR problem</a></li>
<button id="backtotop" onclick="scrollToTop()">up</button>
<br>
<div id = "subtitle" style="width:100%">
<h5>Aaron Lee, 2021 </h5>
<a href = "https://github.com/Fungeey/perceptronxor"><img src = "img/git.png" style = "width:1rem;height:1rem;"></a>
</div>
</div>
</body>
<script src = "main.js"></script>
<script src = "perceptron.js"></script>
</html>