-
Notifications
You must be signed in to change notification settings - Fork 0
/
exp7_theory.html
209 lines (171 loc) · 11.1 KB
/
exp7_theory.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>EXPERIMENT 7 | KJSIT - AI&DS</title>
<meta property="og:title" content="KJSIT AI&DS"/>
<meta property="og:type" content="website" />
<meta property="og:description" content="official website of department of artificial intelligence and data science at K J Somaiya institute of technology" />
<meta property="og:url" content="http://department.kjsieit.in/aids/home.php" />
<meta property="og:image" content="https://kjsit.somaiya.edu.in/assets/kjsieit/images/Logo/kjsieit-logo.svg" />
<link rel="stylesheet" href="styles.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/css/all.min.css"/>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@fortawesome/fontawesome-free@6.2.1/css/fontawesome.min.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.3.0/css/all.min.css">
<link rel="stylesheet" href="css/swiper-bundle.min.css">
<link href='https://unpkg.com/boxicons@2.0.7/css/boxicons.min.css' rel='stylesheet'>
<script src="script.js" defer></script>
<script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
</head>
<!-- Loader -->
<div class="loader">
<div class="loader-spinner"></div>
</div>
<!-- container for sub-header -->
<div id="header-container"></div>
<div class="body-text1">
<div class="title">Experiment 7</div>
</div>
<div class="arrow">
<a href="#" title="Back to Top"><span class="fas fa-angle-up"></span></a>
</div>
<section class="sub-About" style="background:linear-gradient(to bottom right,#ffdbdb,#ffc6c6); width:95%; border-radius: 15px;margin: auto;">
<h1>Back Propagation Neural Network Algorithm</h1>
<h2>AIM:</h2>
<p>Implement Back Propagation Neural Network Algorithm for the given dataset using Python.</p>
<h2>LABORATORY OUTCOMES:</h2>
<ul>
<li>Students will be able to implement Back Propagation Network algorithm for the given dataset.</li>
<li>Students will be able to plot the mean square error and accuracy of Back Propagation Network.</li>
</ul>
<h2>THEORY:</h2>
<div>
<p>Backpropagation neural network is used to improve the accuracy of neural network and make them capable of self-learning. Backpropagation means “backward propagation of errors”. Here error is spread into the reverse direction in order to achieve better performance. Backpropagation is an algorithm for supervised learning of artificial neural networks that uses the gradient descent method to minimize the cost function. It searches for optimal weights that optimize the mean-squared distance between the predicted and actual labels.</p>
</div>
<div>
<h2>How BPN Works?</h2>
<p>BPN learns in an iterative manner. In each iteration, it compares training examples with the actual target label. Target label can be a class label or continuous value. The backpropagation algorithm works in the following three steps:</p>
<ol>
<li><strong>Initialize Network:</strong> BPN randomly initializes the weights.</li>
<li><strong>Forward Propagate:</strong> After initialization, we will propagate into the forward direction. In this phase, we will compute the output and calculate the error from the target output.</li>
<li><strong>Back Propagate Error:</strong> For each observation, weights are modified in order to reduce the error in a technique called the delta rule or gradient descent. It modifies weights in a “backward” direction to all the hidden layers.</li>
</ol>
</div>
<div>
<h2>Implementation Steps of BPN using Python:</h2>
<ol>
<li><strong>Import Libraries:</strong> Let's import the required modules and libraries such as numpy, pandas, scikit-learn, and matplotlib.</li>
<li><strong>Load Dataset:</strong> Let’s first load the Iris dataset using <code>load_iris()</code> function of scikit-learn library and separate them into features and target labels.</li>
<li><strong>Prepare Dataset:</strong> Create dummy variables for class labels using <code>get_dummies()</code> function.</li>
<li><strong>Split Train and Test set:</strong> To understand model performance, dividing the dataset into a training set and a test set is a good strategy. Let’s split dataset by using function <code>train_test_split()</code>. You need to pass basically 3 parameters features, target, and test_set size. Additionally, you can use <code>random_state</code> in order to get the same kind of train and test set.</li>
<li><strong>Initialise Hyperparameters and Weights:</strong> Let's initialize the hyperparameters such as learning rate, iterations, input size, number of hidden layers, and number of output layers. Then, initialize the weights for hidden and output layers with random values.</li>
<li><strong>Helper Function:</strong> Let's create helper functions such as <code>sigmoid</code>, <code>mean_squared_error</code>, and <code>accuracy</code>. We will create a for loop for given number of iterations that execute the three steps (feedforward propagation, error calculation, and backpropagation phase) and update the weights in each iteration.</li>
<li><strong>Plot MSE and Accuracy:</strong> Let's plot mean squared error in each iteration using pandas <code>plot()</code> function.</li>
<li><strong>Predict for Test Data and Evaluate the Performance:</strong> Let’s make prediction for the test data and assess the performance of Backpropagation neural network.</li>
</ol>
</div>
<h2>PROGRAM:</h2>
<pre><code>
# Import libraries
import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
# Loading dataset
data = load_iris()
X = data.data
y = data.target
# Split dataset into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=20, random_state=4)
# Hyperparameters
learning_rate = 0.1
iterations = 5000
N = y_train.size
input_size = 4
hidden_size = 2
output_size = 3
np.random.seed(10)
W1 = np.random.normal(scale=0.5, size=(input_size, hidden_size))
W2 = np.random.normal(scale=0.5, size=(hidden_size, output_size))
# Helper functions
def sigmoid(x):
return 1 / (1 + np.exp(-x))
def mean_squared_error(y_pred, y_true):
# One-hot encode y_true (i.e., convert [0, 1, 2] into [[1, 0, 0], [0, 1, 0], [0, 0, 1]])
y_true_one_hot = np.eye(output_size)[y_true]
# Reshape y_true_one_hot to match y_pred shape
y_true_reshaped = y_true_one_hot.reshape(y_pred.shape)
# Compute the mean squared error between y_pred and y_true_reshaped
error = ((y_pred - y_true_reshaped)**2).sum() / (2*y_pred.size)
return error
def accuracy(y_pred, y_true):
acc = y_pred.argmax(axis=1) == y_true.argmax(axis=1)
return acc.mean()
results = pd.DataFrame(columns=["mse", "accuracy"])
# Training loop
for itr in range(iterations):
# Feedforward propagation
Z1 = np.dot(X_train, W1)
A1 = sigmoid(Z1)
Z2 = np.dot(A1, W2)
A2 = sigmoid(Z2)
# Calculate error
mse = mean_squared_error(A2, y_train)
acc = accuracy(np.eye(output_size)[y_train], A2)
new_row = pd.DataFrame({"mse": [mse], "accuracy": [acc]})
results = pd.concat([results, new_row], ignore_index=True)
# Backpropagation
E1 = A2 - np.eye(output_size)[y_train]
dW1 = E1 * A2 * (1 - A2)
E2 = np.dot(dW1, W2.T)
dW2 = E2 * A1 * (1 - A1)
# Update weights
W2_update = np.dot(A1.T, dW1) / N
W1_update = np.dot(X_train.T, dW2) / N
W2 = W2 - learning_rate * W2_update
W1 = W1 - learning_rate * W1_update
# Visualizing the results
results.mse.plot(title="Mean Squared Error")
plt.show()
results.accuracy.plot(title="Accuracy")
plt.show()
# Test the model
Z1 = np.dot(X_test, W1)
A1 = sigmoid(Z1)
Z2 = np.dot(A1, W2)
A2 = sigmoid(Z2)
test_acc = accuracy(np.eye(output_size)[y_test], A2)
print("Test accuracy: {}".format(test_acc))
</code></pre>
<h2>CONCLUSION:</h2>
<p>Implementation of BPN for Iris dataset is carried out here. BPN is a method to optimize neural networks by propagating the error or loss into a backward direction. It finds loss for each node and updates its weights accordingly in order to minimize the loss using gradient descent.</p>
<h2>TEXT/REFERENCE BOOKS:</h2>
<ul>
<li>“Neural Network a Comprehensive Foundation” By Simon Haykin</li>
<li>“Introduction to Soft Computing” By Dr. S. N. Shivnandam, Mrs. S. N. Deepa</li>
<li>“Neural Network: A classroom Approach” By Satish Kumar</li>
<li>“Neural Network, Fuzzy Logic and Genetic Algorithms” By Rajshekharan S, Vijayalakshmi Pai</li>
<li>“Neural Network Design” by Hagan Demuth, Beale</li>
<li>“Neural Network for Pattern Recognition”, Christopher M. Bishop</li>
</ul>
<h2>WEB ADDRESS (URLS):</h2>
<ul>
<li><a href="https://machinelearninggeek.com/backpropagation-neural-network-using-python/">https://machinelearninggeek.com/backpropagation-neural-network-using-python/</a></li>
<li><a href="https://www.askpython.com/python/examples/backpropagation-in-python">https://www.askpython.com/python/examples/backpropagation-in-python</a></li>
<li><a href="https://www.sebastian-mantey.com/theory-blog/basics-of-deep-learning-p13-implementing-the-backpropagation-algorithm-with-numpy">https://www.sebastian-mantey.com/theory-blog/basics-of-deep-learning-p13-implementing-the-backpropagation-algorithm-with-numpy</a></li>
</ul>
</section>
<br>
<!--load footer here-->
<div id="footer-container"></div>
</body>
</html>
</section>
<br>
<!--load footer here-->
<div id="footer-container"></div>
</body>
</html>