-
Notifications
You must be signed in to change notification settings - Fork 0
/
Housing.html
385 lines (314 loc) · 13.8 KB
/
Housing.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
<!DOCTYPE HTML>
<!--
Zohaib Aftab, Data Scientist
zohaibdr.github.io |
To showcase my projects
-->
<html>
<head>
<title>Project: Housing Price Conundrum </title>
<meta content="" name="description">
<meta content="" name="keywords">
<!-- Favicons -->
<link href="assets/img/favicon.png" rel="icon">
<link href="assets/img/apple-touch-icon.png" rel="apple-touch-icon">
<!-- Vendor CSS Files -->
<link href="assets/vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<link href="assets/vendor/bootstrap-icons/bootstrap-icons.css" rel="stylesheet">
<link href="assets/vendor/glightbox/css/glightbox.min.css" rel="stylesheet">
<link href="assets/vendor/swiper/swiper-bundle.min.css" rel="stylesheet">
<!-- Template Main CSS File -->
<link href="assets/css/style.css" rel="stylesheet">
<style>
img {
max-width: 90%;
display: block;
/* remove extra space below image */
}
</style>
</head>
<!-- Google tag (gtag.js) -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-TSB82NSVFW"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-TSB82NSVFW');
</script>
<!-- ======= Header ======= -->
<header id="header" class="fixed-top">
<div class="container d-flex align-items-center justify-content-between">
<!-- <h1 class="logo"><a href="index.html">DevFolio</a></h1> -->
<!-- Uncomment below if you prefer to use an image logo -->
<a href="index.html" class="logo"><img src="assets/img/logo.png" alt="" class="img-fluid"></a>
<nav id="navbar" class="navbar">
<ul>
<li><a class="nav-link scrollto active" href="index.html">Home</a></li>
<li><a class="nav-link scrollto" href="index.html#about">About</a></li>
<!-- <li><a class="nav-link scrollto" href="index.html#services">Services</a></li> -->
<li><a class="nav-link scrollto " href="index.html#portfolio">Portfolio</a></li>
<!-- <li><a class="nav-link scrollto " href="index.html#blog">Blog</a></li> -->
<li><a class="nav-link scrollto" href="index.html#contact">Contact</a></li>
</ul>
<i class="bi bi-list mobile-nav-toggle"></i>
</nav><!-- .navbar -->
<!-- .navbar -->
</div>
</header><!-- End Header -->
<div class="hero hero-single route bg-image" style="background-image: url(assets/img/overlay-bg.jpg)">
<div class="overlay-mf"></div>
<div class="hero-content display-table">
<div class="table-cell">
<div class="container">
<h2 class="hero-title mb-4">
Housing Price Prediction
</h2>
<ol class="breadcrumb d-flex justify-content-center">
<li class="breadcrumb-item">
<a href="index.html#Portfolio">Portfolio</a>
</li>
<li class="breadcrumb-item active">
Housing Price Prediction
</li>
</ol>
</div>
</div>
</div>
</div>
<main id="main">
<!-- ======= Portfolio Details Section ======= -->
<section id="portfolio-details" class="portfolio-details">
<div class="container">
<div class="row gy-4">
<div class="col-lg-8">
<div class="portfolio-details-slider swiper">
<div class="swiper-wrapper align-items-center">
<div class="swiper-slide">
<img src="images/house.png" alt="">
</div>
<div class="swiper-slide">
<img src="images/housing/scatterPost.png" alt="">
</div>
<div class="swiper-slide">
<img src="images/housing/output.png" alt="">
</div>
</div>
<div class="swiper-pagination"></div>
</div>
</div>
<div class="col-lg-4">
<div class="portfolio-info">
<h3>Project information</h3>
<ul>
<li><strong>Category</strong>: Real Estate </li>
<li><strong>Tags</strong>: Feature Engineering, Ensemble Methods, Tuning </li>
<li><strong>Link to dataset </strong>:<a
href="https://github.com/ageron/handson-ml2/tree/master/datasets/housing"
target="_blank"> <em> Click here </em> </a> </li>
<p class="pt-3"><a class="btn btn-primary btn js-scroll px-4" href="https://github.com/zohaibdr/Portfolio/tree/master/Calif-Housing-Pred" target="_blank"
role="button"> See Code on Github </a></p>
</ul>
</div>
<div class="portfolio-description">
</div>
</div>
<div class="col-lg-10">
<h2>Context</h2>
<p>Estimating a house price in an area is easy if you have a large data set of historical prices against
a myriad of features (square footage, locality, number of rooms, etc.). Trouble is, such data is not
generally available to everybody.</p>
<p>But what is available publically is the census data...with only a limited set attribute, relating to
the locality/district.</p>
<br />
<h2>Problem Statement</h2>
<p> <b>Given the following set of attribute of the district, can we predict house price in the area accurately?</b></p>
<ol>
<li>Median income in the district</li>
<li>Median house age</li>
<li>Total number of rooms in the district</li>
<li>Total number of bedrooms in the district</li>
<li>District population</li>
<li>Total district house occupancy</li>
<li>District latitude</li>
<li>District longitude</li>
<li>Ocean proximity</li>
</ol>
<header>
<h1>Data Analysis</h1>
<p>An Exploratory data analysis reveals the distribution of data and the interrelation between
various attributes. To avoid data leakage, I split the data into train/validation/test sets even
before EDA.</p>
<br>
<p>In the data set, the features of 'total_bedrooms', 'population' and 'households' are highly
correlated as indicated. This information allows reducing the number of features later on by
eliminating some of the redundant features!</p>
<br>
<p><img src="images/housing/Correlation plot 1_edit.png" alt="" /></p>
<p>Next, it is useful to know which features are strongly correlated with the <strong>Target
variable</strong>. For this, I created the table below:</p>
<p><img src="images/housing/top5corr.png" alt="" /></p>
<p>Turns out, the Median income is the single most important determinant of house price. Moreover,
the fact of being away from the ocean (feature 'OceanIn') is negatively correlated, which make
sense.</p>
<br>
<h2>Introducing custom features</h2>
<p>In order to improve the accuracy of machine learning models, a big part of modeling consists of
creating new, more relevant features from raw data. For example, total number of rooms in a
block are not useful attributes in determining house value unless we know the total housholds
there are in that block. Similarly, population per house is more logical than total block
population.</p>
<p>I introduced two features namely <strong>'rooms_per_household'</strong> and
<strong>'population_per_household'</strong> and see their correlation with the target.</p>
<pre>
<header>
#create new variables first for new attributes
NewAttrib1 = Xy_train_reduced['total_rooms']/Xy_train_reduced['households']
NewAttrib2 = Xy_train_reduced['population']/Xy_train_reduced['households']
#creating new variables allows inserting them at the specified location in the our dataframe like here, just before last column (target variable)
Xy_train_reduced.insert(loc = len(Xy_train_reduced.columns.values)-1,
column = 'rooms_per_household', value=NewAttrib1)
Xy_train_reduced.insert(loc = len(Xy_train_reduced.columns.values)-1,
column = 'popolation_per_household', value=NewAttrib2)
</code>
</pre>
It gives:
<p> <img src="images/housing/top5corrNew.png" alt="" /> </p>
Clearly, they have some correlation, if not very strong, to the house price. They will likely
improve prediction accuracy.
<h2> Scaling </h2>
Lastly, I scaled the data using 'StandardScaler' from sklearn library. <br>
<pre>
<code>
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train_reduced) # fit only on training data
X_train_scaled = scaler.transform(X_train_reduced)
# Now apply same transformation to validation and test sets
X_test_scaled = scaler.transform(X_test_reduced)
X_valid_scaled = scaler.transform(X_valid_reduced)
</code>
</pre>
<!-- <hr /> -->
<h1>Results</h1>
<h2>Establishing baseline </h2>
It is best to first establish a baseline for prediction. I use linear regression as a baseline model
in this case. <br>
<br />
<blockquote>
<strong> The cross validated R2 score on the training set is 0.62. The R2 score on test set is
0.63. </strong> <br>
</blockquote>
Below is the regression plot, for the test set. The baseline model predicts house value with a typical (RMSE) error of <strong>US$70k</strong>.
<p> <img src="images/housing/regressPlot.png" alt="" /> </p>
<br/> <br/>
<p> The residual plot (left) shows that the model largely follows the assumption of <strong>homoscedasticity</strong>. The Q-Q plot on the right suggests that the assumptions of <strong>normality</strong> and linearity have also not been violated.
</p>
<br/> <br/>
<p> <img src="images/housing/output.png" alt="" /> </p>
<br/> <br/>
<h2> Improving predictions </h2>
<p> In order to improve upon the prediction accuracy, we could try to optimize hyper-parameters using the SGD regressor (scikit-learn). However, the strategy here is to first find the most powerful
algorithm. </p>
<p> Turning to tree-based and ensemble methods, below are the train and test accuracies for <strong>5 popular algorithms</strong>: </p>
<span> <img src="images/housing/treeR2_unoptim.png" alt="" /> </span>
<strong>All models are overfitting the data at default values which is Normal. </strong>
<br/> <br/>
<h2> Model Tuning </h2>
I tuned the models by adjusting their hyper-parameters using 'RandomSearchCV'.
<pre>
<code>
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.pipeline import Pipeline
DecTree = DecisionTreeRegressor(random_state=10)
RFreg = RandomForestRegressor(random_state=10)
GBreg = GradientBoostingRegressor(random_state=10)
XGBoostreg = XGBRegressor(random_state=10)
param1 = {} #Decision tree
param1['regressor'] = [DecTree]
param1['regressor__max_depth'] = np.arange(2,8)
param1['regressor__min_samples_split']= np.arange(2, 8)
param1['regressor__min_samples_leaf']= np.arange(5, 10)
# param1['regressor__min_impurity_decrease'] = [0, 0.001, 0.002, 0.003]
param2 = {} #Random forests
param2['regressor'] = [RFreg]
param2['regressor__max_depth'] = np.arange(2,8)
param2['regressor__min_samples_split']= np.arange(2, 8)
param2['regressor__min_samples_leaf']= np.arange(5, 10)
param2['regressor__n_estimators'] = [10, 50, 100]
param3 = {} #Gradient Boost
param3['regressor'] = [GBreg]
param3['regressor__max_depth'] = np.arange(2,8)
param3['regressor__n_estimators'] = [50, 100, 200]
param3['regressor__min_samples_split']= np.arange(1, 1000, 100)
param3['regressor__min_samples_leaf']= np.arange(1, 1000, 100)
param4 = {} #XG Boost
param4['regressor'] = [XGBoostreg]
param4['regressor__max_depth'] = np.arange(2,10)
pipeline = Pipeline([('regressor', DecTree)])
params = [param1, param2, param3, param4]
randomSearch = RandomizedSearchCV(pipeline, params, scoring=None, cv=5, n_jobs= -1, verbose = 2, refit = True, return_train_score=True, n_iter = 20, random_state=10)
</code>
</pre>
The resulting best model and optimal parameters were: <br/> <br/>
<strong>GradientBoostingRegressor (max_depth=7,
min_impurity_decrease=0.002,
min_samples_leaf=200,
min_samples_split=300,
random_state=10))]) <br> </strong>
<br/>
The R2 score of the best estimator is: 0.750 <br>
The RMSE of the prediction is: US$36563 <br>
<p>
<blockquote>
<strong>Both of these scores are better than the baseline model (linear regression). </strong>
</blockquote>
</p>
Finally, I calculated and plotted the features importances from the model:
<span> <img src="images/housing/FeatImp.png" alt="" /> </span>
As expected, the median income is the biggest determinant of housing price in the area as also
indicated in the correlation plot.
</div>
</div>
</div>
</section>
<!-- End Portfolio Details Section -->
<!-- adding some space between the end of report and the footer -->
<div class="col-lg-8">
<br>
<br>
<br>
</div>
<!-- ======= Footer ======= -->
<footer>
<div class="container">
<div class="row">
<div class="col-sm-12">
<div class="copyright-box">
<p class="copyright">© All Rights Reserved</p>
<div class="credits">
<!--
All the links in the footer should remain intact.
You can delete the links only if you purchased the pro version.
Licensing information: https://bootstrapmade.com/license/
Purchase the pro version with working PHP/AJAX contact form: https://bootstrapmade.com/buy/?theme=DevFolio
-->
Design credit to <a href="https://bootstrapmade.com/" target="_blank"> BootstrapMade</a>
</div>
</div>
</div>
</div>
</div>
</footer><!-- End Footer -->
<div id="preloader"></div>
<a href="#" class="back-to-top d-flex align-items-center justify-content-center"><i class="bi bi-arrow-up-short"></i></a>
<!-- Vendor JS Files -->
<script src="assets/vendor/purecounter/purecounter_vanilla.js"></script>
<script src="assets/vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<script src="assets/vendor/glightbox/js/glightbox.min.js"></script>
<script src="assets/vendor/swiper/swiper-bundle.min.js"></script>
<script src="assets/vendor/typed.js/typed.min.js"></script>
<script src="assets/vendor/php-email-form/validate.js"></script>
<!-- Template Main JS File -->
<script src="assets/js/main.js"></script>
</body>
</html>