-
Notifications
You must be signed in to change notification settings - Fork 1
/
eda_r.qmd
521 lines (398 loc) · 21.7 KB
/
eda_r.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
# Exploratory Data Analysis {#sec-eda}
{{< include _links.qmd >}}
```{r}
#| label: setup
#| cache: false
#| output: false
#| code-fold: true
#| code-summary: 'Setup Code (Click to Expand)'
# packages needed to run the code in this section
# install.packages(c("tidyverse", "janitor", "skimr", "remotes"))
# remotes::install_github("StatsGary/MLDataR")
# remotes::install_github("NHS-South-Central-and-West/scwplot")
# import packages
suppressPackageStartupMessages({
library(dplyr)
library(ggplot2)
library(scwplot)
})
# set plot theme
theme_set(scwplot::theme_scw(base_size = 16))
# import data
df <- MLDataR::heartdisease |>
janitor::clean_names()
# convert discrete variables to factor type
df <- df |>
mutate(
sex = as.factor(sex),
fasting_bs = as.factor(fasting_bs),
resting_ecg = as.factor(resting_ecg),
angina = as.factor(angina),
heart_disease = as.factor(heart_disease)
)
```
Exploratory data analysis (EDA) is the process of inspecting, visualising, and summarising a dataset. It is the first step in any data science project, and the importance of EDA can often be overlooked. Without exploring the data, it is difficult to know how to construct a piece of analysis or a model, and it is difficult to know if the data is suitable for the task at hand. As a critical step in the data science workflow, it is important to spend time on EDA and to be thorough and methodical in the process. While EDA is often the most time-consuming step in an analysis, taking the time to explore the data can save time in the long run.
EDA is an iterative process. In this tutorial, we will use the `dplyr` and `ggplot2` packages to explore a dataset containing information about heart disease. We will start by inspecting the data itself, to get a sense of the structure and the components of the dataset, and to identify any data quality issues (such as missing values). We will then compute summary statistics to get a better understanding of the distribution and central tendency of the variables that are relevant to the analysis. Finally, we will use data visualisations to explore specific variables in more detail, and to identify any interesting relationships between variables.
## Inspecting the Data
The first step when doing EDA is to inspect the data itself and get an idea of the structure of the dataset, the variable types, and the typical values of each variable. This gives a better understanding of exactly what data is being used and informs decisions both about the next steps in the exploratory process and any modelling choices.
We can use the `head()` and `glimpse()` functions to get a sense of the structure of the data. The `head()` function returns the first five rows of the data, and `glimpse()` returns a summary of the data, including the number of rows, the number of columns, the column names, the data type of each column, and the first few rows of the data. In addition to these two methods, we can use the `distinct()` function to get a list of all unique values of a particular variable. This is useful for discrete variables, such as the outcome variable, which can take on a limited number of values. For continuous variables (or any variables with a large number of unique values) the output of `distinct()` (a tibble) can be difficult to read, so we can use the `unique()` function to get a list of all unique values, which will be returned as a vector.
```{r}
#| label: explore
# view first rows in the dataset
head(df)
# overview of the data
glimpse(df)
# unique values of the outcome variable
df |>
distinct(heart_disease)
# unique values of a continuous explanatory variable
unique(df$cholesterol)
```
## Summary Statistics
Summary statistics are a quick and easy way to get a sense of the distribution, central tendency, and dispersion of the variables in the dataset. We can use the `summary()` function to get a summary of the data, including the mean and median values, the 1st and 3rd quartiles, and the minimum and maximum values of each numeric column. It also returns the count values for each factor column, and the number of NA values for each column.
While the base `summary()` function is pretty effective and works right out of the box, the package `skimr` can provide a more detailed summary of the data, using the `skim()` function. If you are looking for a single function to capture the entire process of inspecting the data and computing summary statistics, `skim()` is the function for the job, giving you a wealth of information about the dataset as a whole and each variable in the data.
If we want to examine a particular variable, the functions `mean()`, `median()`, `quantile()`, `min()`, and `max()` will return the same information as the `summary()` function. We can also get a sense of dispersion by computing the standard deviation or variance of a variable. The `sd()` function returns the standard deviation of a variable, and the `var()` function returns the variance.
Finally, we can use the `count()` function to get a count of the number of observations in each category of a discrete variable. Proportions can also be computed by dividing the count by the total number of observations. Using the `group_by()` function to group the data by a particular variable, and the `mutate()` function to add a new column to the data, we can compute the proportion as `n/sum(n)`.
```{r}
#| label: summary-stats
# summary of the data
summary(df)
# more detailed summary of the data
skimr::skim(df)
# mean age
mean(df$age)
# median age
median(df$age)
# min and max age
min(df$age)
max(df$age)
# dispersion of age
sd(df$age)
var(df$age)
# heart disease count
df |>
count(heart_disease)
# resting ecg count
df |>
count(resting_ecg)
# angina
df |>
count(angina)
# cholesterol
df |>
count(cholesterol)
# heart disease proportion
df |>
group_by(resting_ecg) |>
count(heart_disease) |>
mutate(freq = n/sum(n))
```
## Data Visualisation
While inspecting the data directly and using summary statistics to describe it is a good first step, data visualisation is a more effective way to explore the data. It allows us to quickly identify patterns and relationships in the data, and to identify any data quality issues that might not be immediately obvious without a visual representation of the data.
When using data visualisation for exploratory purposes, the intent is generally to visualise the way data is distributed, both within and between variables. This can be done using a variety of different types of plots, including histograms, bar charts, box plots, scatter plots, and line plots. How variables are distributed can tell us a lot about the variable itself, and how variables are distributed relative to each other can tell us a lot about the potential relationship between the variables.
In this tutorial, we will use the `ggplot2` package to create a series of data visualisations to explore the data in more detail. `ggplot2` is an incredibly flexible and powerful package for creating data visualisations. While it can be a little difficult to make sense of the syntax at first, it is well worth the effort to learn how to use it. Learning how to use `ggplot2` is beyond the scope of this tutorial, but there are a number of excellent resources available online, including the [ggplot2 documentation][ggplot2].
### Visualising Data Distributions
The first step in the exploratory process is to visualise the data distributions of key variables in the dataset. This allows us to get a sense of the typical values and central tendency of the variable, as well as identifying any outliers or other data quality issues.
#### Continuous Distributions
For continuous variables, we can use histograms to visualise the distribution of the data. We can use the `geom_histogram()` function to create a histogram of a continuous variable. The `binwidth` argument can be used to control the width of the bins in the histogram.
```{r}
#| label: fig-continuous-dists
#| layout-ncol: 3
#| column: screen-inset
#| fig-cap: |
#| Histograms plotting the distributions of relevant variables from the heart
#| disease dataset.
#| fig-subcap:
#| - Age
#| - Maximum Heart Rate
#| - Cholesterol
# age distribution
df |>
ggplot(aes(age)) +
geom_histogram(binwidth = 5, colour = "#333333", linewidth = 1) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
labs(x = NULL, y = NULL)
# max hr distribution
df |>
ggplot(aes(max_hr)) +
geom_histogram(binwidth = 10, colour = "#333333", linewidth = 1) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
labs(x = NULL, y = NULL)
# cholesterol distribution
df |>
ggplot(aes(cholesterol)) +
geom_histogram(binwidth = 25, colour = "#333333", linewidth = 1) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
labs(x = NULL, y = NULL)
```
```{r}
#| label: fig-filter-zeroes-cholesterol-dist
#| fig-cap: |
#| The distribution of cholesterol readings, with zero values filtered out to
#| better visualise the non-zero distribution.
# filter zero values
df |>
filter(cholesterol != 0) |>
ggplot(aes(cholesterol)) +
geom_histogram(binwidth = 25, colour = "#333333", linewidth = 1) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
labs(x = "Cholesterol", y = NULL)
```
The inflated zero values in the cholesterol distribution suggests that there may be an issue with data quality that needs addressing.
#### Discrete Distributions
We can use bar charts to visualise the distribution of discrete variables. We can use the `geom_bar()` function to create a bar chart of a discrete variable.
```{r}
#| label: fig-discrete-dists
#| column: page
#| layout-ncol: 2
#| fig-cap: |
#| Bar charts plotting discrete variables from the heart disease dataset.
#| fig-subcap:
#| - Patient Sex
#| - Angina
# sex distribution
df |>
ggplot(aes(sex)) +
geom_bar(colour = "#333333", linewidth = 1) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
labs(x = NULL, y = NULL)
# angina distribution
df |>
ggplot(aes(angina)) +
geom_bar(colour = "#333333", linewidth = 1) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
labs(x = NULL, y = NULL)
```
```{r}
#| label: fig-heart-disease-dist
#| fig-cap: |
#| The distribution of the outcome variable, heart disease.
# heart disease distribution
df |>
ggplot(aes(heart_disease)) +
geom_bar(colour = "#333333", linewidth = 1) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
labs(x = "Heart Disease", y = NULL)
```
### Comparing Distributions
There are a number of ways to compare the distributions of multiple variables. Bar charts can be used to visualise two discrete variables, while histograms and box plots are useful for comparing the distribution of a continuous variable across the groups of a discrete variable, and scatter plots are particularly useful for comparing the distribution of two continuous variables.
#### Visualising Multiple Discrete Variables
Bar charts are an effective way to visualize the observed relationship (or association, at least) between a discrete explanatory variable and a discrete outcome (whether binary, ordinal, or categorical).
We can use the `geom_bar()` function to create bar charts, but the default behaviour is to display the bars as stacked bars, which is not necessarily ideal for visualising discrete variables (though I'd recommend playing around with this yourself to decide what works in each case)
The `position` argument controls how the bars are displayed. The default `position = 'stack'` argument will display the bars as stacked bars, while the `position = 'dodge'` argument will display the bars side-by-side, and the `position = 'fill'` argument will display the bars as a proportion of the total number of observations in each category.
Finally, the `fill` argument splits the bars by a particular variable and display them in different colours.
```{r}
#| label: fig-multi-discrete-dists
#| column: page
#| layout-ncol: 2
#| layout-nrow: 2
#| fig-cap: |
#| Bar charts plotting discrete variables from the heart disease dataset
#| against the outcome variable, heart disease.
#| fig-subcap:
#| - Patient Sex
#| - Resting ECG
#| - Angina
#| - Fasting Blood Sugar
# heart disease by sex
df |>
ggplot(aes(heart_disease, fill = sex)) +
geom_bar(position = 'dodge', colour = "#333333", linewidth = 1) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
scale_fill_qualitative(palette = "scw") +
labs(x = "Heart Disease", y = NULL)
# resting ecg
df |>
ggplot(aes(heart_disease, fill = resting_ecg)) +
geom_bar(position = 'dodge', colour = "#333333", linewidth = 1) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
scale_fill_qualitative(palette = "scw") +
labs(x = "Heart Disease", y = NULL)
# angina
df |>
ggplot(aes(heart_disease, fill = angina)) +
geom_bar(position = 'dodge', colour = "#333333", linewidth = 1) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
scale_fill_qualitative(palette = "scw") +
labs(x = "Heart Disease", y = NULL)
# fasting bs
df |>
ggplot(aes(heart_disease, fill = fasting_bs)) +
geom_bar(position = 'dodge', colour = "#333333", linewidth = 1) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
scale_fill_qualitative(palette = "scw") +
labs(x = "Heart Disease", y = NULL)
```
#### Visualising A Continuous Variable Across Discrete Groups
Histograms and box plots are useful for comparing the distribution of a continuous variable across the groups of a discrete variable.
##### Histogram Plots
We can use the `geom_histogram()` function to create histograms. The `fill` and `position` arguments can be used to split the bars by a particular variable and display them in different colours, as discussed above.
```{r}
#| label: fig-continuous-discrete-dists
#| column: page
#| layout-ncol: 2
#| fig-cap: |
#| Histograms plotting the distribution of continuous variables from the heart
#| disease dataset against the outcome variable, heart disease.
#| fig-subcap:
#| - Patient Age by Heart Disease
#| - Cholesterol (Zero Values Filtered) by Heart Disease
# age distribution by heart disease
df |>
ggplot(aes(age, fill = heart_disease)) +
geom_histogram(
binwidth = 5, position = 'dodge',
colour = "#333333", linewidth = 1
) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
scale_fill_qualitative(palette = "scw") +
labs(x = NULL, y = NULL)
# filter zero values
df |>
filter(cholesterol != 0) |>
ggplot(aes(cholesterol, fill = heart_disease)) +
geom_histogram(
binwidth = 25, position = 'dodge',
colour = "#333333", linewidth = 1
) +
geom_hline(yintercept = 0, colour = "#333333", linewidth = 1) +
scale_fill_qualitative(palette = "scw") +
labs(x = NULL, y = NULL)
```
The fact that there is a significantly larger proportion of positive heart disease cases in the zero cholesterol values further demonstrates the need to address this data quality issue.
##### Box Plots
Box plots visualize the characteristics of a continuous distribution over discrete groups. We can use the `geom_boxplot()` function to create box plots, and the `fill()` argument to split the box plots by a particular variable and display them in different colours.
However, while box plots can be very useful, they are not always the most effective way of visualising this information, as explained [here][boxplots] by Cedric Scherer. This guide uses box plots for the sake of simplicity, but it is worth considering other options when visualising distributions.
```{r}
#| label: fig-box-plots
#| column: page
#| layout-ncol: 2
#| layout-nrow: 2
#| fig-cap: |
#| Box plots visualising continuous distributions over the discrete outcome
#| variable, heart disease.
#| fig-subcap:
#| - Patient Age
#| - Patient Age, Split by Sex
#| - Maximum Heart Rate
#| - Maximum Heart Rate, Split by Sex
# age & heart disease
df |>
ggplot(aes(age, heart_disease)) +
geom_boxplot(size=0.8) +
scale_fill_qualitative(palette = "scw") +
labs(x = "Patient Age", y = "Heart Disease")
# age & heart disease, split by sex
df |>
ggplot(aes(age, heart_disease, fill = sex)) +
geom_boxplot(size=0.8) +
scale_fill_qualitative(palette = "scw") +
labs(x = "Patient Age", y = "Heart Disease")
# max hr & heart disease
df |>
ggplot(aes(max_hr, heart_disease)) +
geom_boxplot(size=0.8) +
scale_fill_qualitative(palette = "scw") +
labs(x = "Maximum Heart Rate", y = "Heart Disease")
# max hr & heart disease, split by sex
df |>
ggplot(aes(max_hr, heart_disease, fill = sex)) +
geom_boxplot(size=0.8) +
scale_fill_qualitative(palette = "scw") +
labs(x = "Maximum Heart Rate", y = "Heart Disease")
```
#### Visualising Multiple Discrete Variables
Scatter plots are an effective way to visualize how two continuous variables vary together. We can use the `geom_point()` function to create scatter plots, and the `colour` argument to split the scatter plots by a particular variable and display them in different colours.
```{r}
#| label: fig-scatter-plots
#| column: page
#| layout-ncol: 2
#| layout-nrow: 2
#| fig-cap: |
#| Scatter plots visualising two continuous distributions together.
#| fig-subcap:
#| - Patient Age & Resting Blood Pressure
#| - Patient Age & Resting Blood Pressure (Zero Values Filtered)
#| - Patient Age & Cholesterol (Zero Values Filtered)
#| - Patient Age & Maximum Heart Rate
# age & resting bp
df |>
ggplot(aes(age, resting_bp)) +
geom_point(alpha = 0.8, size = 3, colour = 'gray30') +
labs(x = "Patient Age", y = "Resting Blood Pressure")
# filter zero values
df |>
filter(resting_bp != 0) |>
ggplot(aes(age, resting_bp)) +
geom_point(alpha = 0.8, size = 3, colour = 'gray30') +
labs(x = "Patient Age", y = "Resting Blood Pressure")
# age & cholesterol
df |>
filter(cholesterol != 0) |>
ggplot(aes(age, cholesterol)) +
geom_point(alpha = 0.8, size = 3, colour = 'gray30') +
labs(x = "Patient Age", y = "Cholesterol")
# age & max hr
df |>
ggplot(aes(age, max_hr)) +
geom_point(alpha = 0.8, size = 3, colour = 'gray30') +
labs(x = "Patient Age", y = "Maximum Heart Rate")
```
The scatter plot visualising age and resting blood pressure highlights another observation that needs to be removed due to data quality issues.
If there appears to be an association between the two continuous variables that you have plotted, as is the case with age and maximum heart rate in the above plot, you can also add a regression line to visualize the strength of that association. The `geom_smooth()` function can be used to add a regression line to a scatter plot. The `method` argument specifies the type of regression line to be added, and the `se` argument specifies whether or not to display the standard error of the regression line.
```{r}
#| label: fig-scatter-plot-with-regression-line
#| fig-cap: |
#| Scatter plot visualising the distribution of patient age and maximum heart
#| rate with a regression line fit to the data.
# age & max hr
df |>
ggplot(aes(age, max_hr)) +
geom_point(alpha = 0.8, size = 3, colour = 'gray30') +
geom_smooth(method = lm, se = FALSE, size = 2, colour='#005EB8') +
labs(x = "Patient Age", y = "Maximum Heart Rate")
```
You can also include discrete variables by assigning the discrete groups different colours in the scatter plot, and if you add regression lines to these plots, separate regression lines will be fit to the discrete groups. This can be useful for visualising how the association between the two continuous variables varies across the discrete groups.
```{r}
#| label: fig-scatter-plots-with-discrete-splits
#| column: screen-inset
#| layout-ncol: 3
#| fig-cap: |
#| Scatter plots visualising continuous distributions together, with the data
#| split and coloured by the discrete outcome variable, heart disease.
#| fig-subcap:
#| - Patient Age & Resting Blood Pressure
#| - Patient Age & Cholesterol (Zero Values Filtered)
#| - Patient Age & Maximum Heart Rate
# age & resting bp, split by heart disease
df |>
filter(resting_bp != 0) |>
ggplot(aes(age, resting_bp, colour = heart_disease)) +
geom_point(alpha = 0.8, size = 3) +
scale_colour_qualitative(palette = "scw") +
labs(x = "Patient Age", y = "Resting Blood Pressure")
# age & cholesterol, split by heart disease (with regression line)
df |>
filter(cholesterol!=0) |>
ggplot(aes(age, cholesterol, colour = heart_disease)) +
geom_point(size = 3, alpha = 0.8) +
geom_smooth(method = lm, se = FALSE, size = 1.5) +
scale_colour_qualitative(palette = "scw") +
labs(x = "Patient Age", y = "Resting Blood Pressure")
# age & max hr, split by heart disease (with regression line)
df |>
ggplot(aes(age, max_hr, colour = heart_disease)) +
geom_point(size = 3, alpha = 0.8)+
geom_smooth(method = lm, se = FALSE, size = 1.5) +
scale_colour_qualitative(palette = "scw") +
labs(x = "Patient Age", y = "Maximum Heart Rate")
```
## Next Steps
There are many more visualisation techniques that you can use to explore your data, and you can find a comprehensive list of them on the [ggplot2 function reference][geoms] page. There are also a wide variety of [ggplot extension packages][ggplot extensions] that can be used to create more complex visualisations.
The next step in the data science process is to build a model to either explain or predict the outcome variable, heart disease. The exploratory work done here can help inform decisions about the choice of the model, and the choice of the variables that will be used to build the model. It will also help clean up the data, particularly the zero values in the cholesterol and resting blood pressure variables, to ensure that the model is built on the best possible data.
## Resources
There are a wealth of resources available to help you learn more about data visualisation.
- [Data Visualization: A Practical Introduction]
- [Fundamentals of Data Visualization]
- [Data Visualization with R]
- [{ggeasy} - Easy Access to {ggplot2} Commands]