/
machinelearning.qmd
117 lines (71 loc) · 14.3 KB
/
machinelearning.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
---
abstract: TODO (150-200 WORDS)
---
{{< include _setup.qmd >}}
# Statistical Learning {#sec-ml}
{{< include _wip.qmd >}}
___
TODO
* $\calD$ - labeled data set with $n$ observation
* Observations: $(\xx^{(i)}, y^{(i)})$ where $\xx^{(i)} \in \calX$ is a $p$-dimensional feature vector and $y^{(i)} \in \calY$ is a label
* Data set: $\calD = ((\xx^{(1)}, y^{(1)}) , . . . , (\xx^{(n)}, y^{(n)}))$
* Assume $\calD \iid (\mathbb{P}_{xy})^n$ (unknown distribution)
* ML model: $f : \calX \rightarrow \mathbb{R}^g$ assigning a prediction in $\mathbb{R}^g$
* ML learners, $\calI$, configured by hyperparameters $\lambda \in \Lambda$, are $\calI: \mathbb{D} \times \Lambda \rightarrow \calH, (\calD, \lambda) \mapsto \hatf$ where $\calH$ is the function space to which a model belongs and $\hatf$ is a trained model with learned hyperparameters $\hat{\theta} \in \calH$.
* Models are evaluated with loss functions which measure the discrepancy between predictions and true values $L: \calY \times \Reals^g \rightarrow \Reals, (\hatf(\xx), \yy) \mapsto l(\hatf(\xx), \yy)$
* To prevent overfitting, models are evaluated on unseen test data to ensure unbiased performance estimation and the generalization error $GE(\calI, \lambda, n_{train}, l) := \lim_{n_{test} \rightarrow \infty} \EE_{\calD_{train},\calD_{test}\sim\PP_{xy}}[l(\yy_{test}, \mathbf{F}_{\calD_{test},\calI(\calD_{train},\lambda)})]$ where $\mathbf{F}_{\calD_{test},\calI(\calD_{train},\lambda)}$ is the matrix of predictions from a model trained on $\calD_{train}$ and making predictions on $\calD_{test}$.
___
## Machine Learning {#sec-surv-setml}
This section begins with a very brief introduction to machine learning and a focus on regression and classification; the survival machine learning task is then introduced (@sec-surv-setmltask). Of the many fields within machine learning (ML), the scope of this book is narrowed to supervised learning. Supervised learning is the sub-field of ML in which predictions are made for outcomes based on data with observed dependent and independent variables. For example predicting someone's height is a supervised learning problem as data can be collected for features (independent variables) such as age and sex, and outcome (dependent variable), which is height. Predictive survival analysis problems fall naturally in the supervised learning framework as there are identifiable features and (multiple types of) outcomes.
### Terminology and Methods {#sec-surv-setml-meth}
Common supervised learning methods are discussed in a simplified setting with features $X \ t.v.i. \ \calX$ and outcomes $Y \ t.v.i. \ \calY$; usually outcomes are referred to as 'targets' (a 'target for prediction'). Let $\calD_0 = \{(X_1,Y_1),...,(X_n,Y_n)\}$ be a (training) dataset where $(X_i, Y_i) \iid (X, Y)$. The methods below extend naturally to the survival setting.
#### Strategies and Models {.unnumbered .unlisted}
In order to clearly separate between similar objects, several terms for machine learning are now introduced and clearly distinguished.
Let $g: \calX \rightarrow \calY$ be the true (but unknown) mapping from the features to outcomes, referred to as the *true prediction functional*. Let $\calG$ be the set of *prediction functionals* such that $\forall \Upsilon \in \calG, \Upsilon: \calX \rightarrow \calY$. A *learning* or *fitting algorithm* is defined to be any function of the form $\calA: \calX^n \times \calY^n \rightarrow \calG$. The goal of supervised learning is to *learn* $g$ with a learning algorithm *fit* on (i.e. the input to the algorithm is) training data, $\hatg := \calA(\dtrain) \in \calG$. Note that $\hatg$ may take hyper-parameters that can be set or tuned (see below). The learning algorithm is 'good' if $\hatg(X) \approx g(X)$ (see 'Evaluation' below).
The learning algorithm is determined by the chosen *learning strategy* and *model*, where a model is a complete specification of a learning strategy including hyper-parameters. These terms are more clearly illustrated by example:
i. Learning strategy -- simple linear regression
i. Model -- $y = \beta_0 + \beta_1 x$ where $x \in \Reals$ is a single covariate, $y \in \Reals$ is the target, and $\beta_0,\beta_1 \in \Reals$ are model coefficients.
i. Learning algorithm (model fitting) -- Minimise the residual sum of squares: $(\hat{\beta_0}, \hat{\beta_1}) := \argmin_{\beta_0,\beta_1} \{\sum^n_{i=1} (y_i - \beta_0 - \beta_1 x_i)^2\}$ for $(x_i,y_i) \in \dtrain, i = 1,...,n$.
i. Prediction functional -- $\hatg(x) = \hat{\beta_0} + \hat{\beta_1}x$
To further illustrate the difference between learning strategy and model, note that the same learning strategy 'simple linear regression' could either utilise the model above or instead a model without intercept, $y = \beta x$, in which case the learning algorithm and prediction functional would also be modified.
The model in (ii) is called *unfitted* as the model coefficients are unknown and the model cannot be used for prediction. After step (iii) the model is said to be fit to the training data and therefore the model is *fitted*.\footnote{The terms 'fitted' and 'unfitted' are used instead of 'fit' and 'unfit' to prevent confusion with words such as 'suitable' and 'unsuitable'.} It is common to refer to the learning algorithm (and associated hyper-parameters) as the unfitted model and to refer to the prediction functional (and associated hyper-parameters) as the fitted model.
#### Evaluation {.unnumbered .unlisted}
Models are *evaluated* by evaluation measures called *losses* or *scores*,\footnote{The term 'loss' is usually utilised to refer to evaluation measures to be minimised, whereas 'scores' should be maximised, this is returned to in (@sec-eval).} $L: \calY \times \calY \rightarrow \ExtReals$. Let $(X^*, Y^*) \sim (X,Y)$ be test data (i.e. independent of $\dtrain$) and let $\hatg: \calX \rightarrow \calY$ be a prediction functional fit on $\dtrain$, then these evaluation measures determine how closely predictions, $\hatg(X^*)$, relate to the truth, $Y^*$, thereby providing a method for determining if a model is 'good'.\footnote{Here evaluation refers specifically to predictive ability; other forms of evaluation and further discussion of the area are provided in (@sec-eval).}
#### Task {.unnumbered .unlisted}
A machine learning *task* is a simple mechanism to outline the problem of interest by providing: i) the data specification; ii) the definition of learning; iii) the definition of success (when is a prediction 'good'?) [@Kiraly2021]. All tasks in this paper have the same definitions of learning and success. For (ii), the aim is to learn the true prediction functional, $g$, by fitting the learning algorithm on training data, $\hatg := \calA(\calD_0)$. For (iii), a predicted functional is considered 'good' if the *expected generalization error*, $\EE[L(Y^*, \hatg(X^*))]$, is low, where $(X^*, Y^*) \sim (X,Y)$ is independent of the training data $\calD_0$, and $L$ is some loss that is chosen according to the domain of interest (regression, classification, survival).
#### Resampling {.unnumbered .unlisted}
Models are *tested* on their ability to make predictions. In order to avoid 'optimism of training error' [@Hastie2013] -- overconfidence caused by testing the model on training data -- models are tested on previously unseen or 'held-out' data. *Resampling* is the procedure of splitting one dataset into two or more for separated training and testing. In this paper only two resampling methods are utilised: *holdout* and *cross-validation*. Holdout is the process of splitting a primary dataset into training data for model fitting and testing data for model predicting. This is an efficient method but may not accurately estimate the expected generalisation error for future model performance, instead this is well-estimated by $K$-fold cross-validation (KCV) [@Hastie2001]. In KCV, data is split into $K \in \PNaturals$ 'folds' such that $K-1$ of the folds are used for model training and the final $K$th fold for testing. The testing fold is iterated over all $K$ folds, so that each at some point is used for testing and then training (though never at the same time). In each iteration the model is fit on the training folds, and predictions are made and evaluated on the testing fold, giving a loss $L_k := L(\hatg(X^k), Y^k)$, where $(X^k, Y^k)$ are data from the $k$th fold. A final loss is defined by, $L^* := \frac{1}{K} \sum^K_{k = 1} L_k$. Commonly $K = 5$ or $K = 10$ [@Breiman1992; @Kohavi1995].
#### Model Performance Benchmarking {.unnumbered .unlisted}
Whilst *benchmarking* often refers to speed tests, i.e. the time taken to complete an operation, it can also refer to any experiment in which objects (mathematical or computational) are compared. In this report, a benchmark experiment will either refer to the comparison of multiple models' predictive abilities, or comparison of computational speeds and object sizes for model fitting; which of these will be clear from context.
#### Model Comparison {.unnumbered .unlisted}
Models can be analytically compared on how well they make predictions for new data. Model comparison is a complex topic with many open questions [@Demsar2006; @Dietterich1998; @Nadeau2003] and as such discussion is limited here. When models are compared on multiple datasets, there is more of a consensus in how to evaluate models [@Demsar2006] and this is expanded on further in [@Sonabend2021b]. Throughout this book there are small simulation experiments for model comparison on single datasets however as these are primarily intended to aid exposition and not to generalise results, it suffices to compare models with the conservative method of constructing confidence intervals around the sample mean and standard error of the loss when available [@Nadeau2003].
#### Hyper-Parameters and Tuning {.unnumbered .unlisted}
A *hyper-parameter* is a model parameter that can be set by the user, as opposed to coefficients that are estimated as part of model fitting. A hyper-parameter can be set before training, or it can be tuned. *Tuning* is the process of choosing the optimal hyper-parameter value via automation. In the simplest setting, tuning is performed by selecting a range of values for the hyper-parameter(s) and treating each choice (combination) as a different model. For example if tuning the number of trees in a random forest (@sec-surv-ml-models-ranfor), $m_r$, then a range of values, say $100, 200, 500$ are chosen, and three models $m_{r100}, m_{r200}, m_{r500}$ are benchmarked. The optimal hyper-parameter is given by whichever model is the best performing. *Nested resampling* is a common method to prevent overfitting that could occur from using overlapping data for tuning, training, or testing. Nested resampling is the process of resampling the training set again for tuning.
\newpage
### Machine Learning in Classification and Regression {#sec-surv-ml-car}
Before introducing machine learning for survival analysis, which is considered 'non-classical', the more standard classification and regression set-ups are provided; these are referenced throughout this book.
#### Classification {#sec-surv-ml-car-class}
Classification problems make predictions about categorical (or discrete) events, these may be *deterministic* or *probabilistic*. Deterministic classification predicts which category an observation falls into, whereas probabilistic classification predicts the probability of an observation falling into each category. In this brief introduction only binary single-label classification is discussed, though the multi-label case is considered in \ref{sec:car_reduxes_r7_mlc}. In binary classification, there are two possible categories an observation can fall into, usually referred to as the 'positive' and 'negative' class. For example predicting the probability of death due to a virus is a probabilistic classification task where the 'positive' event is death.
A probabilistic prediction is more informative than a deterministic one as it encodes uncertainty about the prediction. For example it is clearly more informative to predict a $70%$ chance of rain tomorrow instead of simply 'rain'. Moreover the latter prediction implicitly contains an erroneous assumption of certainty, e.g. 'it will rain tomorrow'.
:::: {.callout-note icon=false}
## Classification Task
::: {#cnj-task-classif}
Let $(X,Y)$ be random variables t.v.i. $\calX \times \calY$ where $\calX \subseteq \Reals^p$ and $\calY = \{0, 1\}$. Then,
* The *probabilistic classification task* is the problem of predicting the probability of a single event taking place and is specified by $g: \calX \rightarrow [0, 1]$.
* The *deterministic classification task* is the problem of predicting if a single event takes place and is specified by $g: \calX \rightarrow \calY$.
The estimated prediction functional $\hatg$ is fit on training data \\$(X_1,Y_1),...,(X_n,Y_n) \iid (X,Y)$ and is considered 'good' if $\EE[L(Y^*, \hatg(X^*))]$ is low, where $(X^*, Y^*) \sim (X, Y)$ is independent of $(X_1,Y_1),...,(X_n,Y_n)$ and $\hatg$.
In the probabilistic case, the prediction $\hat{g}$ maps to the estimated probability mass function $\hat{p}_Y$ such that $\hat{p}_Y(1) = 1 - \hat{p}_Y(0)$.
:::
::::
#### Regression {#sec-surv-ml-regr}
A regression prediction is one in which the goal is to predict a continuous outcome from a set of features. For example predicting the time until an event (without censoring) occurs, is a regression problem.
:::: {.callout-note icon=false}
## Regression Task
::: {#cnj-task-regr}
Let $(X,Y)$ be random variables t.v.i. $\calX \times \calY$ where $\calX \subseteq \Reals^p$ and $\calY \subseteq \Reals$. Let $\calS \subset \Distr(\calY)$ be a convex set of distributions on $\calY$. Then,
* The *probabilistic regression task* is the problem of predicting a conditional distribution over the Reals and is specified by $g : \calX \rightarrow \calS$.
* The *deterministic regression task* is the problem of predicting a single continuous value in the Reals and is specified by $g: \calX \rightarrow \calY$.
The estimated prediction functional $\hatg$ is fit on training data \\$(X_1,Y_1),...,(X_n,Y_n) \iid (X,Y)$ and is considered 'good' if $\EE[L(Y^*, \hatg(X^*))]$ is low, where $(X^*, Y^*) \sim (X, Y)$ is independent of $(X_1,Y_1),...,(X_n,Y_n)$ and $\hatg$.
:::
::::
Whilst regression can be either probabilistic or deterministic, the latter is much more common and therefore in this book 'regression' refers to the deterministic case unless otherwise stated.