regression_prediction.Rmd

---
title: "Regression and Predictions"
output:
  html_notebook: default
---

[Link to Presentation Slides](https://github.com/mnathvt/regression_and_predictions/blob/main/regression_and_predictions.pdf)

## Contents:

This notebook contains the following topics:

-   [Linear Regression]

    -   [Simple Linear Regression]
    -   [Multiple Linear Regression]

-   [Real-world Example using Regression]

-   [Factor Variables]

-   [Regression Diagnostics - Outliers, Influential Values, Correlated Errors]

-   [Additional Points to Remember]

    -   [Multi-collinearity]

    -   [Bias-Variance Trade-off]

    -   [Regularization - Ridge and Lasso Regression]

-   [Polynomial and Spline Regression]

<br>

## Linear Regression

[**Aim**]{.ul}: to establish a linear relationship (a mathematical formula) between the predictor variable(s) and the response variable, to estimate the value of the response (outcome of dependent) variable $y$, when only the predictors (explanatory or independent variables) $x$ values are known.

Given a data set of $n$ data points, $\{y_{i},\,x_{i1},\ldots ,x_{ip}\}_{i=1}^{n},$ the linear model with $p$-vector of predictors $x$ is:

${y}_{i}=\beta _{0}+\beta _{1}x_{i1}+\cdots +\beta _{p}x_{ip}+ \epsilon_{i} = \mathbf{x}_{i}^{\mathsf {T}}{\boldsymbol {\beta }} + \epsilon_{i},\qquad i=1,\ldots ,n,$

where $\mathbf{x}_{i}^{\mathsf {T}} \beta$ is the inner product between vectors $\mathbf{x}_i$ and $\boldsymbol {\beta}$ and $\epsilon$ is the error variable, an unobserved random variable that adds 'noise' to the linear relationship.

In matrix notation, ${\mathbf{y = X}} {\boldsymbol{\beta}} + {\bf{\epsilon}}$, where,

$\mathbf{y} = \begin{pmatrix} y_1 \\ y_2 \\.\\.\\.\\ y_n \end{pmatrix}, \qquad {\boldsymbol {\beta}} = \begin{pmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\.\\.\\.\\ \beta_p \end{pmatrix}, \qquad {\bf{\epsilon}} = \begin{pmatrix} \epsilon_1 \\ \epsilon_2 \\.\\.\\.\\ \epsilon_n \end{pmatrix}$

${\bf{X}} = \begin{pmatrix} \bf{x}_1^{\mathsf{T}} \\ \bf{x}_2^{\mathsf{T}} \\.\\.\\.\\ \bf{x}_n^{\mathsf{T}} \end{pmatrix} = \begin{pmatrix} 1 & x_{11} & \ldots & x_{1p} \\ 1 & x_{21} & \ldots & x_{2p} \\. & . & \ldots & .\\. & . & \ldots & .\\. & . & \ldots & . \\ 1 & x_{n1} & \ldots & x_{np} \end{pmatrix}$

Summarize:

-   $\mathbf{X}$ - independent predictors
-   $\mathbf{y}$ - dependent response
-   $\boldsymbol{\beta}$ - regression coefficients
-   linear with respect to regression coefficients
-   distribution of estimated coefficients, $\widehat{\boldsymbol{\beta}}$ will depend on the distribution of $y$

The best fitted model - one which minimizes the sum of squared prediction errors/residuals, i.e., minimizes $\sum_{i=1}^n \epsilon_i^2 = \sum_{i=1}^n (y_i - \widehat{y}_i)^2$

(i.e., minimizes the least squared error). Here $\widehat{y}_i$ are the predicted responses.

<br> <br>

#### [Linear Regression has the following assumptions:]{.ul}

-   errors/residuals, $\epsilon_i$, are *identically and independently distributed*
-   errors/residuals, $\epsilon_i$, are normally distributed, with mean 0 and equal (unknown) variance - *homoscedasticity*.

<br> <br>

### Simple Linear Regression

Simple linear regression attempts to model the data in the form of the best fitting line:

${y}_i = \beta_0 + \beta_1 x_i + \epsilon_i \qquad i = 1,…, n$

This is the equation of a straight line, recall $y = mx +c$, where $m$ is the slope and $c$ is the intercept.

Minimization $\rightarrow$ take derivatives of $\sum_{i=1}^n \epsilon_i^2$ w.r.t $\beta_0$ and $\beta_1$, set to 0 and solve for $\beta_0$ and $\beta_1$.

$\widehat\beta_0 = \overline{y} - \widehat\beta_1 \overline{x}$

$\widehat\beta_1 = \frac{\sum_{i=1}^n (x_i - \overline{x} )(y_i - \overline{y} )}{\sum_{i=1}^n (x_i - \overline{x})^2}$

where,

$\overline{x}$ is the mean of all of the x-values

$\overline{y}$ is the mean of all of the y-values

$\widehat\beta_0$ tells the estimated regression equation at $x = 0$, if the 'scope of the model' includes $x = 0$, otherwise, $\widehat\beta_0$ is not meaningful.

$\widehat\beta_1$ is the amount by which the mean response vary for every one unit increase in $x$.

<br> <br>

### Multiple Linear Regression

Multiple linear regression is a generalized version of simple regression, where more than one independent variables are used. . The basic model for multiple linear regression is as discussed above:

${y}_{i}=\beta _{0}+\beta _{1}x_{i1}+\cdots +\beta _{p}x_{ip}+ \epsilon_{i} = \mathbf{X}_{i}^{\mathsf {T}}{\boldsymbol {\beta }} + \epsilon_{i},\qquad i=1,\ldots ,n,$

The solution for the coefficients can be easily calculated using the matrix formulation: $\hat{\boldsymbol\beta} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$

Generally, all real-world problems involve multiple predictors.

<br> <br> <br>

## Real-world Example using Regression

Data: New York air quality data in R ([download data from Kaggle](https://www.kaggle.com/pratiksagrawal/practice-data-set-for-air-quality?select=airquality.csv))

-   Ozone: Mean ozone in parts per billion from 1300 to 1500 hours at Roosevelt Island

-   Solar.R: Solar radiation in Langleys in the frequency band 4000--7700 Angstroms from 0800 to 1200 hours at Central Park

-   Wind: Average wind speed in miles per hour at 0700 and 1000 hours at LaGuardia Airport

-   Temp: Maximum daily temperature in degrees Fahrenheit at LaGuardia Airport

-   Month: Numeric value of Month (1--12)

-   Day: Day of month (1--31).

There are 6 variables - our target $\rightarrow$ attribute 'ozone'.

<br>

`lm()` function creates the relationship model between the predictor and the response variable, where the parameters used are -

-   formula is a symbol presenting the relation between $x$ and $y$.

`predict(object, newdata)` function is used for prediction, where the parameters used −

-   object is the formula which is already created using the `lm()` function.
-   newdata is the vector/matrix containing the new value for predictor variable.

```{r}
data('airquality')
str(airquality)
```

```{r}
head(airquality)
```

<br>

**Install necessary libraries**

```{r}

library(ggplot2)
library(olsrr)
library(glmnet)
library(splines)
library(car)
```

<br>

#### Exploratory Analysis and Pre-processing the data

Check for missing values in the dataset.

```{r}
mapply(anyNA, airquality)
```

There are some missing values in columns 'Ozone' and 'Solar.R'. Let's impute these by replacing the *'NA'* with monthly average.

```{r}
for (i in 1:nrow(airquality)){
  if(is.na(airquality[i,'Ozone'])){
    airquality[i,'Ozone'] = mean(airquality[
      which(airquality[, 'Month'] == airquality[i, 'Month']), 
            'Ozone'],na.rm = TRUE)
  }
  
  if(is.na(airquality[i,'Solar.R'])){
    airquality[i,'Solar.R'] = mean(airquality[
      which(airquality[, 'Month'] == airquality[i, 'Month']), 
            'Solar.R'],na.rm = TRUE)
  }
  
}
```

```{r}
head(airquality)
```

<br> <br>

#### Simple Regression on the data

Since simple regression requires only one predictor, let's take 'Temp' as the predictor for the target 'Ozone'.

```{r}
# select predictor attribute
x =  airquality[, 'Temp']
                
# select target attribute
y = airquality[, 'Ozone'] 
```

```{r}

simple_model  = lm(y~x)

# provides regression line coefficients, i.e., slope and y-intercept
simple_model
```

```{r}

# scatter plot between x and y
plot(y~x, xlab = 'Temp', ylab = 'Ozone')
# plot the regression line
abline(simple_model, col = 'blue', lwd = 2)
```

```{r}
summary(simple_model)
```

<br>

#### Interpretation of the output

**Call**: Formula used to fit the data.

<br>

**Residuals**: Difference between the actual observed response values and the response values that the model predicted. This section broken into 5 summary points, which tells the distribution of the residuals. Plotting the residuals will help check whether they are normally distributed.

<br>

**Coefficients**: This section tells the coefficients of the model. In simple linear regression, the coefficients are two unknown constants that represent the *intercept* and *slope* terms in the linear model.

The coefficient standard error measures the standard deviation of the coefficient estimates.

The coefficient t-value is a measure of how many standard deviations the coefficient estimate is far away from 0. The farther away from zero indicates that the null hypothesis (there is no relationship between the predictor and the response) can be rejected - that is, declare a linear relationship between ozone numbers and temperature exists.

The Pr(\>\|t\|) relates to the probability of observing any value equal or larger than $t$. A small p-value indicates that it is unlikely to observe a relationship between the predictor (temperature) and response (ozone) variables due to chance. Typically, a p-value of 5% or less is a good cut-off point. In this example, the p-values are very close to zero.\
Note: 'Signif. Codes' associated to each estimate. Three stars (or asterisks) represent a highly significant p-value. Consequently, a small p-value for the intercept and the slope indicates that the null hypothesis can be rejected.

<br>

**Residual Standard Error**: measure of the quality of a linear regression fit. Theoretically, every linear model is assumed to contain an error term. The residual standard error is the average amount that the response will deviate from the true regression line. This gives an estimate for the sigma squared for the residuals.

<br>

**Multiple R-squared, Adjusted R-squared**: R-squared statistic provides a measure of how well the model is fitting the actual data. It takes the form of a proportion of variance, lies between 0 and 1.

In multiple regression settings, the $R^2$ will always increase as more variables are included in the model. That's why the adjusted $R^2$ is the preferred measure as it adjusts for the number of variables considered.

<br>

**F statistic**: tells if there is a relationship between the dependent and independent variables being tested. Generally, a large F indicates a stronger relationship. The further the F-statistic is from 1 the better it is. However, how much larger the F-statistic needs to be depends on both the number of data points and the number of predictors. Generally, when the number of data points is large, an F-statistic that is only a little bit larger than 1 is already sufficient to reject the null hypothesis

```{r}
p = ggplot(data = airquality, aes(x, y)) + geom_point() +
stat_smooth(method = 'lm', col = 'dodgerblue3') +
theme(panel.background = element_rect(fill = 'white'),
axis.line.x=element_line(),
axis.line.y=element_line()) + ggtitle('Linear Model Fitted to Data')

p + labs(x = 'Temp', y = 'Ozone')
```

The gray shading around the line represents a confidence interval of 95%. In this example, the 95% confidence interval is the probability that the estimated regression line for the temperature will lie within this shaded region approximately 95 times if the experiment is repeated 100 times. There is still variability within the observations, even though the model fits the data well.

<br>

**Check how residuals are distributed**:

-   histogram (normal distribution)

-   normal Q-Q plot of residuals w.r.t. theoretical quantiles

-   standardized residual chart (standardized residual is the residual divided by estimated standard deviation)

```{r}
ggplot(data = airquality, aes(simple_model$residuals)) +
geom_histogram(aes(y=..density..), binwidth = 5, color = 'black', fill = 'grey') +
  geom_density(alpha=.2, fill='lightpink') +
  
theme(panel.background = element_rect(fill = 'white'),
axis.line.x=element_line(),
axis.line.y=element_line()) +
ggtitle('Histogram for Model Residuals')

```

```{r}
plot(simple_model, which = 2)
```

```{r}
ols_plot_resid_stand(simple_model)
```

<br> <br>

#### Multiple Regression on the data

```{r}
# select predictor attribute
x =  data.matrix(airquality[, 2:6])
                
# select target attribute
y = airquality[, 'Ozone']
```

```{r}
multi_model = lm(y~x, data = airquality)

summary(multi_model)
```

<br> <br> <br>

## Q&A

Questions?

<br> <br> <br>

## Factor Variables

These are categorical variables, and the interpretation of the coefficients for such predictors is different from the numerical predictors, i.e., the notion of a slope or unit change no longer makes sense when talking about a categorical variable. Both numeric and character variables can be made into factors.

In the dataset above, 'Month' is a categorical variable.

Most important advantage of converting categorical life to factor variable: they can be used in statistical modeling where they will be implemented correctly, i.e., storing data as factors ensures that the modeling functions will treat such data correctly, and then be assigned the correct number of degrees of freedom.

Ordered factor variables - when the categories of the factor variables have a particular order and it is used to perform comparisons. (Many R models ignore ordering even if present, by default, need to turn the ordering to TRUE.)

<br>

**Ways to handle factor variables**:

-   Dummy variables - a variable takes either $1$, indicating the presence of an attribute or $0$, indicating the absence of that attribute. Creates $k-1$ variables for $k$ categories of factor variables.\
    In the example, dummy variables for 'Month' would be

    -   January: coded '1' if January, 0 otherwise
    -   February: coded '1' if February, 0 otherwise etc.

-   One hot encoding - similar to dummy variables, i.e., it is also dichotomous variable $0$ or $1$. Creates $k$ variables for $k$ categories of factor variables.\
    In the example, one hot encoding would be

$$\begin{matrix} Jan & Feb & Mar \\ 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix}$$

-   Reference coding - a reference category is named and identified as a category of comparison for the other categories, i.e., the other categories are *compared to* the reference. (By default R uses the alpha-numerically first category as the reference category.)

    -   Simple coding: each group is compared to the reference group.

    -   Deviation coding: compares the mean of the dependent variable for a given level to the mean of the dependent variable for the other levels of the variable.

    -   Difference coding: each level is compared to the mean of the previous levels.

Note: Factor variables are hard to update, so it could be a pain during data cleaning.

<br> <br> <br>

## Regression Diagnostics - Outliers, Influential Values, Correlated Errors

**Outliers**: an observation which has a response value that is very different from the predicted value based on a model.

![](outlier.png)

<br>

**Influential values**: a data point which unduly influences any part of a regression analysis, such as the predicted responses, the estimated slope coefficients, or the hypothesis test results.

-   linear regression model is sensitive to outliers and high-leverage points (data points which have extreme predictor values)

-   methods to identify influential values:

    1.  Difference in Fits (DFFITS)

        An observation is deemed influential if

        $|DFFITS| > 2 * \frac{\sqrt{m+1}}{(n-m-1)}$

        where $n$ is the number of observations and $m$ is the number of predictors including intercept

    2.  Cook's Distance: a data point having a large Cook's d indicates that the data point strongly influences the fitted values.

-   In both methods, the following idea is used:

    Delete the observations one at a time, refit the regression model on the remaining $n-1$ observations each time. Then, compare the results using all $n$ observations to the results with the $i^{th}$observation deleted to see how much influence the observation has on the analysis. This enables to assess the potential impact each data point has on the regression analysis.

```{r}
ols_plot_dffits(simple_model)
```

```{r}
ols_plot_cooksd_chart(simple_model)
```

<br>

**Correlated errors**: The assumptions of linear regression state that the errors/residuals are uncorrelated. If correlation exists, it means that the model has not taken into account the additional information present in the data.

Easiest method to check: Plot the residuals to identify any non-random trends.

```{r}
plot(simple_model, which = 1)
```

Note: Time series problems have correlated errors.

<br> <br> <br>

## Additional Points to Remember

<br> <br>

### Multi-collinearity

When two or more of the predictors in a regression model are moderately or highly correlated with one another.

Computing the ***Variance Inflation Factor*** (VIF) for each independent variable helps detect multi-collinearity. A variance inflation factor (*VIF*) quantifies how much the variance is inflated. It exists for each of the predictors in a multiple regression model.\
For example, VIF for the estimated regression coefficient $\beta_j$ is just the factor by which the variance of $\beta_j$, denoted by $VIF_j$ is inflated by the existence of correlation among the predictor variables in the model.

$$VIF_j = \frac{1}{1-R_j^{2}}$$

where $R_j^2$ the $R^2$ value obtained by regressing the $j^{th}$ predictor on the remaining predictors. 

A VIF of 1 $\rightarrow$ there is no correlation among the $j^{th}$ predictor and the remaining predictor variables, and hence the variance of $\beta_j$ is not inflated at all.\
Generally, VIF \> 4 $\rightarrow$ further investigation,\
VIF $\geq$ 10 $\rightarrow$ serious multi-collinearity.

```{r}

vif(lm(Wind ~ Temp+Solar.R, data=airquality))
vif(lm(Temp ~ Wind+Solar.R, data=airquality))
vif(lm(Solar.R ~ Wind+Temp, data=airquality))
```

<br> <br>

### Bias-Variance Trade-off

\- Bias: it is the error from incorrect assumptions in the learning algorithm - under-fitting.

\- Variance: it is the error from sensitivity to small fluctuations in the training set - over-fitting.

[**Over-fitting**]{.ul} and [**under-fitting**]{.ul} are two extreme ends of fitting a model to the data observations.

-   Under-fitting: when model doesn't capture the underlying trend from the data points. e.g.: trying to model a non-linear relationship with a linear model. This scenario is the case of high bias and low variance.
-   Over-fitting: when model tries to fit all the data points, that it starts capturing the noise. e.g.: trying to model a quadratic relationship with a 5th order polynomial. This is the case of low bias and high variance.

Frequently used techniques to reduce under-fitting:

-   Increase model complexity
-   Increase number of features, perform feature engineering
-   Remove noise from the data
-   Increase the number of epochs or increase the duration of training to get better results.

Frequently used techniques to reduce over-fitting:

-   Reduce model complexity
-   Early stopping during the training phase (keep an eye over the loss over the training period as soon as loss begins to increase stop training).
-   Ridge and Lasso Regression - penalize for additional complexity

<br>

### Regularization - Ridge and Lasso Regression

Regularization fits a model containing all $p$ predictors using a technique that constrains or regularizes the coefficient estimates, or shrinks the coefficient estimates to zero. The tuning/penalty parameter is denoted by $\lambda$.

For every model, the mean squared error in estimated target is the sum of **Variance**, **Bias** and in-built **random error in estimation**, $\epsilon$.

To [minimize]{.ul} the expected error, a model with **low variance** and **low bias** is ***ideal***.

-   $\lambda = 0$ : least square model with all $p$ predictors

-   $\lambda = \infty$ : null model with all no predictors

-   as $\lambda$ increases, variance decreases and bias increases

<br>

[**Ridge Regression/L2 Regularization**]{.ul} coefficients are values that minimize

$\sum_{i=1}^n (y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij})^2 + \lambda \sum_{j=1}^p \beta_j^2$

where $\lambda \geq 0$ is the penalty term which controls the relative impact of the two terms on the coefficient estimates.

-   includes all $p$ predictors and although increasing $\lambda$ reduces the magnitudes of the coefficient estimates, but it doesn't exclude them (unless $\lambda = \infty$)

-   as $\lambda$ increases, $L_2$ $norm$ of $\beta$ coefficients decrease

-   works best in situations where least squares estimates have **high variance**

-   best to apply it after standardizing the predictors (so that they are on the same scale)

<br>

[**Lasso Regression/L1 Regularization**]{.ul} coefficients are values that minimize

$\sum_{i=1}^n (y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij})^2 + \lambda \sum_{j=1}^p |\beta_j|$

where $\lambda \geq 0$ is the penalty term which controls the relative impact of the two terms on the coefficient estimates.

-   shrinks the coefficients towards 0 as $\lambda$ increases

-   as $\lambda$ increases, $L_1$ $norm$ of $\beta$ coefficients decrease

-   since the coefficients are reduced to 0, Lasso regression can perform variable selection, which yields sparse models

<br>

[**Example**]{.ul}: Both ridge and lasso regression involves tuning the hyperparameter, $\lambda$. `glmnet()` runs the model many times with a different value of $\lambda$. `cv.glmnet()` uses cross-validation to check how well each model generalizes (visualized in plots below).

-   $\alpha = 0$ for ridge regression

```{r}
# ridge regression

lambda_seq = 10^seq(5, -2, by = -.1)

cv_output_ridge = cv.glmnet(x, y, alpha = 0, lambda = lambda_seq, nfolds = 5)

plot(cv_output_ridge)
```

The lowest point in the plot indicates the optimal $\lambda$.

```{r}
ridge_lambda = cv_output_ridge$lambda.min
ridge_lambda
```

```{r}
# extract the coefficients associated with the value of lambda that gives the minimum cross-validation model

coef(cv_output_ridge, s = 'lambda.min')
```

-   $\alpha = 1$ for lasso regression

```{r}
# lasso regression

lambda_seq = 10^seq(5, -2, by = -.1)

cv_output_lasso = cv.glmnet(x, y, alpha = 1, lambda = lambda_seq, nfolds = 5)

plot(cv_output_lasso)
```

The lowest point in the plot indicates the optimal $\lambda$.

```{r}
lasso_lambda = cv_output_lasso$lambda.min
lasso_lambda
```

```{r}
# extract the coefficients associated with the value of lambda that gives the minimum cross-validation model

coef(cv_output_lasso, s = 'lambda.min')
```

The (.)s represent zeros. In this case the coefficient for the predictor, 'Month' is reduced to 0.

<br> <br> <br>

## Polynomial and Spline Regression {data-link="Polynomial and Spline Regression"}

<br> <br>

**Polynomial Regression** fits a polynomial relationship between the dependent variable $y$ and the independent variable $x$ . Such a model with a single predictor is

$y_{i} = \beta _{0} + \beta _{1} x + \beta _{2} x^2 +\cdots +\beta _{h} x^h + \epsilon_{i},$

where $h$ is called the degree of the polynomial, i.e., $h=2$ is quadratic, $h=3$ is cubic, $h=4$ is quartic, etc. Although there is a non-linear relationship between $y$ and $x$, polynomial regression is still considered linear regression since it is linear in the regression coefficients, i.e., the $\beta$s.

Caution: Polynomials are powerful tools but they can lead to over-fitting. **Over-fitting** happens when the model is trying to fit every single data point. While doing that, it will pick up noise instead of the actual data, even though the model is getting better and better at fitting the existing data (increase in the R-squared value).

```{r}
# select predictor attribute
x =  airquality[, 'Temp']
                
# select target attribute
y = airquality[, 'Ozone'] 

poly_model = lm(y~poly(x, 3))

summary(poly_model)
```

Note: $R^2$ values have increased ($R^2 = 0.4168$, adjusted $R^2 = 0.413$ for the simple linear regression case).

```{r}
plot(y~x, xlab = 'Temp', ylab = 'Ozone')
lines(sort(x), fitted(poly_model)[order(x)], col = 'purple', lwd = 2)
abline(simple_model, col = 'blue', lwd = 2)
legend(55, 150, legend=c('simple reg', 'polynomial reg'),
       col=c('blue', 'purple'), lty = 1:1, box.lty=0)
```

<br> <br>

**Spline Regression** is one method for testing non-linearity in the predictor variables and for modeling non-linear functions. A spline is a function constructed piece-wise from polynomial functions.

Instead of building one model for the full dataset, spline regression divides the data into multiple bins and fits each bin with separate model. The points where the division occurs is called *knots*.

```{r}
# select predictor attribute
x =  airquality[, 'Temp']
                
# select target attribute
y = airquality[, 'Ozone'] 

spline_model = lm(y~ bs(x, knots = c(65, 75, 85)))

summary(spline_model)
```

```{r}
plot(y~x, xlab = 'Temp', ylab = 'Ozone')
lines(sort(x), fitted(spline_model)[order(x)], col = 'purple', lwd = 2)
abline(simple_model, col = 'blue', lwd = 2)
legend(55, 150, legend=c('simple reg', 'spline reg'),
       col=c('blue', 'purple'), lty = 1:1, box.lty=0)
```

<br> <br> <br>

#### [**Regression Limitations**]{.ul}

-   Sensitive to outliers
-   Assumptions of linearity, independent predictors etc. (see assumptions section)

## References

1.  [Penn State STAT 462 Applied Regression Analysis](https://online.stat.psu.edu/stat462/node/77/)
2.  [Khan Academy](https://www.khanacademy.org/math/ap-statistics/bivariate-data-ap/assessing-fit-least-squares-regression/v/influential-points-regression)
3.  [Cornell Machine Learning Course](https://www.cs.cornell.edu/courses/cs4780/2018fa/)
4.  [CMU Data Analytics Course](https://www.stat.cmu.edu/~cshalizi/mreg/15/lectures/06/lecture-06.pdf)
5.  [Measure of Influence](https://cran.r-project.org/web/packages/olsrr/vignettes/influence_measures.html#:~:text=Cook's%20D%20Bar%20Plot,-Bar%20Plot%20of&text=Cook's%20distance%20was%20introduced%20by,y%20value%20of%20the%20observation.)
6.  [UCLA Course - Coding Systems for Categorical Variables](https://stats.idre.ucla.edu/spss/faq/coding-systems-for-categorical-variables-in-regression-analysis-2/#SIMPLE%20EFFECT%20CODING)
7.  [Princeton Spline Regression Example](https://data.princeton.edu/eco572/smoothing2)
8.  [UT Dallas Spline Regression Example](https://personal.utdallas.edu/~Andrew.Wheeler/Splines.html)
9.  [Ridge Regression Example in R](https://drsimonj.svbtle.com/ridge-regression-with-glmnet)
10. [Lasso Regression Example in R](https://rstatisticsblog.com/data-science-in-action/machine-learning/lasso-regression/)