Regression analysis is a cornerstone of statistical methods used for predicting a continuous outcome variable based on one or more predictor variables. It offers insights into the relationships between variables, enabling both prediction and inference. This document covers key aspects of regression analysis, including various types of regression, methods for estimation, residual analysis, and more.
Linear regression models the linear relationship between a dependent variable and independent variables X.
OLS is the most prevalent method for estimating the parameters of a linear regression model, aiming to minimize the sum of squared differences between observed and predicted values.
- WLS addresses heteroscedasticity by assigning weights based on the inverse variance of errors.
- GLS extends WLS to accommodate correlated error terms.
These methods introduce penalties to the regression model to prevent overfitting:
- Lasso (L1 Regularization): Adds the absolute value of coefficients as a penalty to the loss function.
- Ridge (L2 Regularization): Incorporates the squared magnitude of coefficients into the penalty.
Examining residuals is important for validating regression model assumptions, assessing for patterns that suggest issues like non-linearity or heteroscedasticity.
- Heteroscedasticity: Variance of error terms is not constant.
- Homoscedasticity: Desired condition where variance is constant.
- Breusch-Pagan Test: Assesses heteroscedasticity by regressing squared residuals on independent variables.
- White Test: A general test for heteroscedasticity without assuming a specific form.
- Skewness: Measures the asymmetry of the residuals distribution.
- Kurtosis: Indicates the "tailedness" of the residuals distribution.
Adding polynomial features can help model non-linear relationships by including powers or interactions of original variables.
Gradient Descent is a first-order iterative optimization algorithm used to find the minimum of a function. By moving in the negative direction of the function's gradient, it aims to find the parameter values that minimize the function.
Batch Gradient Descent: Computes the gradient using the whole dataset. This is computationally expensive for large datasets. Stochastic Gradient Descent (SGD): Updates the parameters for each training example. It's faster but more fluctuant. Mini-batch Gradient Descent: Strikes a balance between batch and stochastic gradient descent by updating parameters using a subset of the data.
Also known as the Newton-Raphson method, it is a root-finding algorithm that uses function derivatives to find the roots of a real-valued function. For optimization, it's used to find where the derivative (or gradient) is zero. It converges faster than gradient descent but is more computationally expensive due to the calculation of second-order derivatives.
This method is primarily used for solving systems of linear equations, especially those arising from problems with a large number of variables. It's particularly useful for optimizing quadratic functions and is more efficient than gradient descent for certain types of problems.
These methods are used to find local maxima and minima of functions. They are similar to Newton's method but do not require the computation of second-order derivatives, which makes them more suitable for large-scale problems. Examples include the BFGS and L-BFGS algorithms.
Cook's distance is a measure used in regression analysis to identify observations that are potentially influential to a regression model's predictions. It quantifies the influence of each data point on the fitted regression coefficients and helps in detecting outliers and influential points that might affect the regression model disproportionately.