10. Regularization: Simplicity

Regularization: Simplicity

Topic: Regularization: Simplicity

Course: GMLC

Date: 24 February 2019

Professor: Not specified

Regularisation
- A principle in which we penalize model for being too complex
- Instead of aiming to minimise loss minimise(Loss(Data|Model))) we use a process called structural risk minimisation which includes loss + complexity minimise(Loss(Data|Model) + complexity(Model))
- In structural risk minimization we focus on minimizing the loss as well as keeping track of model complexity
L2 Regularisation
- quantifying complexity by summing squares of all feature weights (If we make the model too complex by adding a large weight to a single feature, complexity will go up largely)
- Encourages weights towards 0 but not exactly 0
Lambda
- Regularisation rate
- minimise(Loss(Model|Data) + λ complexity(Model))
- Increasing lambda increases the regularisation effect, thus increasing weight values towards 0 even more, danger of underfitting the data
- Lowering lambda yields a flatter histogram (Less dramatic effects & drops), danger of overfitting the data
- The goal is to choose the right value between model simplicity & data fit
Early stopping
- Way of preventing overfitting by stopping the model training before it converges

Regularisation is a principle which focuses on penalizing the model for being too complex, using L2 regularisation, complexity is quantified and largely increases if a single feature has a large weight.