notebooks/Learning Units/Linear Regression/Linear Regression - Chapter 5 - Ridge regularization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ridge regularization\n",
    "\n",
    "Ridge regularization is an alternative to LASSO. It minimizes the L2-Norm instead of the L1-Norm, which means that the weights are maintained under control but not necessarily to zero. This offers a smoother alternative to LASSO, as the modelled function might still need to be high-dimensional in certain areas.\n",
    "\n",
    "Ridge regularization and its gradient are defined as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Ridge:\n",
    " \n",
    "    def __init__(self, _lambda):\n",
    "        self._lambda = _lambda\n",
    " \n",
    "    def __call__(self, theta):\n",
    " \n",
    "        return self._lambda * 0.5 * np.sum(theta**2)\n",
    " \n",
    "    def gradient(self, theta):\n",
    " \n",
    "        return self._lambda * theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The regularization is controlled by a factor typically referred to as Lambda. If Lambda equals 0, there is no regularization, otherwise, the larger the value of Lambda, the more the regularization term will impact the loss function.\n",
    "\n",
    "![](https://skratch.valentincalomme.com/wp-content/uploads/2018/09/ridge.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it can be seen on the animation above. Ridge regularization is “smoother”, as it maintains some of the complexity of the function, whilst tuning it down."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}