![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Principal Component Analysis (PCA) from Scratch"
+ ],
+ "metadata": {
+ "id": "K1f0SOAdpktT"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Background\n",
+ "Principal Component Analysis (PCA) is a simple dimensionality reduction technique that can capture linear correlations between the features. For a given (standardized) data, PCA can be calculated by eigenvalue decomposition of covariance (or correlation) matrix of the data, or Singular Value Decomposition (SVD) of the data matrix. The data standardization includes mean removal and variance normalization."
+ ],
+ "metadata": {
+ "id": "t1sIZA98rv7U"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from sklearn.datasets import load_iris\n",
+ "iris = load_iris()\n",
+ "X = iris['data']\n",
+ "y = iris['target']\n",
+ "n_samples, n_features = X.shape\n",
+ "\n",
+ "print('Number of samples:', n_samples)\n",
+ "print('Number of features:', n_features)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "biO6LE9Nr9zm",
+ "outputId": "b13245f6-3c94-42b3-bc10-598ddc31d884"
+ },
+ "execution_count": 136,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Number of samples: 150\n",
+ "Number of features: 4\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import numpy as np\n",
+ "import matplotlib.pyplot as plt\n",
+ "%matplotlib inline"
+ ],
+ "metadata": {
+ "id": "CcPCtb_vsP_X"
+ },
+ "execution_count": 13,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "fig, ax = plt.subplots(nrows=n_features, ncols=n_features, figsize= (8, 8))\n",
+ "fig.tight_layout()\n",
+ "\n",
+ "\"\"\"\n",
+ "This loop creates pairwise scatter plots using only the upper triangle of a subplot grid.\n",
+ "\n",
+ "np.triu_indices_from(ax, k=1) gives the (i, j) index pairs where i < j (upper triangle),\n",
+ "to avoid redundant scatter plots and keep the grid clean.\n",
+ "\n",
+ "zip(*...) unpacks the tuple of arrays into separate arguments, pairing corresponding i and j.\n",
+ "\n",
+ "For each (i, j):\n",
+ "- ax[j, i] is used to plot feature j vs feature i.\n",
+ "- ax[i, j] is turned off to avoid mirroring the same plot.\n",
+ "\"\"\"\n",
+ "\n",
+ "names = iris.feature_names\n",
+ "\n",
+ "for i, j in zip(*np.triu_indices_from(ax, k=1)):\n",
+ " ax[j, i].scatter(X[:, j], X[:, i], c = y)\n",
+ " ax[j, i].set_xlabel(names[j])\n",
+ " ax[j, i].set_ylabel(names[i])\n",
+ " ax[i, j].set_axis_off()\n",
+ "\n",
+ "for i in range(n_features):\n",
+ " ax[i, i].hist(X[:, i], color = 'lightblue')\n",
+ " ax[i, i].set_ylabel('Count')\n",
+ " ax[i, i].set_xlabel(names[i])"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 821
+ },
+ "id": "Zh8fdBAvsrWY",
+ "outputId": "c58975f7-7694-4671-b873-da36f8269287"
+ },
+ "execution_count": 23,
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Image compression ~ SVD"
+ ],
+ "metadata": {
+ "id": "7NKOiPVgDQWc"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ZvT3xFwHhCs0",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "cd4bca06-251e-4fd9-a594-3e040f1fb99f"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "/bin/bash: line 1: s: command not found\n",
+ "--2025-06-18 13:21:50-- https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcROcm2N1RrcetnuG9WhfdH0zpojmmKUT8GSbA\n",
+ "Resolving encrypted-tbn0.gstatic.com (encrypted-tbn0.gstatic.com)... 74.125.196.113, 74.125.196.100, 74.125.196.138, ...\n",
+ "Connecting to encrypted-tbn0.gstatic.com (encrypted-tbn0.gstatic.com)|74.125.196.113|:443... connected.\n",
+ "HTTP request sent, awaiting response... 200 OK\n",
+ "Length: 4573 (4.5K) [image/jpeg]\n",
+ "Saving to: ‘images?q=tbn:ANd9GcROcm2N1RrcetnuG9WhfdH0zpojmmKUT8GSbA’\n",
+ "\n",
+ "\r images?q= 0%[ ] 0 --.-KB/s \rimages?q=tbn:ANd9Gc 100%[===================>] 4.47K --.-KB/s in 0s \n",
+ "\n",
+ "2025-06-18 13:21:50 (45.4 MB/s) - ‘images?q=tbn:ANd9GcROcm2N1RrcetnuG9WhfdH0zpojmmKUT8GSbA’ saved [4573/4573]\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "! wget https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcROcm2N1RrcetnuG9WhfdH0zpojmmKUT8GSbA&s"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from PIL import Image\n",
+ "import numpy as np\n",
+ "import matplotlib.pyplot as plt"
+ ],
+ "metadata": {
+ "id": "KdSGfAKBD8yE"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "img = Image.open(\"/content/mcfk\")\n",
+ "imggray = img.convert('LA')\n",
+ "_ = plt.imshow(imggray)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 395
+ },
+ "id": "JnxOfEAKETbS",
+ "outputId": "a9ae134c-1052-49f5-e7bf-a5af610fa8ce"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### RF = bagging + Row level sampling + column level sampling\n",
+ "\n",
+ "---\n"
+ ],
+ "metadata": {
+ "id": "JMGDmDrtjHVD"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 80
+ },
+ "id": "taP8bOWti-rM",
+ "outputId": "19ba63a2-48d2-48ba-d75d-de7a8c32635a"
+ },
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "RandomForestClassifier(max_depth=2, random_state=0)"
+ ],
+ "text/html": [
+ "RandomForestClassifier(max_depth=2, random_state=0)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
RandomForestClassifier(max_depth=2, random_state=0)
ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])<sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>
SimpleImputer(add_indicator=True)
<sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>
OrdinalEncoder(encoded_missing_value=-2, handle_unknown='use_encoded_value',\n", + " unknown_value=-1)
ColumnTransformer(transformers=[('pipeline',\n",
+ " Pipeline(steps=[('standardscaler',\n",
+ " StandardScaler()),\n",
+ " ('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True))]),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('onehotencoder',\n",
+ " OneHotEncoder(handle_unknown='ignore'),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. ColumnTransformer(transformers=[('pipeline',\n",
+ " Pipeline(steps=[('standardscaler',\n",
+ " StandardScaler()),\n",
+ " ('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True))]),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('onehotencoder',\n",
+ " OneHotEncoder(handle_unknown='ignore'),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])<sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>
StandardScaler()
SimpleImputer(add_indicator=True)
<sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>
OneHotEncoder(handle_unknown='ignore')
Pipeline(steps=[('columntransformer',\n",
+ " ColumnTransformer(transformers=[('pipeline',\n",
+ " Pipeline(steps=[('standardscaler',\n",
+ " StandardScaler()),\n",
+ " ('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True))]),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('onehotencoder',\n",
+ " OneHotEncoder(handle_unknown='ignore'),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])),\n",
+ " ('lassocv', LassoCV())])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. Pipeline(steps=[('columntransformer',\n",
+ " ColumnTransformer(transformers=[('pipeline',\n",
+ " Pipeline(steps=[('standardscaler',\n",
+ " StandardScaler()),\n",
+ " ('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True))]),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('onehotencoder',\n",
+ " OneHotEncoder(handle_unknown='ignore'),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])),\n",
+ " ('lassocv', LassoCV())])ColumnTransformer(transformers=[('pipeline',\n",
+ " Pipeline(steps=[('standardscaler',\n",
+ " StandardScaler()),\n",
+ " ('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True))]),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('onehotencoder',\n",
+ " OneHotEncoder(handle_unknown='ignore'),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])<sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>
StandardScaler()
SimpleImputer(add_indicator=True)
<sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>
OneHotEncoder(handle_unknown='ignore')
LassoCV()
Pipeline(steps=[('columntransformer',\n",
+ " ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])),\n",
+ " ('randomforestregressor',\n",
+ " RandomForestRegressor(random_state=42))])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. Pipeline(steps=[('columntransformer',\n",
+ " ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])),\n",
+ " ('randomforestregressor',\n",
+ " RandomForestRegressor(random_state=42))])ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])<sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>
SimpleImputer(add_indicator=True)
<sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>
OrdinalEncoder(encoded_missing_value=-2, handle_unknown='use_encoded_value',\n", + " unknown_value=-1)
RandomForestRegressor(random_state=42)
Pipeline(steps=[('columntransformer',\n",
+ " ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])),\n",
+ " ('histgradientboostingregressor',\n",
+ " HistGradientBoostingRegressor(random_state=0))])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. Pipeline(steps=[('columntransformer',\n",
+ " ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])),\n",
+ " ('histgradientboostingregressor',\n",
+ " HistGradientBoostingRegressor(random_state=0))])ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])<sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>
SimpleImputer(add_indicator=True)
<sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>
OrdinalEncoder(encoded_missing_value=-2, handle_unknown='use_encoded_value',\n", + " unknown_value=-1)
HistGradientBoostingRegressor(random_state=0)
StackingRegressor(estimators=[('Random Forest',\n",
+ " Pipeline(steps=[('columntransformer',\n",
+ " ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_v...\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])),\n",
+ " ('histgradientboostingregressor',\n",
+ " HistGradientBoostingRegressor(random_state=0))]))],\n",
+ " final_estimator=RidgeCV())In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. StackingRegressor(estimators=[('Random Forest',\n",
+ " Pipeline(steps=[('columntransformer',\n",
+ " ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_v...\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])),\n",
+ " ('histgradientboostingregressor',\n",
+ " HistGradientBoostingRegressor(random_state=0))]))],\n",
+ " final_estimator=RidgeCV())ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])<sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>
SimpleImputer(add_indicator=True)
<sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>
OrdinalEncoder(encoded_missing_value=-2, handle_unknown='use_encoded_value',\n", + " unknown_value=-1)
RandomForestRegressor(random_state=42)
ColumnTransformer(transformers=[('pipeline',\n",
+ " Pipeline(steps=[('standardscaler',\n",
+ " StandardScaler()),\n",
+ " ('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True))]),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('onehotencoder',\n",
+ " OneHotEncoder(handle_unknown='ignore'),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])<sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>
StandardScaler()
SimpleImputer(add_indicator=True)
<sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>
OneHotEncoder(handle_unknown='ignore')
LassoCV()
ColumnTransformer(transformers=[('simpleimputer',\n",
+ " SimpleImputer(add_indicator=True),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>),\n",
+ " ('ordinalencoder',\n",
+ " OrdinalEncoder(encoded_missing_value=-2,\n",
+ " handle_unknown='use_encoded_value',\n",
+ " unknown_value=-1),\n",
+ " <sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>)])<sklearn.compose._column_transformer.make_column_selector object at 0x7dda18ac1b20>
SimpleImputer(add_indicator=True)
<sklearn.compose._column_transformer.make_column_selector object at 0x7dda11bbf950>
OrdinalEncoder(encoded_missing_value=-2, handle_unknown='use_encoded_value',\n", + " unknown_value=-1)
HistGradientBoostingRegressor(random_state=0)
RidgeCV()
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 1. Generate a simple binary classification dataset"
+ ],
+ "metadata": {
+ "id": "tbhQs32SYTOP"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from sklearn.datasets import make_classification\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "# Create dataset with 2 classes\n",
+ "X, y = make_classification(\n",
+ " n_samples=200, # number of points\n",
+ " n_features=2, # only 2 features for easy visualization\n",
+ " n_redundant=0,\n",
+ " n_informative=2,\n",
+ " n_clusters_per_class=1, # controlling the non linearity for classes\n",
+ " weights=[0.9, 0.1], # imbalanced dataset (90% class 0, 10% class 1)\n",
+ " random_state=42\n",
+ ")"
+ ],
+ "metadata": {
+ "id": "K6Kvda_NXYmN"
+ },
+ "execution_count": 17,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "weights=[0.9,0.1] → makes class imbalance (real-world like phishing vs legitimate).\n",
+ "\n",
+ "You’ll see class 0 (blue) has more points than class 1 (red)."
+ ],
+ "metadata": {
+ "id": "VpVyJvz8YFUD"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Plot\n",
+ "plt.scatter(X[:,0], X[:,1], c=y, cmap='coolwarm', edgecolors='k')\n",
+ "plt.xlabel(\"Feature 1\")\n",
+ "plt.ylabel(\"Feature 2\")\n",
+ "plt.title(\"Synthetic Dataset\")\n",
+ "plt.show()"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 472
+ },
+ "id": "Gl6DXaYzu3IR",
+ "outputId": "e7756126-98a3-418f-c275-25d57f379a0a"
+ },
+ "execution_count": 13,
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## ***Implementing Adaline in Python***"
+ ],
+ "metadata": {
+ "id": "xumKeRGuD_PN"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "oD5k1e8YDto4"
+ },
+ "outputs": [],
+ "source": [
+ "class AdalineGD:\n",
+ " \"\"\"ADAptive LInear NEuron classifier.\n",
+ "\n",
+ " Parameters\n",
+ " ------------\n",
+ " eta : float\n",
+ " Learning rate (between 0.0 and 1.0)\n",
+ " n_iter : int\n",
+ " Passes over the training dataset.\n",
+ " random_state : int\n",
+ " Random number generator seed for random weight\n",
+ " initialization.\n",
+ "\n",
+ "\n",
+ " Attributes\n",
+ " -----------\n",
+ " w_ : 1d-array\n",
+ " Weights after fitting.\n",
+ " b_ : Scalar\n",
+ " Bias unit after fitting.\n",
+ " losses_ : list\n",
+ " Mean squared eror loss function values in each epoch.\n",
+ " \"\"\"\n",
+ " def __init__(self, eta=0.01, n_iter=50, random_state=1):\n",
+ " self.eta = eta\n",
+ " self.n_iter = n_iter\n",
+ " self.random_state = random_state\n",
+ "\n",
+ " def fit(self, X,y):\n",
+ " \"\"\" Fit training data.\n",
+ "\n",
+ " Parameters\n",
+ " ----------\n",
+ " X : {array-like}, shape = [n_examples, n_features]\n",
+ " Training vectors, where n_examples is the number of examples and\n",
+ " n_features is the number of features.\n",
+ " y : array-like, shape = [n_examples]\n",
+ " Target values.\n",
+ "\n",
+ " Returns\n",
+ " -------\n",
+ " self : object\n",
+ " \"\"\"\n",
+ " rgen = np.random.RandomState(self.random_state)\n",
+ " self.w_ = rgen.normal(loc=0.0, scale=0.01, size=X.shape[1]+1)\n",
+ " self.losses_ = []\n",
+ "\n",
+ " for i in range(self.n_iter):\n",
+ " net_input = self.net_input(X)\n",
+ " output = self.activation(net_input)\n",
+ " errors = y-output\n",
+ " self.w_[0] += self.eta*2.0*errors.mean()\n",
+ " self.w_[1:] += self.eta*2.0*X.T.dot(errors)/X.shape[0]\n",
+ " loss = (errors**2).mean() # Mean Squared Error\n",
+ " self.losses_.append(loss)\n",
+ "\n",
+ " return self\n",
+ "\n",
+ "\n",
+ " def net_input(self, X):\n",
+ " \"\"\"Calculate net input\"\"\"\n",
+ " return np.dot(X, self.w_[1:]) + self.w_[0]\n",
+ "\n",
+ " def activation(self, X):\n",
+ " \"Compute linear activation sigma(z) = z\"\n",
+ " return X\n",
+ "\n",
+ " def predict(self, X):\n",
+ " \"\"\"Return class label after unit step\"\"\"\n",
+ " return np.where(self.activation(self.net_input(X)) >= 0.5, 1, 0)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import matplotlib.pyplot as plt"
+ ],
+ "metadata": {
+ "id": "A2V17lTrE7Ms"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "s = \"https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data\"\n",
+ "\n",
+ "df = pd.read_csv(s, header=None, encoding='utf-8')\n",
+ "df.columns = ['sepal length in cm','sepal width in cm','petal length in cm','petal width in cm','class']\n",
+ "df.head()"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 206
+ },
+ "id": "hWB0mJksEICL",
+ "outputId": "f5f63715-5784-4e13-dfb3-57e565796e18"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " sepal length in cm sepal width in cm petal length in cm \\\n",
+ "0 5.1 3.5 1.4 \n",
+ "1 4.9 3.0 1.4 \n",
+ "2 4.7 3.2 1.3 \n",
+ "3 4.6 3.1 1.5 \n",
+ "4 5.0 3.6 1.4 \n",
+ "\n",
+ " petal width in cm class \n",
+ "0 0.2 Iris-setosa \n",
+ "1 0.2 Iris-setosa \n",
+ "2 0.2 Iris-setosa \n",
+ "3 0.2 Iris-setosa \n",
+ "4 0.2 Iris-setosa "
+ ],
+ "text/html": [
+ "\n",
+ " | \n", + " | sepal length in cm | \n", + "sepal width in cm | \n", + "petal length in cm | \n", + "petal width in cm | \n", + "class | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "5.1 | \n", + "3.5 | \n", + "1.4 | \n", + "0.2 | \n", + "Iris-setosa | \n", + "
| 1 | \n", + "4.9 | \n", + "3.0 | \n", + "1.4 | \n", + "0.2 | \n", + "Iris-setosa | \n", + "
| 2 | \n", + "4.7 | \n", + "3.2 | \n", + "1.3 | \n", + "0.2 | \n", + "Iris-setosa | \n", + "
| 3 | \n", + "4.6 | \n", + "3.1 | \n", + "1.5 | \n", + "0.2 | \n", + "Iris-setosa | \n", + "
| 4 | \n", + "5.0 | \n", + "3.6 | \n", + "1.4 | \n", + "0.2 | \n", + "Iris-setosa | \n", + "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## ***Perceptron from scratch***\n",
+ "*It is based on MCP Neuron model which automatically learns the optimal weight coeffs. that would be multiplied with the i/p features in order to make the decision weather a neuron fires(transmits a signal) or not. In context of classification, such an algorithm predicts weather a new data point belongs to one class or the other.*\n"
+ ],
+ "metadata": {
+ "id": "xSW16FJMxo0e"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import numpy as np\n",
+ "\n",
+ "class Perceptron:\n",
+ " \"\"\"Perceptron classifier.\n",
+ "\n",
+ " Parameters\n",
+ " ------------\n",
+ " eta : float\n",
+ " Learning rate (between 0.0 and 1.0)\n",
+ " n_iter : int\n",
+ " Passes over the training dataset.\n",
+ " random_state : int\n",
+ " Random number generator seed for random weight initialization.\n",
+ "\n",
+ " Attributes\n",
+ " -----------\n",
+ " w_ : 1d-array\n",
+ " Weights after fitting.\n",
+ " b_ : Scalar\n",
+ " Bias unit after fitting.\n",
+ " errors_ : list\n",
+ " Number of misclassifications (updates) in each epoch.\n",
+ " \"\"\"\n",
+ " def __init__(self, eta=0.01, n_iter=50, random_state=1):\n",
+ " self.eta = eta\n",
+ " self.n_iter = n_iter\n",
+ " self.random_state = random_state\n",
+ "\n",
+ " def fit(self, X, y):\n",
+ " \"\"\"Fit training data.\n",
+ "\n",
+ " Parameters\n",
+ " ----------\n",
+ " X : {array-like}, shape = [n_examples, n_features], X ∈ R^(n*d)\n",
+ " Training vectors, where n_examples is the number of examples and\n",
+ " n_features is the number of features.\n",
+ " y : array-like, shape = [n_examples]\n",
+ " Target values.\n",
+ "\n",
+ " Returns\n",
+ " -------\n",
+ " self : object\n",
+ " \"\"\"\n",
+ "\n",
+ " #RandomState exposes a number of methods for generating random numbers drawn from a variety of probability distributions.\n",
+ " rgen = np.random.RandomState(seed=self.random_state)\n",
+ "\n",
+ " # here we are initializing the weight vector with mean=0, std_dev=0.01\n",
+ " self.w_ = rgen.normal(loc=0.0, scale=0.01, size=X.shape[1]+1)\n",
+ " # initializing bias = 0\n",
+ " # self.b_ = np.float_(0.)\n",
+ " self.errors_ = []\n",
+ "\n",
+ " for i in range(self.n_iter):\n",
+ " errors = 0\n",
+ " for x_i, target in zip(X,y):\n",
+ " update = self.eta * (target - self.predict(x_i))\n",
+ " self.w_[0] += update\n",
+ " self.w_[1:] += update*x_i\n",
+ " errors += int(update!=0.0)\n",
+ " self.errors_.append(errors)\n",
+ " return self\n",
+ "\n",
+ " def net_input(self, X):\n",
+ " '''Calculate the net input'''\n",
+ " return np.dot(X,self.w_[1:]) + self.w_[0]\n",
+ "\n",
+ " def predict(self, X):\n",
+ " '''Return class label after unit step'''\n",
+ " return np.where(self.net_input(X)>=0.0, 1, 0)"
+ ],
+ "metadata": {
+ "id": "y0sNQyxVFR3X"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Applying Perceptron to iris dataset"
+ ],
+ "metadata": {
+ "id": "paPKoX0SIDHE"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import pandas as pd\n",
+ "import matplotlib.pyplot as plt"
+ ],
+ "metadata": {
+ "id": "CyqTMKjUJQDS"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "s = \"https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data\"\n",
+ "\n",
+ "df = pd.read_csv(s, header=None, encoding='utf-8')\n",
+ "df.columns = ['sepal length in cm','sepal width in cm','petal length in cm','petal width in cm','class']\n",
+ "df.head()"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 206
+ },
+ "id": "TQHD4mlwJQ24",
+ "outputId": "edee7b53-f03e-496c-b8aa-2de04bbe6ae4"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " sepal length in cm sepal width in cm petal length in cm \\\n",
+ "0 5.1 3.5 1.4 \n",
+ "1 4.9 3.0 1.4 \n",
+ "2 4.7 3.2 1.3 \n",
+ "3 4.6 3.1 1.5 \n",
+ "4 5.0 3.6 1.4 \n",
+ "\n",
+ " petal width in cm class \n",
+ "0 0.2 Iris-setosa \n",
+ "1 0.2 Iris-setosa \n",
+ "2 0.2 Iris-setosa \n",
+ "3 0.2 Iris-setosa \n",
+ "4 0.2 Iris-setosa "
+ ],
+ "text/html": [
+ "\n",
+ " | \n", + " | sepal length in cm | \n", + "sepal width in cm | \n", + "petal length in cm | \n", + "petal width in cm | \n", + "class | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "5.1 | \n", + "3.5 | \n", + "1.4 | \n", + "0.2 | \n", + "Iris-setosa | \n", + "
| 1 | \n", + "4.9 | \n", + "3.0 | \n", + "1.4 | \n", + "0.2 | \n", + "Iris-setosa | \n", + "
| 2 | \n", + "4.7 | \n", + "3.2 | \n", + "1.3 | \n", + "0.2 | \n", + "Iris-setosa | \n", + "
| 3 | \n", + "4.6 | \n", + "3.1 | \n", + "1.5 | \n", + "0.2 | \n", + "Iris-setosa | \n", + "
| 4 | \n", + "5.0 | \n", + "3.6 | \n", + "1.4 | \n", + "0.2 | \n", + "Iris-setosa | \n", + "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Watch the Gradient Descent Without any external tools"
+ ],
+ "metadata": {
+ "id": "3iUk6RHjehYG"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Gradient Descent Parameters\n",
+ "alpha = 0.00001 # learning rate (hyperparameter)\n",
+ "epochs = 1000 # number of iterations\n",
+ "m = len(X) # number of data points\n",
+ "\n",
+ "# Initialize parameters (theta_0 = intercept, theta_1 = slope)\n",
+ "theta_0 = 0\n",
+ "theta_1 = 0\n",
+ "\n",
+ "# Training loop\n",
+ "for epoch in range(epochs):\n",
+ " y_pred = [theta_0 + theta_1 * x_i for x_i in X] # predictions\n",
+ " error = [y_pred[i] - y[i] for i in range(m)] # prediction error\n",
+ "\n",
+ " # Compute gradients\n",
+ " d_theta_0 = (1/m) * sum(error)\n",
+ " d_theta_1 = (1/m) * sum([error[i] * X[i] for i in range(m)])\n",
+ "\n",
+ " # Update parameters\n",
+ " theta_0 -= alpha * d_theta_0\n",
+ " theta_1 -= alpha * d_theta_1\n",
+ "\n",
+ " # Optional: Print cost every 100 iterations\n",
+ " if epoch % 100 == 0:\n",
+ " cost = (1/(2*m)) * sum([e**2 for e in error])\n",
+ " print(f\"Epoch {epoch}: Cost={cost:.4f}, Intercept={theta_0:.4f}, Slope={theta_1:.4f}\")\n"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "wL7VyU-cfd5d",
+ "outputId": "597f9971-564d-41b3-b67f-bef546c34ab9"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Epoch 0: Cost=55000.0000, Intercept=0.0030, Slope=0.1100\n",
+ "Epoch 100: Cost=6010.3914, Intercept=0.1834, Slope=6.7256\n",
+ "Epoch 200: Cost=656.8200, Intercept=0.2430, Slope=8.9126\n",
+ "Epoch 300: Cost=71.7831, Intercept=0.2627, Slope=9.6356\n",
+ "Epoch 400: Cost=7.8504, Intercept=0.2692, Slope=9.8746\n",
+ "Epoch 500: Cost=0.8639, Intercept=0.2713, Slope=9.9536\n",
+ "Epoch 600: Cost=0.1004, Intercept=0.2719, Slope=9.9797\n",
+ "Epoch 700: Cost=0.0170, Intercept=0.2721, Slope=9.9883\n",
+ "Epoch 800: Cost=0.0079, Intercept=0.2721, Slope=9.9912\n",
+ "Epoch 900: Cost=0.0069, Intercept=0.2721, Slope=9.9921\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Multivarirate Vectorized Linear Regression: Using Numpy (computinng the Graident Descent)\n"
+ ],
+ "metadata": {
+ "id": "VSAfzgIlFz9W"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import numpy as np\n",
+ "\n",
+ "# Step 1: Input data (without bias column)\n",
+ "X_raw = np.array([\n",
+ " [2100, 3],\n",
+ " [1600, 2],\n",
+ " [2400, 4]\n",
+ "]) # shape: (3, 2)\n",
+ "\n",
+ "y = np.array([\n",
+ " [400],\n",
+ " [330],\n",
+ " [369]\n",
+ "]) # shape: (3, 1)\n",
+ "\n",
+ "m = len(y) # number of examples\n",
+ "\n",
+ "# Step 2: Feature Normalization (mean normalization)\n",
+ "mean = np.mean(X_raw, axis=0)\n",
+ "std = np.std(X_raw, axis=0)\n",
+ "X_scaled = (X_raw - mean) / std # normalized features\n",
+ "\n",
+ "# Step 3: Add bias term (column of 1s)\n",
+ "X = np.c_[np.ones((m, 1)), X_scaled] # shape: (3, 3)\n",
+ "\n",
+ "# Step 4: Initialize parameters\n",
+ "theta = np.zeros((X.shape[1], 1)) # shape: (3, 1)\n",
+ "\n",
+ "# Step 5: Hyperparameters\n",
+ "alpha = 0.01\n",
+ "epochs = 1000\n",
+ "\n",
+ "# Step 6: Gradient Descent loop\n",
+ "for epoch in range(epochs):\n",
+ " y_pred = X @ theta # predictions: shape (m, 1)\n",
+ " error = y_pred - y # error: shape (m, 1)\n",
+ " gradients = (1/m) * (X.T @ error) # shape: (n+1, 1)\n",
+ " theta -= alpha * gradients # weigths update\n",
+ "\n",
+ " # Optional: print cost every 100 epochs\n",
+ " if epoch % 100 == 0:\n",
+ " cost = (1/(2*m)) * np.sum(error**2)\n",
+ " print(f\"Epoch {epoch}: Cost = {cost:.4f}\")\n",
+ "\n",
+ "# Final theta\n",
+ "print(\"\\nFinal parameters:\")\n",
+ "print(f\"Intercept (theta_0): {theta[0][0]:.4f}\")\n",
+ "print(f\"Slope for feature 1 (theta_1): {theta[1][0]:.4f}\")\n",
+ "print(f\"Slope for feature 2 (theta_2): {theta[2][0]:.4f}\")"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "MvedeQyKOE3y",
+ "outputId": "7fcb3a9e-1fdd-4c6d-b96a-37f6f98089df"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Epoch 0: Cost = 67510.1667\n",
+ "Epoch 100: Cost = 9243.1174\n",
+ "Epoch 200: Cost = 1449.7473\n",
+ "Epoch 300: Cost = 401.6122\n",
+ "Epoch 400: Cost = 256.9782\n",
+ "Epoch 500: Cost = 233.4748\n",
+ "Epoch 600: Cost = 226.2841\n",
+ "Epoch 700: Cost = 221.3611\n",
+ "Epoch 800: Cost = 216.8223\n",
+ "Epoch 900: Cost = 212.4137\n",
+ "\n",
+ "Final parameters:\n",
+ "Intercept (theta_0): 366.3175\n",
+ "Slope for feature 1 (theta_1): 24.2014\n",
+ "Slope for feature 2 (theta_2): -6.5705\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### That means your final learned hypothesis function (in normalized feature space) is:\n",
+ "\n",
+ "𝑦\n",
+ "^\n",
+ "=\n",
+ "366.3175\n",
+ "+\n",
+ "24.2014\n",
+ "⋅\n",
+ "𝑥\n",
+ "1\n",
+ "(\n",
+ "𝑠\n",
+ "𝑐\n",
+ "𝑎\n",
+ "𝑙\n",
+ "𝑒\n",
+ "𝑑\n",
+ ")\n",
+ "−\n",
+ "6.5705\n",
+ "⋅\n",
+ "𝑥\n",
+ "2\n",
+ "(\n",
+ "𝑠\n",
+ "𝑐\n",
+ "𝑎\n",
+ "𝑙\n",
+ "𝑒\n",
+ "𝑑\n",
+ ")\n",
+ "y\n",
+ "^\n",
+ "\n",
+ " =366.3175+24.2014⋅x\n",
+ "1\n",
+ "(scaled)\n",
+ "\n",
+ " −6.5705⋅x\n",
+ "2\n",
+ "(scaled)\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "qXIZC6OcQePX"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This confirms:\n",
+ "\n",
+ "\n",
+ "\n",
+ "* Gradient descent is converging.\n",
+ "* Feature scaling worked (no nan or exploding values).\n",
+ "* Learning rate alpha = 0.01 is reasonable.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "Csha3TBCQqDN"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [],
+ "metadata": {
+ "id": "Hk504ONsQp_u"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Boston House Price Prediction (Linear Regression Model)\n",
+ "\n",
+ "\n",
+ "---\n",
+ "In this project you will work on developing an end to end machine learning project using linear regression, as this will be your first project in your machine learning journey. We will be doing extensive data visualization, we will perform data feature engineering, we will also see how we can select features based on the correlation of the features."
+ ],
+ "metadata": {
+ "id": "nJhUTYCk9fj-"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import matplotlib.pyplot as plt\n",
+ "import seaborn as sns\n",
+ "%matplotlib inline"
+ ],
+ "metadata": {
+ "id": "pGUrj4Qx9pAe"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import pandas as pd\n",
+ "import numpy as np\n",
+ "\n",
+ "\"\"\"DATA Preprocessing:\"\"\"\n",
+ "\n",
+ "data_url = \"http://lib.stat.cmu.edu/datasets/boston\"\n",
+ "raw_df = pd.read_csv(data_url, sep=\"\\s+\", skiprows=22, header=None)\n",
+ "data_array = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])\n",
+ "target_array = raw_df.values[1::2, 2]\n",
+ "\n",
+ "# Create a pandas DataFrame with appropriate column names\n",
+ "feature_names = [\n",
+ " 'CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX',\n",
+ " 'PTRATIO', 'B', 'LSTAT'\n",
+ "]\n",
+ "data = pd.DataFrame(data_array, columns=feature_names)\n",
+ "data[\"SalePrice\"] = target_array # saleprice\n",
+ "data.head()"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 206
+ },
+ "id": "7iElMruH_SHa",
+ "outputId": "829d31a2-e44d-4969-c0c3-2b01dfac3b54"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " CRIM ZN INDUS CHAS NOX RM AGE DIS RAD TAX \\\n",
+ "0 0.00632 18.0 2.31 0.0 0.538 6.575 65.2 4.0900 1.0 296.0 \n",
+ "1 0.02731 0.0 7.07 0.0 0.469 6.421 78.9 4.9671 2.0 242.0 \n",
+ "2 0.02729 0.0 7.07 0.0 0.469 7.185 61.1 4.9671 2.0 242.0 \n",
+ "3 0.03237 0.0 2.18 0.0 0.458 6.998 45.8 6.0622 3.0 222.0 \n",
+ "4 0.06905 0.0 2.18 0.0 0.458 7.147 54.2 6.0622 3.0 222.0 \n",
+ "\n",
+ " PTRATIO B LSTAT SalePrice \n",
+ "0 15.3 396.90 4.98 24.0 \n",
+ "1 17.8 396.90 9.14 21.6 \n",
+ "2 17.8 392.83 4.03 34.7 \n",
+ "3 18.7 394.63 2.94 33.4 \n",
+ "4 18.7 396.90 5.33 36.2 "
+ ],
+ "text/html": [
+ "\n",
+ " | \n", + " | CRIM | \n", + "ZN | \n", + "INDUS | \n", + "CHAS | \n", + "NOX | \n", + "RM | \n", + "AGE | \n", + "DIS | \n", + "RAD | \n", + "TAX | \n", + "PTRATIO | \n", + "B | \n", + "LSTAT | \n", + "SalePrice | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "0.00632 | \n", + "18.0 | \n", + "2.31 | \n", + "0.0 | \n", + "0.538 | \n", + "6.575 | \n", + "65.2 | \n", + "4.0900 | \n", + "1.0 | \n", + "296.0 | \n", + "15.3 | \n", + "396.90 | \n", + "4.98 | \n", + "24.0 | \n", + "
| 1 | \n", + "0.02731 | \n", + "0.0 | \n", + "7.07 | \n", + "0.0 | \n", + "0.469 | \n", + "6.421 | \n", + "78.9 | \n", + "4.9671 | \n", + "2.0 | \n", + "242.0 | \n", + "17.8 | \n", + "396.90 | \n", + "9.14 | \n", + "21.6 | \n", + "
| 2 | \n", + "0.02729 | \n", + "0.0 | \n", + "7.07 | \n", + "0.0 | \n", + "0.469 | \n", + "7.185 | \n", + "61.1 | \n", + "4.9671 | \n", + "2.0 | \n", + "242.0 | \n", + "17.8 | \n", + "392.83 | \n", + "4.03 | \n", + "34.7 | \n", + "
| 3 | \n", + "0.03237 | \n", + "0.0 | \n", + "2.18 | \n", + "0.0 | \n", + "0.458 | \n", + "6.998 | \n", + "45.8 | \n", + "6.0622 | \n", + "3.0 | \n", + "222.0 | \n", + "18.7 | \n", + "394.63 | \n", + "2.94 | \n", + "33.4 | \n", + "
| 4 | \n", + "0.06905 | \n", + "0.0 | \n", + "2.18 | \n", + "0.0 | \n", + "0.458 | \n", + "7.147 | \n", + "54.2 | \n", + "6.0622 | \n", + "3.0 | \n", + "222.0 | \n", + "18.7 | \n", + "396.90 | \n", + "5.33 | \n", + "36.2 | \n", + "
| \n", + " | 0 | \n", + "1 | \n", + "2 | \n", + "3 | \n", + "4 | \n", + "5 | \n", + "6 | \n", + "7 | \n", + "8 | \n", + "9 | \n", + "10 | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|
| count | \n", + "1012.000000 | \n", + "1012.000000 | \n", + "1012.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "
| mean | \n", + "180.143778 | \n", + "12.008350 | \n", + "16.834792 | \n", + "0.069170 | \n", + "0.554695 | \n", + "6.284634 | \n", + "68.574901 | \n", + "3.795043 | \n", + "9.549407 | \n", + "408.237154 | \n", + "18.455534 | \n", + "
| std | \n", + "188.132839 | \n", + "17.250728 | \n", + "9.912616 | \n", + "0.253994 | \n", + "0.115878 | \n", + "0.702617 | \n", + "28.148861 | \n", + "2.105710 | \n", + "8.707259 | \n", + "168.537116 | \n", + "2.164946 | \n", + "
| min | \n", + "0.006320 | \n", + "0.000000 | \n", + "0.460000 | \n", + "0.000000 | \n", + "0.385000 | \n", + "3.561000 | \n", + "2.900000 | \n", + "1.129600 | \n", + "1.000000 | \n", + "187.000000 | \n", + "12.600000 | \n", + "
| 25% | \n", + "0.257830 | \n", + "0.000000 | \n", + "8.375000 | \n", + "0.000000 | \n", + "0.449000 | \n", + "5.885500 | \n", + "45.025000 | \n", + "2.100175 | \n", + "4.000000 | \n", + "279.000000 | \n", + "17.400000 | \n", + "
| 50% | \n", + "24.021000 | \n", + "7.240000 | \n", + "18.100000 | \n", + "0.000000 | \n", + "0.538000 | \n", + "6.208500 | \n", + "77.500000 | \n", + "3.207450 | \n", + "5.000000 | \n", + "330.000000 | \n", + "19.050000 | \n", + "
| 75% | \n", + "391.435000 | \n", + "16.780000 | \n", + "21.890000 | \n", + "0.000000 | \n", + "0.624000 | \n", + "6.623500 | \n", + "94.075000 | \n", + "5.188425 | \n", + "24.000000 | \n", + "666.000000 | \n", + "20.200000 | \n", + "
| max | \n", + "396.900000 | \n", + "100.000000 | \n", + "50.000000 | \n", + "1.000000 | \n", + "0.871000 | \n", + "8.780000 | \n", + "100.000000 | \n", + "12.126500 | \n", + "24.000000 | \n", + "711.000000 | \n", + "22.000000 | \n", + "
| \n", + " | CRIM | \n", + "ZN | \n", + "INDUS | \n", + "CHAS | \n", + "NOX | \n", + "RM | \n", + "AGE | \n", + "DIS | \n", + "RAD | \n", + "TAX | \n", + "PTRATIO | \n", + "B | \n", + "LSTAT | \n", + "SalePrice | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| count | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "506.000000 | \n", + "
| mean | \n", + "3.613524 | \n", + "11.363636 | \n", + "11.136779 | \n", + "0.069170 | \n", + "0.554695 | \n", + "6.284634 | \n", + "68.574901 | \n", + "3.795043 | \n", + "9.549407 | \n", + "408.237154 | \n", + "18.455534 | \n", + "356.674032 | \n", + "12.653063 | \n", + "22.532806 | \n", + "
| std | \n", + "8.601545 | \n", + "23.322453 | \n", + "6.860353 | \n", + "0.253994 | \n", + "0.115878 | \n", + "0.702617 | \n", + "28.148861 | \n", + "2.105710 | \n", + "8.707259 | \n", + "168.537116 | \n", + "2.164946 | \n", + "91.294864 | \n", + "7.141062 | \n", + "9.197104 | \n", + "
| min | \n", + "0.006320 | \n", + "0.000000 | \n", + "0.460000 | \n", + "0.000000 | \n", + "0.385000 | \n", + "3.561000 | \n", + "2.900000 | \n", + "1.129600 | \n", + "1.000000 | \n", + "187.000000 | \n", + "12.600000 | \n", + "0.320000 | \n", + "1.730000 | \n", + "5.000000 | \n", + "
| 25% | \n", + "0.082045 | \n", + "0.000000 | \n", + "5.190000 | \n", + "0.000000 | \n", + "0.449000 | \n", + "5.885500 | \n", + "45.025000 | \n", + "2.100175 | \n", + "4.000000 | \n", + "279.000000 | \n", + "17.400000 | \n", + "375.377500 | \n", + "6.950000 | \n", + "17.025000 | \n", + "
| 50% | \n", + "0.256510 | \n", + "0.000000 | \n", + "9.690000 | \n", + "0.000000 | \n", + "0.538000 | \n", + "6.208500 | \n", + "77.500000 | \n", + "3.207450 | \n", + "5.000000 | \n", + "330.000000 | \n", + "19.050000 | \n", + "391.440000 | \n", + "11.360000 | \n", + "21.200000 | \n", + "
| 75% | \n", + "3.677083 | \n", + "12.500000 | \n", + "18.100000 | \n", + "0.000000 | \n", + "0.624000 | \n", + "6.623500 | \n", + "94.075000 | \n", + "5.188425 | \n", + "24.000000 | \n", + "666.000000 | \n", + "20.200000 | \n", + "396.225000 | \n", + "16.955000 | \n", + "25.000000 | \n", + "
| max | \n", + "88.976200 | \n", + "100.000000 | \n", + "27.740000 | \n", + "1.000000 | \n", + "0.871000 | \n", + "8.780000 | \n", + "100.000000 | \n", + "12.126500 | \n", + "24.000000 | \n", + "711.000000 | \n", + "22.000000 | \n", + "396.900000 | \n", + "37.970000 | \n", + "50.000000 | \n", + "
| \n", + " | 0 | \n", + "
|---|---|
| CRIM | \n", + "0 | \n", + "
| ZN | \n", + "0 | \n", + "
| INDUS | \n", + "0 | \n", + "
| CHAS | \n", + "0 | \n", + "
| NOX | \n", + "0 | \n", + "
| RM | \n", + "0 | \n", + "
| AGE | \n", + "0 | \n", + "
| DIS | \n", + "0 | \n", + "
| RAD | \n", + "0 | \n", + "
| TAX | \n", + "0 | \n", + "
| PTRATIO | \n", + "0 | \n", + "
| B | \n", + "0 | \n", + "
| LSTAT | \n", + "0 | \n", + "
| SalePrice | \n", + "0 | \n", + "
LinearRegression()In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
LinearRegression()
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "id": "bsc-isJ4vm8Y"
+ },
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "\n",
+ "class LogisticRegression:\n",
+ " \"\"\"\n",
+ " Logistic Regression classifier using gradient descent.\n",
+ "\n",
+ " Parameters:\n",
+ " ----------\n",
+ " learning_rate : float\n",
+ " The step size for gradient descent updates.\n",
+ " num_iter : int\n",
+ " Number of iterations for training loop.\n",
+ "\n",
+ " Attributes:\n",
+ " ----------\n",
+ " w : ndarray\n",
+ " Weight vector.\n",
+ " b : float\n",
+ " Bias term.\n",
+ " \"\"\"\n",
+ "\n",
+ " def __init__(self, learning_rate=0.01, num_iter=1000):\n",
+ " self.learning_rate = learning_rate\n",
+ " self.num_iter = num_iter\n",
+ " self.w = None\n",
+ " self.b = None\n",
+ "\n",
+ " def sigmoid(self, z):\n",
+ " \"\"\"\n",
+ " Sigmoid activation function.\n",
+ "\n",
+ " Parameters:\n",
+ " ----------\n",
+ " z : ndarray\n",
+ " Linear combination of weights and inputs.\n",
+ "\n",
+ " Returns:\n",
+ " -------\n",
+ " ndarray\n",
+ " Output of the sigmoid function applied element-wise.\n",
+ " \"\"\"\n",
+ " return 1 / (1 + np.exp(-z))\n",
+ "\n",
+ " def fit(self, X, y):\n",
+ " \"\"\"\n",
+ " Train the logistic regression model using gradient descent.\n",
+ "\n",
+ " Parameters:\n",
+ " ----------\n",
+ " X : ndarray of shape (n_samples, n_features)\n",
+ " Training data.\n",
+ " y : ndarray of shape (n_samples,)\n",
+ " Binary target labels (0 or 1).\n",
+ " \"\"\"\n",
+ " n_samples, n_features = X.shape\n",
+ " self.w = np.zeros(n_features)\n",
+ " self.b = 0\n",
+ "\n",
+ " for _ in range(self.num_iter):\n",
+ " # Linear model prediction\n",
+ " linear_model = np.dot(X, self.w) + self.b\n",
+ " y_predicted = self.sigmoid(linear_model)\n",
+ "\n",
+ " # Compute gradients\n",
+ " dw = (1 / n_samples) * np.dot(X.T, (y_predicted - y))\n",
+ " db = (1 / n_samples) * np.sum(y_predicted - y)\n",
+ "\n",
+ " # Update parameters\n",
+ " self.w -= self.learning_rate * dw\n",
+ " self.b -= self.learning_rate * db\n",
+ "\n",
+ " def predict_proba(self, X):\n",
+ " \"\"\"\n",
+ " Predict class probabilities for input data.\n",
+ "\n",
+ " Parameters:\n",
+ " ----------\n",
+ " X : ndarray of shape (n_samples, n_features)\n",
+ "\n",
+ " Returns:\n",
+ " -------\n",
+ " ndarray\n",
+ " Probabilities for class 1.\n",
+ " \"\"\"\n",
+ " linear_model = np.dot(X, self.w) + self.b\n",
+ " return self.sigmoid(linear_model)\n",
+ "\n",
+ " def predict(self, X):\n",
+ " \"\"\"\n",
+ " Predict binary class labels for input data.\n",
+ "\n",
+ " Parameters:\n",
+ " ----------\n",
+ " X : ndarray of shape (n_samples, n_features)\n",
+ "\n",
+ " Returns:\n",
+ " -------\n",
+ " ndarray\n",
+ " Predicted class labels (0 or 1).\n",
+ " \"\"\"\n",
+ " y_probs = self.predict_proba(X)\n",
+ " return np.where(y_probs >= 0.5, 1, 0)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [],
+ "metadata": {
+ "id": "DzKzJWOVy-O6"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [],
+ "metadata": {
+ "id": "0BzbHCHXy-MO"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## LogisticRegression implementation:\n",
+ "\n",
+ "### Init:\n",
+ "set learning rate, iterations, initialize weights w and bias b.\n",
+ "\n",
+ "### Sigmoid:\n",
+ "squash input z into range (0,1) using 1/(1+exp(-z)).\n",
+ "\n",
+ "### Fit:\n",
+ "loop num_iter → compute linear model (Xw+b) → apply sigmoid → compute gradients (dw, db) → update weights and bias with gradient descent.\n",
+ "\n",
+ "### Predict_proba:\n",
+ "return sigmoid of (Xw+b) as probabilities.\n",
+ "\n",
+ "### Predict:\n",
+ "threshold probabilities at 0.5 → return 0 or 1 labels."
+ ],
+ "metadata": {
+ "id": "WV05DXP4zIbB"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "X = np.array([\n",
+ " [1, 2],\n",
+ " [2, 1],\n",
+ " [2, 3],\n",
+ " [3, 2],\n",
+ " [3, 4],\n",
+ " [4, 3]\n",
+ "])\n",
+ "\n",
+ "y = np.array([0, 0, 0, 1, 1, 1])"
+ ],
+ "metadata": {
+ "id": "9YkHjJ4qwUIl"
+ },
+ "execution_count": 2,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "model = LogisticRegression (learning_rate=0.01, num_iter=1000)\n",
+ "model.fit(X, y)"
+ ],
+ "metadata": {
+ "id": "M8PL-8QxwlJQ"
+ },
+ "execution_count": 3,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "print(\"Weights:\", model.w)\n",
+ "print(\"Bias:\", model.b)\n"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "6OE3jFuCwxCH",
+ "outputId": "62eca782-8f07-446b-e5c1-5397f981ee66"
+ },
+ "execution_count": 5,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Weights: [ 0.777979 -0.21385172]\n",
+ "Bias: -0.9945904172925308\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "preds = model.predict(X)\n",
+ "print(\"Predictions:\", preds)\n",
+ "print(\"True labels:\", y)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "wFEXXBgUxSVl",
+ "outputId": "8316b033-3dba-4742-a13d-21d616e82b8c"
+ },
+ "execution_count": 8,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Predictions: [0 1 0 1 1 1]\n",
+ "True labels: [0 0 0 1 1 1]\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "probs = model.predict_proba(X)\n",
+ "print(\"Predicted probabilities:\", probs)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "UXzgHTJFxcVj",
+ "outputId": "5ec42352-200b-4d5a-976a-ee59ab393324"
+ },
+ "execution_count": 9,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Predicted probabilities: [0.34427181 0.58601505 0.47996384 0.71333628 0.61867775 0.81393288]\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "accuracy = np.mean(preds == y)\n",
+ "print(f\"Accuracy:, {np.round(accuracy,2)*100}%\")"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "dmjc7sJWx9s1",
+ "outputId": "55d5eacc-6dda-489b-e46b-3dcadc36dbd4"
+ },
+ "execution_count": 13,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Accuracy:, 83.0%\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# testing on unseen samples\n",
+ "X_test = np.array([\n",
+ " [1, 1], # should lean towards class 0\n",
+ " [4, 4] # should lean towards class 1\n",
+ "])\n",
+ "\n",
+ "print(\"Test predictions:\", model.predict(X_test))\n",
+ "print(\"Test probabilities:\", model.predict_proba(X_test))"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "ULfo8Yy-yGos",
+ "outputId": "de671bb8-b579-4550-dca8-c265cc2a3a89"
+ },
+ "execution_count": 14,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Test predictions: [0 1]\n",
+ "Test probabilities: [0.39401574 0.77935623]\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [],
+ "metadata": {
+ "id": "ZcrEYB0AyzvX"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [],
+ "metadata": {
+ "id": "003dtL-azwkx"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "## Example: Using the Class with make_classification"
+ ],
+ "metadata": {
+ "id": "h5bmTtyDz0w6"
+ },
+ "execution_count": 15,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from sklearn.datasets import make_classification\n",
+ "from sklearn.model_selection import train_test_split\n",
+ "from sklearn.metrics import accuracy_score\n",
+ "\n",
+ "# Generate synthetic binary classification dataset\n",
+ "X, y = make_classification(\n",
+ " n_samples=1000,\n",
+ " n_features=10,\n",
+ " n_classes=2,\n",
+ " random_state=42\n",
+ ")\n",
+ "\n",
+ "# Split into training and test sets\n",
+ "X_train, X_test, y_train, y_test = train_test_split(\n",
+ " X, y, test_size=0.2, random_state=42\n",
+ ")\n",
+ "\n",
+ "# Initialize and train model\n",
+ "model = LogisticRegression(learning_rate=0.1, num_iter=1000)\n",
+ "model.fit(X_train, y_train)\n",
+ "\n",
+ "# Predict and evaluate\n",
+ "y_pred = model.predict(X_test)\n",
+ "acc = accuracy_score(y_test, y_pred)\n",
+ "\n",
+ "print(\"Test Accuracy:\", acc)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "Hpov3mxnzwVy",
+ "outputId": "fa883083-278b-4030-ec23-99b6eabf5ce2"
+ },
+ "execution_count": 16,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Test Accuracy: 0.825\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [],
+ "metadata": {
+ "id": "ZlCq7Ox0z3RH"
+ },
+ "execution_count": null,
+ "outputs": []
+ }
+ ]
+}
diff --git a/implementations/trees/decision_trees_hands_on.ipynb b/implementations/trees/decision_trees_hands_on.ipynb
new file mode 100644
index 000000000..5ae62b535
--- /dev/null
+++ b/implementations/trees/decision_trees_hands_on.ipynb
@@ -0,0 +1,844 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "source": [
+ "# Decision Tree Classification in Python\n",
+ "**In this tutorial, learn Decision Tree Classification, attribute selection measures, and how to build and optimize Decision Tree Classifier using Python Scikit-learn package.**"
+ ],
+ "metadata": {
+ "id": "strange-investigator"
+ },
+ "id": "strange-investigator",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "As a marketing manager, you want a set of customers who are most likely to purchase your product. This is how you can save your marketing budget by finding your audience. As a loan manager, you need to identify risky loan applications to achieve a lower loan default rate. This process of classifying customers into a group of potential and non-potential customers or safe or risky loan applications is known as a classification problem. Classification is a two-step process, learning step and prediction step. In the learning step, the model is developed based on given training data. In the prediction step, the model is used to predict the response for given data. Decision Tree is one of the easiest and popular classification algorithms to understand and interpret. It can be utilized for both classification and regression kind of problem.\n",
+ "\n",
+ "In this tutorial, you are going to cover the following topics:\n",
+ "\n",
+ "- Decision Tree Algorithm\n",
+ "- How does the Decision Tree algorithm work?\n",
+ "- Attribute Selection Measures\n",
+ " - Information Gain\n",
+ " - Gain Ratio\n",
+ " - Gini index\n",
+ "- Optimizing Decision Tree Performance \n",
+ "- Classifier Building in Scikit-learn\n",
+ "- Pros and Cons\n",
+ "- Conclusion\n"
+ ],
+ "metadata": {
+ "id": "precious-tuning"
+ },
+ "id": "precious-tuning",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "## Decision Tree Algorithm\n",
+ "\n",
+ "A decision tree is a flowchart-like tree structure where an internal node represents feature(or attribute), the branch represents a decision rule, and each leaf node represents the outcome. The topmost node in a decision tree is known as the root node. It learns to partition on the basis of the attribute value. It partitions the tree in a recursive manner called recursive partitioning. This flowchart-like structure helps you in decision making. It can be visualized like a flowchart diagram which easily mimics human level thinking. That is why decision trees are easy to understand and interpret.\n",
+ "\n",
+ ""
+ ],
+ "metadata": {
+ "id": "patient-giving"
+ },
+ "id": "patient-giving",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "Decision Tree is a white box type of ML algorithm. It shares internal decision-making logic, which is not available in the black box type of algorithms such as Neural Network. Its training time is faster compared to the neural network algorithm. The time complexity of decision trees is a function of the number of records and number of attributes in the given data. The decision tree is a distribution-free or non-parametric method, which does not depend upon probability distribution assumptions. Decision trees can handle high dimensional data with good accuracy. "
+ ],
+ "metadata": {
+ "id": "cordless-installation"
+ },
+ "id": "cordless-installation",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "## How does the Decision Tree algorithm work?\n",
+ "\n",
+ "The basic idea behind any decision tree algorithm is as follows:\n",
+ "\n",
+ "1. Select the best attribute using Attribute Selection Measures(ASM) to split the records.\n",
+ "2. Make that attribute a decision node and breaks the dataset into smaller subsets.\n",
+ "3. Start tree building by repeating this process recursively for each child until one of the condition will match:\n",
+ " - All the tuples belong to the same attribute value.\n",
+ " - There are no more remaining attributes.\n",
+ " - There are no more instances.\n",
+ " \n",
+ ""
+ ],
+ "metadata": {
+ "id": "laughing-gateway"
+ },
+ "id": "laughing-gateway",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "## Attribute Selection Measures\n",
+ "\n",
+ "Attribute selection measure is a heuristic for selecting the splitting criterion that partition data into the best possible manner. It is also known as splitting rules because it helps us to determine breakpoints for tuples on a given node. ASM provides a rank to each feature(or attribute) by explaining the given dataset. Best score attribute will be selected as a splitting attribute ([Source](http://www.ijoart.org/docs/Construction-of-Decision-Tree--Attribute-Selection-Measures.pdf)). In the case of a continuous-valued attribute, split points for branches also need to define. Most popular selection measures are Information Gain, Gain Ratio, and Gini Index."
+ ],
+ "metadata": {
+ "id": "backed-potential"
+ },
+ "id": "backed-potential",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "### Information Gain \n",
+ "\n",
+ "Shannon invented the concept of entropy, which measures the impurity of the input set. In physics and mathematics, entropy referred as the randomness or the impurity in the system. In information theory, it refers to the impurity in a group of examples. Information gain is the decrease in entropy. Information gain computes the difference between entropy before split and average entropy after split of the dataset based on given attribute values. ID3 (Iterative Dichotomiser) decision tree algorithm uses information gain.\n",
+ "\n",
+ "$$\\textrm{Info}(D) = -\\sum_{i = 1}^{m} p_i\\log_2 p_i$$\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "secret-curtis"
+ },
+ "id": "secret-curtis",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "Where, Pi is the probability that an arbitrary tuple in D belongs to class Ci.\n",
+ "\n",
+ "$$\\textrm{Info}_A(D) = \\sum_{j = 1}^{V} \\dfrac{|D_j|}{|D|} \\textrm{ x }\\textrm{Info}(D_j)$$\n",
+ "\n",
+ "$$\\textrm{Gain}(A) = \\textrm{Info}(D) - \\textrm{Info}_A(D)$$\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "thorough-version"
+ },
+ "id": "thorough-version",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "Where:\n",
+ "\n",
+ "- **Info(D)** is the average amount of information needed to identify the class label of a tuple in D.\n",
+ "- **|Dj|/|D|** acts as the weight of the jth partition.\n",
+ "- **InfoA(D)** is the expected information required to classify a tuple from D based on the partitioning by A.\n",
+ "\n",
+ "The attribute A with the highest information gain, **Gain(A)**, is chosen as the splitting attribute at node N()."
+ ],
+ "metadata": {
+ "id": "accompanied-happening"
+ },
+ "id": "accompanied-happening",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "### Gain Ratio\n",
+ "\n",
+ "Information gain is biased for the attribute with many outcomes. It means it prefers the attribute with a large number of distinct values. For instance, consider an attribute with a unique identifier such as customer_ID has zero info(D) because of pure partition. This maximizes the information gain and creates useless partitioning.\n",
+ "\n",
+ "C4.5, an improvement of ID3, uses an extension to information gain known as the gain ratio. Gain ratio handles the issue of bias by normalizing the information gain using Split Info. Java implementation of the C4.5 algorithm is known as J48, which is available in WEKA data mining tool.\n",
+ "\n",
+ "$$ \\textrm{SplitInfo}_A(D) = - \\sum_{j = 1}^{V} \\dfrac{|D_j|}{|D|} \\textrm{ x } log_2(\\frac{|D_j|}{|D|})$$"
+ ],
+ "metadata": {
+ "id": "italian-backup"
+ },
+ "id": "italian-backup",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "Where:\n",
+ "- |Dj|/|D| acts as the weight of the jth partition.\n",
+ "- v is the number of discrete values in attribute A."
+ ],
+ "metadata": {
+ "id": "included-valve"
+ },
+ "id": "included-valve",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "The gain ratio can be defined as:\n",
+ "\n",
+ "$$ \\textrm{GainRatio}(A) = \\dfrac{\\textrm{Gain(A)}}{\\textrm{SplitInfo}_A(D)} $$"
+ ],
+ "metadata": {
+ "id": "liable-fiction"
+ },
+ "id": "liable-fiction",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "The attribute with the highest gain ratio is chosen as the splitting attribute ([Source](http://www.enggjournals.com/ijcse/doc/IJCSE10-02-09-092.pdf))."
+ ],
+ "metadata": {
+ "id": "coordinate-separate"
+ },
+ "id": "coordinate-separate",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "### Gini index\n",
+ "\n",
+ "Another decision tree algorithm CART (Classification and Regression Tree) uses the Gini method to create split points.\n",
+ "\n",
+ "$$Gini(D) = 1 - \\sum_{i = 1}^{m} P_i^2 $$"
+ ],
+ "metadata": {
+ "id": "hearing-ensemble"
+ },
+ "id": "hearing-ensemble",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "Where, pi is the probability that a tuple in D belongs to class Ci.\n",
+ "\n",
+ "The Gini Index considers a binary split for each attribute. You can compute a weighted sum of the impurity of each partition. If a binary split on attribute A partitions data D into D1 and D2, the Gini index of D is:\n",
+ "\n",
+ "$$Gini_A(D) = \\frac{|D_1|}{D} Gini(D_1) + \\frac{D_2}{D} Gini(D_2)$$"
+ ],
+ "metadata": {
+ "id": "ordinary-tuner"
+ },
+ "id": "ordinary-tuner",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "In case of a discrete-valued attribute, the subset that gives the minimum gini index for that chosen is selected as a splitting attribute. In the case of continuous-valued attributes, the strategy is to select each pair of adjacent values as a possible split-point and point with smaller gini index chosen as the splitting point.\n",
+ "\n",
+ "$$ \\Delta Gini(A) = Gini(D) - Gini_A(D)$$\n"
+ ],
+ "metadata": {
+ "id": "expired-syracuse"
+ },
+ "id": "expired-syracuse",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "The attribute with minimum Gini index is chosen as the splitting attribute."
+ ],
+ "metadata": {
+ "id": "continental-australian"
+ },
+ "id": "continental-australian",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "## Decision Tree Classifier Building in Scikit-learn\n",
+ "\n",
+ "### Importing Required Libraries\n",
+ "\n",
+ "Let's first install the required packages and load the required libraries.\n"
+ ],
+ "metadata": {
+ "id": "accepted-dependence"
+ },
+ "id": "accepted-dependence",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "#!pip install --quiet -r requirements.txt;"
+ ],
+ "metadata": {
+ "id": "variable-detroit"
+ },
+ "id": "variable-detroit",
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "source": [
+ "# Load libraries\n",
+ "import pandas as pd\n",
+ "from sklearn.tree import DecisionTreeClassifier # Import Decision Tree Classifier\n",
+ "from sklearn.model_selection import train_test_split # Import train_test_split function\n",
+ "from sklearn import metrics #Import scikit-learn metrics module for accuracy calculation"
+ ],
+ "metadata": {
+ "id": "prerequisite-highway"
+ },
+ "id": "prerequisite-highway",
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "source": [
+ "### Loading Data\n",
+ "\n",
+ "Let's first load the required Pima Indian Diabetes dataset using pandas' read CSV function. You can download the data [here](https://www.kaggle.com/uciml/pima-indians-diabetes-database).\n"
+ ],
+ "metadata": {
+ "id": "suspected-fence"
+ },
+ "id": "suspected-fence",
+ "cell_type": "markdown"
+ },
+ {
+ "source": [
+ "col_names = ['pregnant', 'glucose', 'bp', 'skin', 'insulin', 'bmi', 'pedigree', 'age', 'label']\n",
+ "# load dataset\n",
+ "pima = pd.read_csv(\"data/pima-indians-diabetes.csv\", header=None, names=col_names)"
+ ],
+ "metadata": {
+ "id": "cutting-institution"
+ },
+ "id": "cutting-institution",
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "source": [
+ "pima.head()"
+ ],
+ "metadata": {
+ "outputsMetadata": {
+ "0": {
+ "height": 550,
+ "type": "dataFrame",
+ "tableState": {}
+ }
+ },
+ "id": "grateful-benjamin",
+ "outputId": "659115cc-b961-470b-ecb9-e6c346bcc5eb"
+ },
+ "id": "grateful-benjamin",
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "application/com.datacamp.data-table.v1+json": {
+ "table": {
+ "schema": {
+ "fields": [
+ {
+ "name": "index",
+ "type": "integer"
+ },
+ {
+ "name": "pregnant",
+ "type": "integer"
+ },
+ {
+ "name": "glucose",
+ "type": "integer"
+ },
+ {
+ "name": "bp",
+ "type": "integer"
+ },
+ {
+ "name": "skin",
+ "type": "integer"
+ },
+ {
+ "name": "insulin",
+ "type": "integer"
+ },
+ {
+ "name": "bmi",
+ "type": "number"
+ },
+ {
+ "name": "pedigree",
+ "type": "number"
+ },
+ {
+ "name": "age",
+ "type": "integer"
+ },
+ {
+ "name": "label",
+ "type": "integer"
+ }
+ ],
+ "primaryKey": [
+ "index"
+ ],
+ "pandas_version": "1.4.0"
+ },
+ "data": [
+ {
+ "index": 0,
+ "pregnant": 6,
+ "glucose": 148,
+ "bp": 72,
+ "skin": 35,
+ "insulin": 0,
+ "bmi": 33.6,
+ "pedigree": 0.627,
+ "age": 50,
+ "label": 1
+ },
+ {
+ "index": 1,
+ "pregnant": 1,
+ "glucose": 85,
+ "bp": 66,
+ "skin": 29,
+ "insulin": 0,
+ "bmi": 26.6,
+ "pedigree": 0.351,
+ "age": 31,
+ "label": 0
+ },
+ {
+ "index": 2,
+ "pregnant": 8,
+ "glucose": 183,
+ "bp": 64,
+ "skin": 0,
+ "insulin": 0,
+ "bmi": 23.3,
+ "pedigree": 0.672,
+ "age": 32,
+ "label": 1
+ },
+ {
+ "index": 3,
+ "pregnant": 1,
+ "glucose": 89,
+ "bp": 66,
+ "skin": 23,
+ "insulin": 94,
+ "bmi": 28.1,
+ "pedigree": 0.167,
+ "age": 21,
+ "label": 0
+ },
+ {
+ "index": 4,
+ "pregnant": 0,
+ "glucose": 137,
+ "bp": 40,
+ "skin": 35,
+ "insulin": 168,
+ "bmi": 43.1,
+ "pedigree": 2.288,
+ "age": 33,
+ "label": 1
+ }
+ ]
+ },
+ "total_rows": 5,
+ "truncation_type": null
+ },
+ "text/plain": " pregnant glucose bp skin insulin bmi pedigree age label\n0 6 148 72 35 0 33.6 0.627 50 1\n1 1 85 66 29 0 26.6 0.351 31 0\n2 8 183 64 0 0 23.3 0.672 32 1\n3 1 89 66 23 94 28.1 0.167 21 0\n4 0 137 40 35 168 43.1 2.288 33 1",
+ "text/html": "| \n | pregnant | \nglucose | \nbp | \nskin | \ninsulin | \nbmi | \npedigree | \nage | \nlabel | \n
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n6 | \n148 | \n72 | \n35 | \n0 | \n33.6 | \n0.627 | \n50 | \n1 | \n
| 1 | \n1 | \n85 | \n66 | \n29 | \n0 | \n26.6 | \n0.351 | \n31 | \n0 | \n
| 2 | \n8 | \n183 | \n64 | \n0 | \n0 | \n23.3 | \n0.672 | \n32 | \n1 | \n
| 3 | \n1 | \n89 | \n66 | \n23 | \n94 | \n28.1 | \n0.167 | \n21 | \n0 | \n
| 4 | \n0 | \n137 | \n40 | \n35 | \n168 | \n43.1 | \n2.288 | \n33 | \n1 | \n