LASSO_FeatureSelector.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying Lasso to perform feature selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy.stats\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.linear_model import Lasso"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boston dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\91779\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\sklearn\\utils\\deprecation.py:87: FutureWarning: Function load_boston is deprecated; `load_boston` is deprecated in 1.0 and will be removed in 1.2.\n",
      "\n",
      "    The Boston housing prices dataset has an ethical problem. You can refer to\n",
      "    the documentation of this function for further details.\n",
      "\n",
      "    The scikit-learn maintainers therefore strongly discourage the use of this\n",
      "    dataset unless the purpose of the code is to study and educate about\n",
      "    ethical issues in data science and machine learning.\n",
      "\n",
      "    In this special case, you can fetch the dataset from the original\n",
      "    source::\n",
      "\n",
      "        import pandas as pd\n",
      "        import numpy as np\n",
      "\n",
      "\n",
      "        data_url = \"http://lib.stat.cmu.edu/datasets/boston\"\n",
      "        raw_df = pd.read_csv(data_url, sep=\"\\s+\", skiprows=22, header=None)\n",
      "        data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])\n",
      "        target = raw_df.values[1::2, 2]\n",
      "\n",
      "    Alternative datasets include the California housing dataset (i.e.\n",
      "    :func:`~sklearn.datasets.fetch_california_housing`) and the Ames housing\n",
      "    dataset. You can load the datasets as follows::\n",
      "\n",
      "        from sklearn.datasets import fetch_california_housing\n",
      "        housing = fetch_california_housing()\n",
      "\n",
      "    for the California housing dataset and::\n",
      "\n",
      "        from sklearn.datasets import fetch_openml\n",
      "        housing = fetch_openml(name=\"house_prices\", as_frame=True)\n",
      "\n",
      "    for the Ames housing dataset.\n",
      "    \n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "X,y = load_boston(return_X_y=True)\n",
    "features = load_boston()['feature_names']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wine dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_wine\n",
    "X,y = load_wine(return_X_y=True)\n",
    "features = load_wine()['feature_names']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iris dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "X,y = load_iris(return_X_y=True)\n",
    "features = load_iris()['feature_names']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Digits dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "X,y = load_digits(return_X_y=True)\n",
    "features = load_digits()['feature_names']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pipleline for scaling data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline([\n",
    "                     ('scaler',StandardScaler()),\n",
    "                     ('model',Lasso())\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we are going to test several values from 0.1 to 10 with 0.1 step. For each value, we calculate the average value of the mean squared error in a 5-folds cross-validation and select the value of α that minimizes such average performance metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "search = GridSearchCV(pipeline,\n",
    "                      {'model__alpha':np.arange(0.1,10,0.1)},\n",
    "                      cv = 5, scoring=\"neg_mean_squared_error\",verbose=3\n",
    "                      )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use neg_mean_squared_error because the grid search tries to maximize the performance metrics, so we add a minus sign to minimize the mean squared error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 99 candidates, totalling 495 fits\n",
      "[CV 1/5] END .................model__alpha=0.1;, score=-3.629 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=0.1;, score=-3.915 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=0.1;, score=-4.127 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=0.1;, score=-4.161 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=0.1;, score=-4.705 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=0.2;, score=-4.154 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=0.2;, score=-4.421 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=0.2;, score=-4.885 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=0.2;, score=-4.817 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=0.2;, score=-5.045 total time=   0.0s\n",
      "[CV 1/5] END .model__alpha=0.30000000000000004;, score=-4.772 total time=   0.0s\n",
      "[CV 2/5] END .model__alpha=0.30000000000000004;, score=-4.990 total time=   0.0s\n",
      "[CV 3/5] END .model__alpha=0.30000000000000004;, score=-5.643 total time=   0.0s\n",
      "[CV 4/5] END .model__alpha=0.30000000000000004;, score=-5.619 total time=   0.0s\n",
      "[CV 5/5] END .model__alpha=0.30000000000000004;, score=-5.489 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=0.4;, score=-5.288 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=0.4;, score=-5.394 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=0.4;, score=-6.143 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=0.4;, score=-6.188 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=0.4;, score=-5.947 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=0.5;, score=-5.774 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=0.5;, score=-5.829 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=0.5;, score=-6.639 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=0.5;, score=-6.741 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=0.5;, score=-6.435 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=0.6;, score=-6.379 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=0.6;, score=-6.334 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=0.6;, score=-7.169 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=0.6;, score=-7.275 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=0.6;, score=-7.046 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=0.7000000000000001;, score=-6.821 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=0.7000000000000001;, score=-6.755 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=0.7000000000000001;, score=-7.638 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=0.7000000000000001;, score=-7.641 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=0.7000000000000001;, score=-7.598 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=0.8;, score=-7.126 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=0.8;, score=-7.117 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=0.8;, score=-7.992 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=0.8;, score=-7.909 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=0.8;, score=-7.857 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=0.9;, score=-7.357 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=0.9;, score=-7.348 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=0.9;, score=-8.225 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=0.9;, score=-8.069 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=0.9;, score=-8.022 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=1.0;, score=-7.555 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=1.0;, score=-7.554 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=1.0;, score=-8.448 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=1.0;, score=-8.208 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=1.0;, score=-8.210 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=1.1;, score=-7.772 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=1.1;, score=-7.778 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=1.1;, score=-8.693 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=1.1;, score=-8.366 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=1.1;, score=-8.419 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=1.2000000000000002;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=1.2000000000000002;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=1.2000000000000002;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=1.2000000000000002;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=1.2000000000000002;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=1.3000000000000003;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=1.3000000000000003;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=1.3000000000000003;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=1.3000000000000003;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=1.3000000000000003;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=1.4000000000000001;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=1.4000000000000001;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=1.4000000000000001;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=1.4000000000000001;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=1.4000000000000001;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=1.5000000000000002;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=1.5000000000000002;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=1.5000000000000002;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=1.5000000000000002;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=1.5000000000000002;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=1.6;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=1.6;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=1.6;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=1.6;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=1.6;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=1.7000000000000002;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=1.7000000000000002;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=1.7000000000000002;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=1.7000000000000002;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=1.7000000000000002;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=1.8000000000000003;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=1.8000000000000003;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=1.8000000000000003;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=1.8000000000000003;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=1.8000000000000003;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=1.9000000000000001;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=1.9000000000000001;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=1.9000000000000001;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=1.9000000000000001;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=1.9000000000000001;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=2.0;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=2.0;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=2.0;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=2.0;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=2.0;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=2.1;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=2.1;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=2.1;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=2.1;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=2.1;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=2.2;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=2.2;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=2.2;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=2.2;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=2.2;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=2.3000000000000003;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=2.3000000000000003;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=2.3000000000000003;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=2.3000000000000003;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=2.3000000000000003;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=2.4000000000000004;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=2.4000000000000004;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=2.4000000000000004;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=2.4000000000000004;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=2.4000000000000004;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=2.5000000000000004;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=2.5000000000000004;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=2.5000000000000004;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=2.5000000000000004;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=2.5000000000000004;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=2.6;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=2.6;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=2.6;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=2.6;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=2.6;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=2.7;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=2.7;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=2.7;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=2.7;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=2.7;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=2.8000000000000003;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=2.8000000000000003;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=2.8000000000000003;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=2.8000000000000003;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=2.8000000000000003;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=2.9000000000000004;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=2.9000000000000004;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=2.9000000000000004;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=2.9000000000000004;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=2.9000000000000004;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=3.0000000000000004;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=3.0000000000000004;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=3.0000000000000004;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=3.0000000000000004;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=3.0000000000000004;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=3.1;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=3.1;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=3.1;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=3.1;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=3.1;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=3.2;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=3.2;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=3.2;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=3.2;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=3.2;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=3.3000000000000003;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=3.3000000000000003;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=3.3000000000000003;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=3.3000000000000003;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=3.3000000000000003;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=3.4000000000000004;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=3.4000000000000004;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=3.4000000000000004;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=3.4000000000000004;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=3.4000000000000004;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=3.5000000000000004;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=3.5000000000000004;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=3.5000000000000004;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=3.5000000000000004;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=3.5000000000000004;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=3.6;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=3.6;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=3.6;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=3.6;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=3.6;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=3.7;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=3.7;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=3.7;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=3.7;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=3.7;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=3.8000000000000003;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=3.8000000000000003;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=3.8000000000000003;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=3.8000000000000003;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=3.8000000000000003;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=3.9000000000000004;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=3.9000000000000004;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=3.9000000000000004;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=3.9000000000000004;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=3.9000000000000004;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=4.0;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=4.0;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=4.0;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=4.0;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=4.0;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=4.1;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=4.1;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=4.1;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=4.1;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=4.1;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=4.2;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=4.2;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=4.2;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=4.2;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=4.2;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=4.3;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=4.3;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=4.3;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=4.3;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=4.3;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ..model__alpha=4.3999999999999995;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ..model__alpha=4.3999999999999995;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ..model__alpha=4.3999999999999995;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ..model__alpha=4.3999999999999995;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ..model__alpha=4.3999999999999995;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=4.5;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=4.5;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=4.5;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=4.5;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=4.5;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=4.6;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=4.6;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=4.6;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=4.6;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=4.6;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=4.7;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=4.7;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=4.7;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=4.7;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=4.7;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=4.8;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=4.8;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=4.8;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=4.8;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=4.8;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=4.9;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=4.9;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=4.9;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=4.9;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=4.9;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=5.0;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=5.0;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=5.0;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=5.0;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=5.0;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=5.1;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=5.1;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=5.1;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=5.1;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=5.1;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=5.2;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=5.2;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=5.2;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=5.2;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=5.2;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=5.3;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=5.3;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=5.3;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=5.3;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=5.3;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=5.4;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=5.4;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=5.4;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=5.4;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=5.4;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=5.5;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=5.5;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=5.5;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=5.5;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=5.5;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=5.6;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=5.6;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=5.6;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=5.6;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=5.6;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=5.7;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=5.7;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=5.7;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=5.7;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=5.7;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=5.8;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=5.8;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=5.8;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=5.8;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=5.8;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=5.9;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=5.9;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=5.9;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=5.9;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=5.9;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=6.0;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=6.0;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=6.0;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=6.0;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=6.0;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=6.1;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=6.1;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=6.1;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=6.1;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=6.1;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=6.2;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=6.2;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=6.2;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=6.2;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=6.2;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=6.3;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=6.3;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=6.3;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=6.3;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=6.3;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=6.4;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=6.4;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=6.4;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=6.4;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=6.4;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=6.5;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=6.5;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=6.5;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=6.5;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=6.5;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=6.6;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=6.6;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=6.6;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=6.6;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=6.6;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=6.7;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=6.7;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=6.7;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=6.7;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=6.7;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=6.8;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=6.8;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=6.8;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=6.8;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=6.8;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=6.9;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=6.9;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=6.9;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=6.9;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=6.9;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=7.0;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=7.0;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=7.0;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=7.0;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=7.0;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=7.1;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=7.1;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=7.1;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=7.1;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=7.1;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=7.2;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=7.2;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=7.2;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=7.2;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=7.2;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=7.3;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=7.3;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=7.3;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=7.3;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=7.3;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=7.4;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=7.4;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=7.4;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=7.4;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=7.4;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=7.5;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=7.5;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=7.5;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=7.5;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=7.5;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=7.6;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=7.6;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=7.6;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=7.6;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=7.6;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=7.7;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=7.7;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=7.7;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=7.7;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=7.7;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=7.8;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=7.8;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=7.8;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=7.8;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=7.8;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=7.9;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=7.9;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=7.9;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=7.9;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=7.9;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=8.0;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=8.0;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=8.0;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=8.0;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=8.0;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=8.1;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=8.1;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=8.1;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=8.1;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=8.1;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=8.2;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=8.2;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=8.2;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=8.2;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=8.2;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=8.3;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=8.3;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=8.3;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=8.3;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=8.3;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=8.4;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=8.4;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=8.4;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=8.4;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=8.4;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=8.5;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=8.5;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=8.5;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=8.5;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=8.5;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=8.6;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=8.6;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=8.6;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=8.6;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=8.6;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=8.7;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=8.7;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=8.7;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=8.7;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=8.7;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=8.8;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=8.8;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=8.8;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=8.8;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=8.8;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=8.9;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=8.9;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=8.9;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=8.9;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=8.9;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=9.0;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=9.0;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=9.0;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=9.0;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=9.0;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=9.1;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=9.1;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=9.1;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=9.1;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=9.1;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=9.2;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=9.2;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=9.2;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=9.2;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=9.2;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=9.3;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=9.3;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=9.3;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=9.3;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=9.3;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=9.4;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=9.4;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=9.4;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=9.4;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=9.4;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=9.5;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=9.5;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=9.5;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=9.5;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=9.5;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=9.6;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=9.6;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=9.6;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=9.6;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=9.6;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END ...model__alpha=9.700000000000001;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END ...model__alpha=9.700000000000001;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END ...model__alpha=9.700000000000001;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END ...model__alpha=9.700000000000001;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END ...model__alpha=9.700000000000001;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=9.8;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=9.8;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=9.8;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=9.8;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=9.8;, score=-8.516 total time=   0.0s\n",
      "[CV 1/5] END .................model__alpha=9.9;, score=-7.827 total time=   0.0s\n",
      "[CV 2/5] END .................model__alpha=9.9;, score=-7.791 total time=   0.0s\n",
      "[CV 3/5] END .................model__alpha=9.9;, score=-8.715 total time=   0.0s\n",
      "[CV 4/5] END .................model__alpha=9.9;, score=-8.498 total time=   0.0s\n",
      "[CV 5/5] END .................model__alpha=9.9;, score=-8.516 total time=   0.0s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[('scaler', StandardScaler()),\n",
       "                                       ('model', Lasso())]),\n",
       "             param_grid={'model__alpha': array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2, 1.3,\n",
       "       1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. , 2.1, 2.2, 2.3, 2.4, 2.5, 2.6,\n",
       "       2.7, 2.8, 2.9, 3. , 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9,\n",
       "       4. , 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9, 5. , 5.1, 5.2,\n",
       "       5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 5.9, 6. , 6.1, 6.2, 6.3, 6.4, 6.5,\n",
       "       6.6, 6.7, 6.8, 6.9, 7. , 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8,\n",
       "       7.9, 8. , 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7, 8.8, 8.9, 9. , 9.1,\n",
       "       9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9])},\n",
       "             scoring='neg_mean_squared_error', verbose=3)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best value for α can be found using,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best value for alpha is =  0.100000\n"
     ]
    }
   ],
   "source": [
    "best_alhpa = search.best_params_\n",
    "print(\"Best value for alpha is = \", '%.6f'%float(best_alhpa['model__alpha']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have to get the values of the coefficients of Lasso regression. The importance of a feature is the absolute value of its coefficient,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.         0.         0.         0.05752486 0.02467769 0.\n",
      " 0.         0.02555828 0.         0.         0.16912822 0.\n",
      " 0.26492136 0.         0.14839312 0.         0.         0.\n",
      " 0.41873888 0.         0.39662162 0.         0.         0.\n",
      " 0.         0.28950904 0.         0.45674386 0.35150967 0.409443\n",
      " 0.         0.         0.         0.25354802 0.         0.58469506\n",
      " 0.         0.17711474 0.         0.         0.         0.0044046\n",
      " 0.         0.         0.33296506 0.20888257 0.         0.\n",
      " 0.         0.         0.         0.22205225 0.96016851 0.19538036\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.10861776 0.01063974 0.         0.14104189]\n"
     ]
    }
   ],
   "source": [
    "coefficients = search.best_estimator_.named_steps['model'].coef_\n",
    "importance = np.abs(coefficients)\n",
    "print(importance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set a threshold value to allow us to control the level of impactfulness required from nthe features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the set threshold,\n",
    "\n",
    "The features that survived the Lasso regression are,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['pixel_0_3', 'pixel_0_4', 'pixel_0_7', 'pixel_1_2', 'pixel_1_4',\n",
       "       'pixel_1_6', 'pixel_2_2', 'pixel_2_4', 'pixel_3_1', 'pixel_3_3',\n",
       "       'pixel_3_4', 'pixel_3_5', 'pixel_4_1', 'pixel_4_3', 'pixel_4_5',\n",
       "       'pixel_5_1', 'pixel_5_4', 'pixel_5_5', 'pixel_6_3', 'pixel_6_4',\n",
       "       'pixel_6_5', 'pixel_7_4', 'pixel_7_5', 'pixel_7_7'], dtype='<U9')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(features)[importance > threshold]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The features that did not survive the Lasso regression are,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['pixel_0_0', 'pixel_0_1', 'pixel_0_2', 'pixel_0_5', 'pixel_0_6',\n",
       "       'pixel_1_0', 'pixel_1_1', 'pixel_1_3', 'pixel_1_5', 'pixel_1_7',\n",
       "       'pixel_2_0', 'pixel_2_1', 'pixel_2_3', 'pixel_2_5', 'pixel_2_6',\n",
       "       'pixel_2_7', 'pixel_3_0', 'pixel_3_2', 'pixel_3_6', 'pixel_3_7',\n",
       "       'pixel_4_0', 'pixel_4_2', 'pixel_4_4', 'pixel_4_6', 'pixel_4_7',\n",
       "       'pixel_5_0', 'pixel_5_2', 'pixel_5_3', 'pixel_5_6', 'pixel_5_7',\n",
       "       'pixel_6_0', 'pixel_6_1', 'pixel_6_2', 'pixel_6_6', 'pixel_6_7',\n",
       "       'pixel_7_0', 'pixel_7_1', 'pixel_7_2', 'pixel_7_3', 'pixel_7_6'],\n",
       "      dtype='<U9')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(features)[importance <= threshold]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying correlation to do the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[      nan 6.570e-02 1.340e-02 3.380e-02 7.950e-02 1.835e-01 2.178e-01\n",
      " 1.182e-01 1.700e-03 3.130e-02 9.490e-02 1.034e-01 2.618e-01 4.660e-02\n",
      " 2.326e-01 9.120e-02 2.700e-02 4.860e-02 7.910e-02 1.086e-01 1.575e-01\n",
      " 9.280e-02 9.910e-02 3.840e-02 2.730e-02 1.601e-01 6.890e-02 2.758e-01\n",
      " 2.199e-01 1.863e-01 3.190e-02 6.000e-03       nan 2.057e-01 1.600e-02\n",
      " 2.847e-01 1.826e-01 1.435e-01 3.690e-02       nan 1.040e-02 1.143e-01\n",
      " 1.020e-01 6.500e-03 2.000e-04 1.950e-02 5.550e-02 4.980e-02 1.290e-02\n",
      " 2.600e-02 1.472e-01 1.832e-01 3.938e-01 2.070e-01 2.550e-02 7.480e-02\n",
      " 2.430e-02 5.940e-02 1.600e-03 3.640e-02 2.140e-01 7.660e-02 1.011e-01\n",
      " 1.720e-01]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\91779\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\scipy\\stats\\stats.py:4023: PearsonRConstantInputWarning: An input array is constant; the correlation coefficient is not defined.\n",
      "  warnings.warn(PearsonRConstantInputWarning())\n"
     ]
    }
   ],
   "source": [
    "r_col = y_train\n",
    "n_ft = len(X_train[0])\n",
    "n_cof = [None]*n_ft\n",
    "\n",
    "for i in range(n_ft):\n",
    "    i_col = df = X_train[:,i]\n",
    "    n_cof[i] = scipy.stats.pearsonr(r_col, i_col)[0]\n",
    "\n",
    "importance_corr = np.abs(np.round(n_cof,4))\n",
    "\n",
    "print(importance_corr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "selecting some small number to check if the most significant are always being selected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_imp = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding the highest impactful features wrt correlation coefficient values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   nan    nan    nan 0.3938 0.2847]\n",
      "[   nan    nan    nan 0.3938 0.2847]\n"
     ]
    }
   ],
   "source": [
    "imp_arr = importance_corr*[importance > threshold]\n",
    "rel_imp = imp_arr[np.nonzero(imp_arr)]\n",
    "\n",
    "print(np.sort(importance_corr)[::-1][:n_imp])\n",
    "\n",
    "print(np.sort(rel_imp)[::-1][:n_imp])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.4 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "6bf3ebb51ba865f281ce1f84b3745d1f53cddfef84166cfb0862af6341610cdf"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}