tutorial/tutorial01-Shapash-Overview-Launch-WebApp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Start the Shapash Web App on a sample dataset\n",
    "\n",
    "<b>With this tutorial you:</b><br />\n",
    "Understand how shapash works with a simple use case<br />\n",
    "Start WebApp to understand your model and save these results\n",
    "\n",
    "Contents:\n",
    "- Build a Regressor\n",
    "- Compile Shapash SmartExplainer\n",
    "- Start Shapash WebApp\n",
    "- Export synt with to_pandas function\n",
    "- Save Shapash object in pickle file\n",
    "\n",
    "Data from Kaggle [House Prices](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from category_encoders import OrdinalEncoder\n",
    "from lightgbm import LGBMRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import ExtraTreesRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Supervized Model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from shapash.data.data_loader import data_loading\n",
    "house_df, house_dict = data_loading('house_prices')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_df=house_df['SalePrice'].to_frame()\n",
    "X_df=house_df[house_df.columns.difference(['SalePrice'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1460, 73)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "house_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>LandSlope</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>...</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2-Story 1946 &amp; Newer</td>\n",
       "      <td>Residential Low Density</td>\n",
       "      <td>8450</td>\n",
       "      <td>Paved</td>\n",
       "      <td>Regular</td>\n",
       "      <td>Near Flat/Level</td>\n",
       "      <td>All public Utilities (E,G,W,&amp; S)</td>\n",
       "      <td>Inside lot</td>\n",
       "      <td>Gentle slope</td>\n",
       "      <td>College Creek</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>Warranty Deed - Conventional</td>\n",
       "      <td>Normal Sale</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1-Story 1946 &amp; Newer All Styles</td>\n",
       "      <td>Residential Low Density</td>\n",
       "      <td>9600</td>\n",
       "      <td>Paved</td>\n",
       "      <td>Regular</td>\n",
       "      <td>Near Flat/Level</td>\n",
       "      <td>All public Utilities (E,G,W,&amp; S)</td>\n",
       "      <td>Frontage on 2 sides of property</td>\n",
       "      <td>Gentle slope</td>\n",
       "      <td>Veenker</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>Warranty Deed - Conventional</td>\n",
       "      <td>Normal Sale</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2-Story 1946 &amp; Newer</td>\n",
       "      <td>Residential Low Density</td>\n",
       "      <td>11250</td>\n",
       "      <td>Paved</td>\n",
       "      <td>Slightly irregular</td>\n",
       "      <td>Near Flat/Level</td>\n",
       "      <td>All public Utilities (E,G,W,&amp; S)</td>\n",
       "      <td>Inside lot</td>\n",
       "      <td>Gentle slope</td>\n",
       "      <td>College Creek</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>Warranty Deed - Conventional</td>\n",
       "      <td>Normal Sale</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2-Story 1945 &amp; Older</td>\n",
       "      <td>Residential Low Density</td>\n",
       "      <td>9550</td>\n",
       "      <td>Paved</td>\n",
       "      <td>Slightly irregular</td>\n",
       "      <td>Near Flat/Level</td>\n",
       "      <td>All public Utilities (E,G,W,&amp; S)</td>\n",
       "      <td>Corner lot</td>\n",
       "      <td>Gentle slope</td>\n",
       "      <td>Crawford</td>\n",
       "      <td>...</td>\n",
       "      <td>272</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>Warranty Deed - Conventional</td>\n",
       "      <td>Abnormal Sale</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2-Story 1946 &amp; Newer</td>\n",
       "      <td>Residential Low Density</td>\n",
       "      <td>14260</td>\n",
       "      <td>Paved</td>\n",
       "      <td>Slightly irregular</td>\n",
       "      <td>Near Flat/Level</td>\n",
       "      <td>All public Utilities (E,G,W,&amp; S)</td>\n",
       "      <td>Frontage on 2 sides of property</td>\n",
       "      <td>Gentle slope</td>\n",
       "      <td>Northridge</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>Warranty Deed - Conventional</td>\n",
       "      <td>Normal Sale</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 73 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                         MSSubClass                 MSZoning  LotArea Street  \\\n",
       "Id                                                                             \n",
       "1              2-Story 1946 & Newer  Residential Low Density     8450  Paved   \n",
       "2   1-Story 1946 & Newer All Styles  Residential Low Density     9600  Paved   \n",
       "3              2-Story 1946 & Newer  Residential Low Density    11250  Paved   \n",
       "4              2-Story 1945 & Older  Residential Low Density     9550  Paved   \n",
       "5              2-Story 1946 & Newer  Residential Low Density    14260  Paved   \n",
       "\n",
       "              LotShape      LandContour                         Utilities  \\\n",
       "Id                                                                          \n",
       "1              Regular  Near Flat/Level  All public Utilities (E,G,W,& S)   \n",
       "2              Regular  Near Flat/Level  All public Utilities (E,G,W,& S)   \n",
       "3   Slightly irregular  Near Flat/Level  All public Utilities (E,G,W,& S)   \n",
       "4   Slightly irregular  Near Flat/Level  All public Utilities (E,G,W,& S)   \n",
       "5   Slightly irregular  Near Flat/Level  All public Utilities (E,G,W,& S)   \n",
       "\n",
       "                          LotConfig     LandSlope   Neighborhood  ...  \\\n",
       "Id                                                                ...   \n",
       "1                        Inside lot  Gentle slope  College Creek  ...   \n",
       "2   Frontage on 2 sides of property  Gentle slope        Veenker  ...   \n",
       "3                        Inside lot  Gentle slope  College Creek  ...   \n",
       "4                        Corner lot  Gentle slope       Crawford  ...   \n",
       "5   Frontage on 2 sides of property  Gentle slope     Northridge  ...   \n",
       "\n",
       "   EnclosedPorch 3SsnPorch ScreenPorch PoolArea  MiscVal  MoSold  YrSold  \\\n",
       "Id                                                                         \n",
       "1              0         0           0        0        0       2    2008   \n",
       "2              0         0           0        0        0       5    2007   \n",
       "3              0         0           0        0        0       9    2008   \n",
       "4            272         0           0        0        0       2    2006   \n",
       "5              0         0           0        0        0      12    2008   \n",
       "\n",
       "                        SaleType  SaleCondition SalePrice  \n",
       "Id                                                         \n",
       "1   Warranty Deed - Conventional    Normal Sale    208500  \n",
       "2   Warranty Deed - Conventional    Normal Sale    181500  \n",
       "3   Warranty Deed - Conventional    Normal Sale    223500  \n",
       "4   Warranty Deed - Conventional  Abnormal Sale    140000  \n",
       "5   Warranty Deed - Conventional    Normal Sale    250000  \n",
       "\n",
       "[5 rows x 73 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "house_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Encoding Categorical Features "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from category_encoders import OrdinalEncoder\n",
    "\n",
    "categorical_features = [col for col in X_df.columns if X_df[col].dtype == 'object']\n",
    "\n",
    "encoder = OrdinalEncoder(\n",
    "    cols=categorical_features,\n",
    "    handle_unknown='ignore',\n",
    "    return_df=True).fit(X_df)\n",
    "\n",
    "X_df=encoder.transform(X_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train / Test Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "Xtrain, Xtest, ytrain, ytest = train_test_split(X_df, y_df, train_size=0.75, random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Fitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "regressor = LGBMRegressor(n_estimators=100).fit(Xtrain,ytrain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding my model with shapash"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Declare and Compile SmartExplainer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from shapash import SmartExplainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "xpl = SmartExplainer(\n",
    "    model=regressor,\n",
    "    preprocessing=encoder,   # Optional: compile step can use inverse_transform method\n",
    "    features_dict=house_dict # optional parameter, specifies label for features name \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "xpl.compile(x=Xtest,\n",
    "            y_target=ytest # Optional: allows to display True Values vs Predicted Values\n",
    "           )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Start WebApp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "app = xpl.run_app(title_story='House Prices', port=8020)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Link to App: <br />\n",
    "[shapash-monitor link](https://shapash-demo.ossbymaif.fr/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stop the WebApp after using it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "app.kill()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Export local explaination in DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_df= xpl.to_pandas(\n",
    "    max_contrib=3, # Number Max of features to show in summary\n",
    "    threshold=5000,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pred</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>value_1</th>\n",
       "      <th>contribution_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>value_2</th>\n",
       "      <th>contribution_2</th>\n",
       "      <th>feature_3</th>\n",
       "      <th>value_3</th>\n",
       "      <th>contribution_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>259</th>\n",
       "      <td>203102.618265</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>1792</td>\n",
       "      <td>10170.153594</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>7</td>\n",
       "      <td>9886.60162</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>268</th>\n",
       "      <td>165504.066858</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>5</td>\n",
       "      <td>-21896.320133</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>2192</td>\n",
       "      <td>16807.388625</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>289</th>\n",
       "      <td>141844.323422</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>5</td>\n",
       "      <td>-20785.923401</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>900</td>\n",
       "      <td>-10577.685</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>650</th>\n",
       "      <td>116849.365350</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>4</td>\n",
       "      <td>-27677.524884</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>630</td>\n",
       "      <td>-12140.106966</td>\n",
       "      <td>Total square feet of basement area</td>\n",
       "      <td>630</td>\n",
       "      <td>-7142.980699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1234</th>\n",
       "      <td>160989.488908</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>5</td>\n",
       "      <td>-20986.378284</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>1188</td>\n",
       "      <td>-8761.318312</td>\n",
       "      <td>Total square feet of basement area</td>\n",
       "      <td>1188</td>\n",
       "      <td>5591.086195</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               pred                                 feature_1 value_1  \\\n",
       "259   203102.618265            Ground living area square feet    1792   \n",
       "268   165504.066858  Overall material and finish of the house       5   \n",
       "289   141844.323422  Overall material and finish of the house       5   \n",
       "650   116849.365350  Overall material and finish of the house       4   \n",
       "1234  160989.488908  Overall material and finish of the house       5   \n",
       "\n",
       "     contribution_1                                 feature_2 value_2  \\\n",
       "259    10170.153594  Overall material and finish of the house       7   \n",
       "268   -21896.320133            Ground living area square feet    2192   \n",
       "289   -20785.923401            Ground living area square feet     900   \n",
       "650   -27677.524884            Ground living area square feet     630   \n",
       "1234  -20986.378284            Ground living area square feet    1188   \n",
       "\n",
       "     contribution_2                           feature_3 value_3 contribution_3  \n",
       "259      9886.60162                                 NaN     NaN            NaN  \n",
       "268    16807.388625                                 NaN     NaN            NaN  \n",
       "289      -10577.685                                 NaN     NaN            NaN  \n",
       "650   -12140.106966  Total square feet of basement area     630   -7142.980699  \n",
       "1234   -8761.318312  Total square feet of basement area    1188    5591.086195  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save SmartExplainer in Pickle File\n",
    "\n",
    "You can save the SmartExplainer Object in a pickle file to make new plots later or launch the WebApp again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "xpl.save('./xpl.pkl')"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Aucun(e)",
  "hide_input": false,
  "kernelspec": {
   "display_name": "shapash_picking",
   "language": "python",
   "name": "shapash_picking"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}