examples/XGBoost_on_California_housing_prices_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "36bcbf97",
   "metadata": {},
   "source": [
    "# New Relic ML Performance Monitoring - Bring Your Own Data\n",
    "\n",
    "[ml-performance-monitoring](https://github.com/newrelic-experimental/ml-performance-monitoring) provides a Python library for sending machine learning models' inference data and performance metrics into New Relic. \n",
    "<br>\n",
    "By using this package, you can easily and quickly monitor your model, directly from a Jupyter notebook or a cloud service. \n",
    "<br>\n",
    "The package is ML framework agnostic and can be quickly integrated. It is based on the newrelic-telemetry-sdk-python library.\n",
    "<br>\n",
    "It is based on the [newrelic-telemetry-sdk-python](https://github.com/newrelic/newrelic-telemetry-sdk-python) library.\n",
    "\n",
    "\n",
    "This notebook provides an example of sending inference data and metrics of an XGBoost model\n",
    "\n",
    "<U>Note</U>- this notebook uses the libraries:\n",
    "* numpy\n",
    "* pandas\n",
    "* sklearn\n",
    "* xgboost"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77eda1c8",
   "metadata": {},
   "source": [
    "### 0. Install libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e984c34f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install git+https://github.com/newrelic-experimental/ml-performance-monitoring.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "41d4b494",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install pandas scikit-learn xgboost"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e65b6a8",
   "metadata": {},
   "source": [
    "### 1. Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e2275faa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import xgboost as xgb\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "from ml_performance_monitoring.monitor import MLPerformanceMonitoring"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5829e9a7",
   "metadata": {},
   "source": [
    "### 2. Load the California housing prices dataset and split it into train and test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "06308232",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "california_dataset = fetch_california_housing()\n",
    "X, y = (\n",
    "    california_dataset[\"data\"],\n",
    "    california_dataset[\"target\"],\n",
    ")\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=123\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f4f21208-2123-423b-a14c-78d6a88c3076",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3.51000e-02 9.50000e+01 2.68000e+00 0.00000e+00 4.16100e-01 7.85300e+00\n",
      "  3.32000e+01 5.11800e+00 4.00000e+00 2.24000e+02 1.47000e+01 3.92780e+02\n",
      "  3.81000e+00]\n",
      " [9.72418e+00 0.00000e+00 1.81000e+01 0.00000e+00 7.40000e-01 6.40600e+00\n",
      "  9.72000e+01 2.06510e+00 2.40000e+01 6.66000e+02 2.02000e+01 3.85960e+02\n",
      "  1.95200e+01]\n",
      " [1.39140e-01 0.00000e+00 4.05000e+00 0.00000e+00 5.10000e-01 5.57200e+00\n",
      "  8.85000e+01 2.59610e+00 5.00000e+00 2.96000e+02 1.66000e+01 3.96900e+02\n",
      "  1.46900e+01]\n",
      " [1.22040e-01 0.00000e+00 2.89000e+00 0.00000e+00 4.45000e-01 6.62500e+00\n",
      "  5.78000e+01 3.49520e+00 2.00000e+00 2.76000e+02 1.80000e+01 3.57980e+02\n",
      "  6.65000e+00]\n",
      " [1.36000e-02 7.50000e+01 4.00000e+00 0.00000e+00 4.10000e-01 5.88800e+00\n",
      "  4.76000e+01 7.31970e+00 3.00000e+00 4.69000e+02 2.11000e+01 3.96900e+02\n",
      "  1.48000e+01]] [48.5 17.1 23.1 28.4 18.9]\n"
     ]
    }
   ],
   "source": [
    "print(X_train[:5], y_train[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e6a6dfd",
   "metadata": {},
   "source": [
    "### 3. Fitting XGBoost regression model\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2107be0d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBRegressor(alpha=10, base_score=0.5, booster='gbtree', callbacks=None,\n",
       "             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.3,\n",
       "             early_stopping_rounds=None, enable_categorical=False,\n",
       "             eval_metric=None, gamma=0, gpu_id=-1, grow_policy='depthwise',\n",
       "             importance_type=None, interaction_constraints='',\n",
       "             learning_rate=0.1, max_bin=256, max_cat_to_onehot=4,\n",
       "             max_delta_step=0, max_depth=5, max_leaves=0, min_child_weight=1,\n",
       "             missing=nan, monotone_constraints='()', n_estimators=10, n_jobs=0,\n",
       "             num_parallel_tree=1, predictor='auto', random_state=0,\n",
       "             reg_alpha=10, ...)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xg_reg = xgb.XGBRegressor(\n",
    "    objective=\"reg:squarederror\",\n",
    "    colsample_bytree=0.3,\n",
    "    learning_rate=0.1,\n",
    "    max_depth=5,\n",
    "    alpha=10,\n",
    "    n_estimators=10,\n",
    ")\n",
    "xg_reg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4531cc2b",
   "metadata": {},
   "source": [
    "### 4. Predicting the test set results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f505f688",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 9.107417 , 17.003168 , 23.554592 , 11.127839 , 18.847708 ,\n",
       "       14.83594  , 17.920063 ,  7.8613553, 12.147531 , 18.064953 ,\n",
       "       18.074614 , 14.411147 , 11.660895 , 15.656607 , 14.011221 ,\n",
       "       14.031893 , 14.247164 , 23.057888 , 13.79595  , 11.450888 ,\n",
       "       12.218057 , 15.310874 , 20.628437 , 23.554592 , 15.16144  ,\n",
       "       13.139177 , 11.691522 , 15.382896 , 15.382896 , 11.007488 ,\n",
       "       14.734173 , 18.862646 ,  8.471523 , 15.349405 , 15.463427 ,\n",
       "       19.77869  , 15.875313 ,  9.95701  , 12.919248 , 23.60624  ,\n",
       "       18.295853 , 12.589657 , 14.179869 , 21.97658  , 12.664099 ,\n",
       "       16.489174 , 14.038502 , 15.660043 , 12.919248 , 14.453838 ,\n",
       "       18.862646 , 17.252022 , 14.038502 ,  7.9073224, 14.332668 ,\n",
       "       10.871179 , 10.826269 ,  8.5319   , 19.534363 ,  8.196141 ,\n",
       "       12.403829 , 14.235118 , 10.42806  , 13.162136 , 14.247164 ,\n",
       "       15.931118 , 16.565668 ,  9.804545 , 14.915205 , 18.295853 ,\n",
       "       13.15519  , 15.318869 , 12.919248 , 16.430174 , 11.854328 ,\n",
       "       12.32626  , 11.756507 , 14.426096 , 18.501059 ,  9.180772 ,\n",
       "       19.065006 ,  8.588221 , 15.463427 , 14.011221 , 12.898891 ,\n",
       "       15.079189 , 11.267364 , 15.272911 , 14.247164 , 19.482258 ,\n",
       "        9.210783 , 18.699503 , 13.79595  , 15.878302 , 19.92443  ,\n",
       "       17.05043  , 11.450888 , 20.330242 , 16.072117 , 20.87532  ,\n",
       "       14.714855 , 11.429612 ], dtype=float32)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = xg_reg.predict(X_test)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fddebfbb",
   "metadata": {},
   "source": [
    "### 5. Record inference data to New Relic\n",
    "\n",
    "The MLPerformanceMonitoring parameters: \n",
    "   * Required parameters:\n",
    "      * `model_name` - must be unique per model\n",
    "      *  `insert_key` - [Get your key](https://one.newrelic.com/launcher/api-keys-ui.api-keys-launcher) (also referenced as `ingest - license`) and set it as environment variable: `NEW_RELIC_LICENSE_KEY`.\n",
    "[Click here](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key) for more details and instructions.\n",
    "\n",
    "* Optional parameters:\n",
    "   * `metadata` (dictionary) - will be added to each event (row) of the data \n",
    "   * `send_data_metrics` (boolean) - send data metrics (statistics) to New Relic (False as default)\n",
    "   * `features_columns`(list) - the features' names ordered as X columns.\n",
    "   * `labels_columns` (list) - the labels' names ordered as y columns. \n",
    "\n",
    "(note: The parameters `features_columns` and `labels_columns` are only relevant when sending the data as an np.array. When the data is sent as a dataframe, the dataframes (X,y) columns' names will be taken as features and labels names respectively. In addition, if you send your data as an np.array without sending the features_columns and labels_columns, on New Relic data, the names will appear as \"feature_{n}\" and \"lablel_{n}\" numbered by the features/labels order)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cee39519",
   "metadata": {},
   "source": [
    "5.1. Define monitoring parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "532ae6a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = {\"environment\": \"notebook\", \"dataset\": \"California housing prices\"}\n",
    "model_version = \"1.0\"\n",
    "features_columns, labels_columns = (\n",
    "    list(california_dataset[\"feature_names\"]),\n",
    "    [\"target\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54400202-8548-466d-bdd7-c435b604d16b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "print(\"You can get your License-key here: https://one.newrelic.com/launcher/api-keys-ui.api-keys-launcher\")\n",
    "insert_key = getpass.getpass(\"Paste an License-key and hit enter:\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7534e68c",
   "metadata": {},
   "source": [
    "5.2 Create model monitor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3aba1d3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_monitor = MLPerformanceMonitoring(\n",
    "    insert_key=insert_key,  # set the environment variable NEW_RELIC_LICENSE_KEY or send your license-key here\n",
    "    model_name=\"XGBoost Regression on California housing Dataset\",\n",
    "    metadata=metadata,\n",
    "    send_data_metrics=True,\n",
    "    features_columns=features_columns,\n",
    "    labels_columns=labels_columns,\n",
    "    model_version=model_version,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3264a3a2",
   "metadata": {},
   "source": [
    "5.3 Send your data as an np.array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48f8ba42",
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_monitor.record_inference_data(X=X_test, y=y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dea28209",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "5.4  Send your data as a pd.DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "49231f6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51.13580</td>\n",
       "      <td>0.0</td>\n",
       "      <td>18.10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5970</td>\n",
       "      <td>5.757</td>\n",
       "      <td>100.0</td>\n",
       "      <td>1.4130</td>\n",
       "      <td>24.0</td>\n",
       "      <td>666.0</td>\n",
       "      <td>20.2</td>\n",
       "      <td>2.60</td>\n",
       "      <td>10.11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.05735</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.49</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4490</td>\n",
       "      <td>6.630</td>\n",
       "      <td>56.1</td>\n",
       "      <td>4.4377</td>\n",
       "      <td>3.0</td>\n",
       "      <td>247.0</td>\n",
       "      <td>18.5</td>\n",
       "      <td>392.30</td>\n",
       "      <td>6.53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.03578</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.33</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4429</td>\n",
       "      <td>7.820</td>\n",
       "      <td>64.5</td>\n",
       "      <td>4.6947</td>\n",
       "      <td>5.0</td>\n",
       "      <td>216.0</td>\n",
       "      <td>14.9</td>\n",
       "      <td>387.31</td>\n",
       "      <td>3.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12.04820</td>\n",
       "      <td>0.0</td>\n",
       "      <td>18.10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6140</td>\n",
       "      <td>5.648</td>\n",
       "      <td>87.6</td>\n",
       "      <td>1.9512</td>\n",
       "      <td>24.0</td>\n",
       "      <td>666.0</td>\n",
       "      <td>20.2</td>\n",
       "      <td>291.55</td>\n",
       "      <td>14.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.03150</td>\n",
       "      <td>95.0</td>\n",
       "      <td>1.47</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4030</td>\n",
       "      <td>6.975</td>\n",
       "      <td>15.3</td>\n",
       "      <td>7.6534</td>\n",
       "      <td>3.0</td>\n",
       "      <td>402.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.56</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       CRIM    ZN  INDUS  CHAS     NOX     RM    AGE     DIS   RAD    TAX  \\\n",
       "0  51.13580   0.0  18.10   0.0  0.5970  5.757  100.0  1.4130  24.0  666.0   \n",
       "1   0.05735   0.0   4.49   0.0  0.4490  6.630   56.1  4.4377   3.0  247.0   \n",
       "2   0.03578  20.0   3.33   0.0  0.4429  7.820   64.5  4.6947   5.0  216.0   \n",
       "3  12.04820   0.0  18.10   0.0  0.6140  5.648   87.6  1.9512  24.0  666.0   \n",
       "4   0.03150  95.0   1.47   0.0  0.4030  6.975   15.3  7.6534   3.0  402.0   \n",
       "\n",
       "   PTRATIO       B  LSTAT  \n",
       "0     20.2    2.60  10.11  \n",
       "1     18.5  392.30   6.53  \n",
       "2     14.9  387.31   3.76  \n",
       "3     20.2  291.55  14.10  \n",
       "4     17.0  396.90   4.56  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_df = pd.DataFrame(\n",
    "    list(map(np.ravel, X_test)),\n",
    "    columns=features_columns,\n",
    ")\n",
    "\n",
    "y_pred_df = pd.DataFrame(\n",
    "    list(map(np.ravel, y_pred)),\n",
    "    columns=labels_columns,\n",
    ")\n",
    "X_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00ef6592",
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_monitor.record_inference_data(X=X_df, y=y_pred_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c7e59ebc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9.107417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>17.003168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>23.554592</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11.127839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>18.847708</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      target\n",
       "0   9.107417\n",
       "1  17.003168\n",
       "2  23.554592\n",
       "3  11.127839\n",
       "4  18.847708"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45c5f850",
   "metadata": {},
   "source": [
    "### 6. Record metrics to New Relic\n",
    "You can stream custom metrics to New Relic, monitoring your model performance or model data. These metrics will be sent to NRDB as [metric data](https://docs.newrelic.com/docs/data-apis/ingest-apis/metric-api/introduction-metric-api/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c2f04d4b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 10.517\n"
     ]
    }
   ],
   "source": [
    "rmse = round(np.sqrt(mean_squared_error(y_test, y_pred)), 3)\n",
    "print(f\"RMSE: {rmse}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a50c4fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics = {\n",
    "    \"RMSE\": rmse,\n",
    "}\n",
    "ml_monitor.record_metrics(metrics=metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bde8388",
   "metadata": {},
   "source": [
    "### 7. Monitor and alert\n",
    "Done! Check your application in the [New Relic UI](https://one.newrelic.com/nr1-core?filters=%28domain%20%3D%20%27MLOPS%27%20AND%20type%20%3D%20%27MACHINE_LEARNING_MODEL%27%29) to see the real time data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}