xgboost-training/xgboost-training/xgboost-training.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://ubiops.com/wp-content/uploads/2020/12/Group-2.svg\" title=\"UbiOps Logo\" width=100px/>\n",
    "\n",
    "# Applying hyperparameter tuning on an XGBoost model\n",
    "\n",
    "In this example, we will show you how you can initiate multiple training runs that train an XGBoost model with different hyperparameter combinations. We do this by creating an `environment` in which our training job can run. Then we will \n",
    "define a `train.py` script that we can apply to our environment. The training script is based on the XGBoost tutorial, \n",
    "where the [kc_house_data](https://www.kaggle.com/datasets/shivachandel/kc-house-data) dataset is used to train an XGBoost \n",
    "model that predicts house prices. We will initiate several runs using different sets of hyperparameters. After all the \n",
    "runs have been completed, we will explain how you can look at the results using the WebApp.\n",
    "\n",
    "\n",
    "The output of the script is a trained XGBoost model (`xgboost_model.joblib`) and the the accuracy (`xgboost_score`) of \n",
    "the model. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set project variables and initialize UbiOps API Client\n",
    "First, make sure you create an **[API token](https://ubiops.com/docs/organizations/service-users/)** with `project editor` permissions in your UbiOps project and paste it below. Also, fill in your corresponding UbiOps project name. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade ubiops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "import yaml\n",
    "import os\n",
    "import ubiops\n",
    "\n",
    "API_TOKEN = '<INSERT API_TOKEN WITH PROJECT EDITOR RIGHTS>' # Make sure this is in the format \"Token token-code\"\n",
    "PROJECT_NAME = '<INSERT PROJECT NAME IN YOUR ACCOUNT>'\n",
    "BUCKET_NAME = 'default'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "configuration = ubiops.Configuration(host=\"https://api.ubiops.com/v2.1\")\n",
    "configuration.api_key['Authorization'] = API_TOKEN\n",
    "\n",
    "api_client = ubiops.ApiClient(configuration)\n",
    "core_instance = ubiops.CoreApi(api_client=api_client)\n",
    "training_instance = ubiops.Training(api_client=api_client)\n",
    "print(core_instance.service_status())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set-up a training instance in case you have not done this yet in your project. This action will create a base training deployment, that is used to host training experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_instance = ubiops.Training(api_client=api_client)\n",
    "try:\n",
    "    training_instance.initialize(project_name=PROJECT_NAME)\n",
    "except ubiops.exceptions.ApiException as e:\n",
    "    print(f\"The training feature may already have been initialized in your project:\\n{e}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make the training environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_environment_dir = 'training_environment'\n",
    "ENVIRONMENT_NAME = 'xgboost-training-env'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%mkdir {training_environment_dir}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile {training_environment_dir}/requirements.txt\n",
    "pandas==1.5.2\n",
    "scikit-learn==1.0.2\n",
    "scipy==1.10.0\n",
    "xgboost==1.3.1\n",
    "ubiops==3.9.0\n",
    "fsspec==2022.1.0\n",
    "joblib\n",
    "pathlib"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now zip the environment like you would zip a deployment package, and create an environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil \n",
    "training_environment_archive = shutil.make_archive(f'{training_environment_dir}', 'zip', '.', f'{training_environment_dir}')\n",
    "\n",
    "# Create experiment. Your environment is set-up in this step. It may take some time to run.\n",
    "\n",
    "try:\n",
    "    api_response = core_instance.environments_create(\n",
    "        project_name=PROJECT_NAME,\n",
    "        data=ubiops.EnvironmentCreate(\n",
    "        name=ENVIRONMENT_NAME,\n",
    "        display_name= 'XGBoost training',\n",
    "        base_environment='python3-10',\n",
    "        description='XGboost training',\n",
    "        )\n",
    "    )\n",
    "    \n",
    "    core_instance.environment_revisions_file_upload(\n",
    "        project_name=PROJECT_NAME,\n",
    "        environment_name=ENVIRONMENT_NAME,\n",
    "        file=training_environment_archive\n",
    "    )\n",
    "except ubiops.exceptions.ApiException as e:\n",
    "    print(e)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure an experiment\n",
    "\n",
    "You can run experiments in your new environment. The experiments can help segment series of training runs, and run on one dedicated instance type. You can perform multiple runs in parallel in your experiment.\n",
    "\n",
    "In this example, note that you are required to have a bucket inside your project. This bucket will be used to store your training jobs and model callbacks. In case you want to continue without [creating a bucket](https://github.com/UbiOps/client-library-python/blob/master/docs/Files.md#buckets_creates), you can use the `default` bucket. This bucket is always automatically generated for every project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET_NAME = 'default'\n",
    "EXPERIMENT_NAME = 'xgboost-training-tutorial'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    experiment = training_instance.experiments_create(\n",
    "        project_name=PROJECT_NAME,\n",
    "        data=ubiops.ExperimentCreate(\n",
    "            instance_type='4096mb',\n",
    "            description='Train test experiment',\n",
    "            name=EXPERIMENT_NAME,\n",
    "            environment=ENVIRONMENT_NAME,\n",
    "            default_bucket= BUCKET_NAME\n",
    "        )\n",
    "    )\n",
    "except ubiops.exceptions.ApiException as e:\n",
    "    print(e)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Then create the training runs\n",
    "Now that we have our training experiment set-up, we can initiate training runs. For this example we will initiate parallel\n",
    "training runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "RUN_NAME = 'training-run'\n",
    "RUN_SCRIPT = f'{RUN_NAME}.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile {RUN_SCRIPT}\n",
    "\n",
    "import pandas as pd\n",
    "import xgboost\n",
    "import math\n",
    "import os\n",
    "import ubiops\n",
    "import joblib\n",
    "import pathlib\n",
    "from scipy.stats import pearsonr\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn import tree, linear_model\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import explained_variance_score\n",
    "\n",
    "\n",
    "def train(training_data, parameters, context):\n",
    "\n",
    "    # All code inside this function will run when a call to the deployment is made.\n",
    "    # Read the data into a data frame\n",
    "    \n",
    "    data = pd.read_csv(training_data)\n",
    "\n",
    "    print(\"Data loaded \")\n",
    "\n",
    "    new_data = data[['sqft_living','grade', 'sqft_above', 'sqft_living15','bathrooms','view','sqft_basement','lat','waterfront','yr_built','bedrooms']].values\n",
    "    X = new_data\n",
    "    print('X loaded in')\n",
    "    target_data = data[['price']]\n",
    "    y = target_data.values\n",
    "    print(\"splitting data\")\n",
    "   \n",
    "    # Create train test sets\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y ,test_size=0.2)\n",
    "    print('training model')\n",
    "    \n",
    "    # Set up the parameters\n",
    "    n_est = parameters['n_estimators']\n",
    "    le_ra = parameters['learning_rate']\n",
    "    subsam = parameters['subsample']\n",
    "\n",
    "    xgb = xgboost.XGBRegressor(n_estimators = n_est,learning_rate = le_ra, gamma = 0, subsample = subsam,\n",
    "                                colsample_bytree = 1, max_depth = 7)\n",
    "\n",
    "    print('parameters have been setup')\n",
    "\n",
    "    # Train the model\n",
    "    xgb.fit(X_train,y_train)\n",
    "    print('model_trained')\n",
    "\n",
    "    # Make predictions using the xgboost model\n",
    "    predictions = xgb.predict(X_test)\n",
    "    print('predictions made')\n",
    "\n",
    "    # Check how the xgboost model scores on accuracy on our test set\n",
    "    xgboost_score = explained_variance_score(predictions,y_test)\n",
    "\n",
    "    print(f'Score of the xgboost model {xgboost_score}')\n",
    "\n",
    "    # Save the model\n",
    "    joblib.dump(xgb, 'xgboost_model.pkl') \n",
    "    print('XGBoost model built and saved successfully!')\n",
    "\n",
    "    return {\n",
    "        'artifact': 'xgboost_model.pkl',\n",
    "        'metrics': {'xgboost_score': xgboost_score}\n",
    "    }"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training data\n",
    "\n",
    "For this example we will download the training dataset locally, so we can show you how you can use a local dataset in a training run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "\n",
    "url = 'https://storage.googleapis.com/ubiops/data/Deploying%20with%20popular%20DS%20libraries/xgboost_tutorial/kc_house_data.csv'\n",
    "training_data= 'kc_house_data.csv'\n",
    "\n",
    "urllib.request.urlretrieve(url, training_data)\n",
    "\n",
    "print(f\"File downloaded successfully to '{training_data}'.\")\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the parameters\n",
    "\n",
    "As shown in the `train.py`, this model uses six parameters. For simplicity we will only apply the hyperparameters to three\n",
    "of those parameters. \n",
    "\n",
    "After the run is completed you can navigate to the `Training` tab, click the `Evaluation` button, select the three runs \n",
    "we completed, click the `Compare runs` button, and compare the results. The metrics `n_estimators`, `learning_rate`, and \n",
    "`subsample` from all three runs with different sets of hyperparameters, can then be compared with eachother. Here, we \n",
    "notice that the second set of parameters (`\"n_estimators\": 150, \"learning_rate\": 0.12, \"subsample\": 0.75`) achieves the \n",
    "highest score.\n",
    " \n",
    "Alternatively, you can go to the experiment of which you want to compare runs (in this case the xgboost-training\n",
    "-tutorial), select all the runs you want to compare by checking the boxes and then click on the `Compare runs` button."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_parameters = [\n",
    "    {\n",
    "    \"n_estimators\": 100,\n",
    "    \"learning_rate\": 0.08,\n",
    "    \"subsample\": 0.5\n",
    "    },\n",
    "     {\n",
    "    \"n_estimators\": 150,\n",
    "    \"learning_rate\": 0.12,\n",
    "    \"subsample\": 0.75\n",
    "    },\n",
    "    {\n",
    "    \"n_estimators\": 200,\n",
    "    \"learning_rate\": 0.16,\n",
    "    \"subsample\": 1\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, run_parameters in enumerate(run_parameters):\n",
    "    run_template = ubiops.ExperimentRunCreate(\n",
    "            name=f\"run{i}estimators{run_parameters['n_estimators']}_learning_rate{run_parameters['learning_rate']}\",\n",
    "            description='Trying out a first run run with ',\n",
    "            training_code= RUN_SCRIPT,\n",
    "            training_data= training_data, #path to data\n",
    "            parameters= run_parameters\n",
    "    )\n",
    "    training_instance.experiment_runs_create(\n",
    "        project_name=PROJECT_NAME,\n",
    "        experiment_name=EXPERIMENT_NAME,\n",
    "        data= run_template,\n",
    "        timeout=14400\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping up\n",
    "\n",
    "Now all that is left to do is to close the connection to the UbiOps API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_client.close()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And there you have it! We have successfully created a set-up where we can easily try out different sets of hyperparameters to train an XGBoost model! If you want you can check out the full [Jupyter Notebook](ttps://download-github.ubiops.com/#!/home?url=https://github.com/UbiOps/tutorials/tree/master/xgboost-training/xgboost-training/xgboost-training.ipynb), fill in your API token and project name, and run it yourself to upload it to your own UbiOps environment."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.1"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}