ray_tune_micro_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "source": [
    "# Micro tutorial on how to run and scale hyperparameter optimization with LightGBM and Tune\n",
    "<img src=\"https://docs.ray.io/en/latest/_images/tune_overview.png\" alt=\"Tune and integrations\" width=\"500\">\n",
    "\n",
    "Aug 2022. San Francisco, CA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: single LightGBM training session\n",
    "<img src=\"https://lightgbm.readthedocs.io/en/latest/_images/LightGBM_logo_black_text.svg\" alt=\"LightGBM Logo\" width=\"500\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[LightGBM](https://lightgbm.readthedocs.io) is a gradient boosting framework that uses tree based learning algorithms. It has Python API for model training and evaluation. Trained model can be inspected in multiple ways including visualizations like feature importance or trees plotting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preliminaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import lightgbm as lgb\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare dataset\n",
    "X, y = load_digits(return_X_y=True)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.2, random_state=7707)\n",
    "\n",
    "train_data = lgb.Dataset(data=X_train, label=y_train, free_raw_data=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we use [digits dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html) (classification) and create LightGBM Dataset object that will be used for training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set training parameters for single training run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_parameters = {\n",
    "    \"objective\": \"multiclass\",\n",
    "    \"metric\": \"multi_logloss\",\n",
    "    \"num_class\": 10,\n",
    "    \"num_leaves\": 5,          # max number of leaves in one tree\n",
    "    \"learning_rate\": 0.001,   # boosting learning rate\n",
    "    \"feature_fraction\": 0.5,  # fraction of features on each iteration\n",
    "    \"bagging_fraction\": 0.5,  # like \"feature_fraction\", but this will randomly select part of data without resampling\n",
    "    \"bagging_freq\": 50,       # frequency for bagging\n",
    "    \"max_depth\": 2,           # max depth of the tree\n",
    "    \"verbose\": -1,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize and train LightGBM model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize booster\n",
    "gbm = lgb.Booster(params=training_parameters, train_set=train_data)\n",
    "\n",
    "# Train booster for 200 iterations\n",
    "for i in range(200):\n",
    "    gbm = lgb.train(\n",
    "        params=training_parameters,\n",
    "        train_set=train_data,\n",
    "        num_boost_round=1,\n",
    "        init_model=gbm,\n",
    "        keep_training_booster=True,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Report accuracy on validation data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = np.argmax(gbm.predict(X_valid), axis=1)\n",
    "acc = accuracy_score(y_true=y_valid, y_pred=y_pred)\n",
    "print(f\"Accuracy on valid set: {acc:.4f}, after {gbm.current_iteration()} iterations.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "We just ran single LightGBM training session. To do that we prepared dataset and training hyperparameters.\n",
    "\n",
    "#### Next\n",
    "Let's have a closer look at Tune."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Part 2: Tune quickstart\n",
    "<img src=\"https://docs.ray.io/en/latest/_images/tune.png\" alt=\"Tune logo\" width=\"500\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction to Tune\n",
    "#### Key concepts\n",
    "\n",
    "<img src=\"https://docs.ray.io/en/latest/_images/tune_flow.png\" alt=\"Tune key concepts\" width=\"800\">\n",
    "\n",
    "Learn more about it from the [Key concepts](https://docs.ray.io/en/latest/tune/key-concepts.html) docs page.\n",
    "\n",
    "#### Scaling of the tuning jobs\n",
    "\n",
    "<img src=\"https://miro.medium.com/max/700/0*EZKV8RTgDt0NfL49\" alt=\"scaling\" width=\"600\">\n",
    "\n",
    "Learn more from the Richard Liaw et al. [paper](https://arxiv.org/abs/1807.05118) introducing Tune."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Ray cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ray\n",
    "\n",
    "if ray.is_initialized:\n",
    "    ray.shutdown()\n",
    "cluster_info = ray.init(num_cpus=8)\n",
    "cluster_info.address_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `ray.init()` starts Ray runtime on a single machine. By default it will utilize all cores available on the machine. Here, we parametrized it to use `num_cpus=8`.\n",
    "* Check [configuring ray](https://docs.ray.io/en/latest/ray-core/configure.html#configuring-ray) page for more in depth analysis of available options.\n",
    "* This runtime will be used for all tuning jobs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Tune"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ray import tune"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define search space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_space = {\n",
    "    \"objective\": \"multiclass\",\n",
    "    \"metric\": \"multi_logloss\",\n",
    "    \"num_class\": 10,\n",
    "    \"num_leaves\": tune.choice([2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 100]),\n",
    "    \"learning_rate\": tune.loguniform(1e-4, 1e-1),\n",
    "    \"feature_fraction\": tune.uniform(0.5, 0.999),\n",
    "    \"bagging_fraction\": 0.5,\n",
    "    \"bagging_freq\": tune.randint(1, 50),\n",
    "    \"max_depth\": tune.randint(1, 11),\n",
    "    \"verbose\": -1,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice that you can freely mix tune functions for defining search space (i.e. `tune.randint(1, 11)`) with fixed values (i.e. `\"num_class\": 10`).\n",
    "* [Search space API](https://docs.ray.io/en/latest/tune/api_docs/search_space.html) has variety of functions that you can use to define your search space in a way that suits your needs. Function used above are just few examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_lgbm(training_params, checkpoint_dir=None):\n",
    "    train_data = lgb.Dataset(data=X_train, label=y_train, free_raw_data=False)\n",
    "\n",
    "    # Initialize booster\n",
    "    gbm = lgb.Booster(params=training_params, train_set=train_data)\n",
    "\n",
    "    # Train booster for 200 iterations\n",
    "    for i in range(200):\n",
    "        gbm = lgb.train(\n",
    "            params=training_params,\n",
    "            train_set=train_data,\n",
    "            num_boost_round=1,\n",
    "            init_model=gbm,\n",
    "            keep_training_booster=True,\n",
    "        )\n",
    "\n",
    "        y_pred = np.argmax(gbm.predict(X_valid), axis=1)\n",
    "        acc = accuracy_score(y_true=y_valid, y_pred=y_pred)\n",
    "\n",
    "        # Send accuracy back to Tune\n",
    "        tune.report(valid_acc=acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Trainable (`train_lgbm`) is a function that will be evaluated multiples times during tuning.\n",
    "* LightGBM model training logic is the same as in the \"vanilla\" example above.\n",
    "* It is executed on a separate Ray Actor (process), so we need to communicate the performance of the model back to Tune (which is on the main Python process). Here, `tune.report()` comes into play - it sends the performance value back to Tune. In this case it is `acc`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run hyperparameter tuning, single trial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analysis = tune.run(train_lgbm, config=search_space)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* When you call `tune.run()`, the trainable (`train_lgbm`) is evaluated with hyperparameters sampled from the search space (`search_space`).\n",
    "* Tune handles sampling and executing the trainable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display info about this trial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = analysis.dataframe(metric=\"valid_acc\")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "We just ran tuning job with Tune 🚀.\n",
    "\n",
    "#### Key concepts in this section\n",
    "* Search space\n",
    "* Trainable\n",
    "* Trial\n",
    "\n",
    "#### Key API elements in this section\n",
    "* `ray.init()` -> start ray runtime.\n",
    "* `tune.report()` -> log the performance values. Called in the trainable function.\n",
    "* `tune.run()` -> execute tuning.\n",
    "\n",
    "#### Next\n",
    "We will modify `tune.run()` in order to run tuning with 100 trials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Execute 100 tuning runs with Tune"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run hyperparameter tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analysis = tune.run(\n",
    "    train_lgbm,\n",
    "    config=search_space,\n",
    "    num_samples=100,\n",
    "    metric=\"valid_acc\",\n",
    "    resources_per_trial={\"cpu\": 1},\n",
    "    verbose=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* When `tune.run()` is called, trainable (`train_lgbm`) is evaluated `num_samples` times (100 trials) in parallel (subject to available compute resources).\n",
    "* Each trial has hyperparameters sampled from the search space (`search_space`).\n",
    "* Tune handles parallel execution, sampling from the search space and collecting the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display info about best trials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = analysis.dataframe(metric=\"valid_acc\")\n",
    "df.sort_values(by=[\"valid_acc\"], ascending=False).head(n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optionally you can use parallel coordinates plot to visualise results from all tuning runs. You can use [Plotly](https://plotly.com/python/parallel-coordinates-plot/) or [HiPlot](https://github.com/facebookresearch/hiplot)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "We optimized hyperparameters by executing 100 tuning trials.\n",
    "\n",
    "#### Key API elements in this section\n",
    "* `tune.run(num_samples=...)` -> specify number of trials.\n",
    "\n",
    "#### Next\n",
    "We will introduce `scheduler` to early stop unpromising trials and as a result save compute time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: ASHA with Tune"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction to ASHA (Asynchronous Successive Halving Algorithm)\n",
    "<img src=\"https://lh4.googleusercontent.com/E6KJ-5KQgfYVleJEXxaldICsEXm-dRUlsiD9AFbckXov0uaYfnIBKskLT6z1eLfptdKjxTCF05LBAz0W9evXbyWAViA5qYFGOaIYCuoz-h9n8rluHkl3ZOj-0IPKrdA4ES34Ybpo\" alt=\"synchronous promotions\" width=\"1000\">\n",
    "\n",
    "<img src=\"https://lh6.googleusercontent.com/ncYQXlFoVzhEsun2I-0LfTySEySc-uwEAd2vdPXGHvwprwXApuHuU4o17uJ1ITgHw9_sxId0995xOdfs-r7K3lWB4QQ7v9s33GnBs-EZ7cECIqj9Cq_eDQapJSAEG6P6A0oLZxm6\" alt=\"asynchronous promotions\" width=\"1000\">\n",
    "\n",
    "* Promote configurations whenever possible, hence utilize resources.\n",
    "* Asynchronous SHA utilizes resources efficiently. Workers are always busy by expanding the base rung if no configurations can be promoted to higher rungs.\n",
    "* Read more about ASHA in the CMU ML [blogpost](https://blog.ml.cmu.edu/2018/12/12/massively-parallel-hyperparameter-optimization/).\n",
    "\n",
    "_(Visualization is from the same [blogpost](https://blog.ml.cmu.edu/2018/12/12/massively-parallel-hyperparameter-optimization/). Date accessed: 2022.08.04)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import ASHA from Tune schedulers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ray.tune.schedulers import ASHAScheduler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create ASHA scheduler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "asha = ASHAScheduler(\n",
    "    time_attr=\"training_iteration\",\n",
    "    mode=\"max\",\n",
    "    grace_period=50,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run hyperparameter tuning with ASHA scheduler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analysis = tune.run(\n",
    "    train_lgbm,\n",
    "    config=search_space,\n",
    "    num_samples=100,\n",
    "    metric=\"valid_acc\",\n",
    "    resources_per_trial={\"cpu\": 1},\n",
    "    scheduler=asha,\n",
    "    verbose=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display info about best trials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = analysis.dataframe(metric=\"valid_acc\")\n",
    "df.sort_values(by=[\"valid_acc\"], ascending=False).head(n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "We ran hyperparameter tuning with 100 trials. ASHA scheduler terminated unpromising trials early. Saving compute resources.\n",
    "\n",
    "#### Key concepts in this section\n",
    "* Scheduler\n",
    "* Early stopping (of the unpromising trials)\n",
    "\n",
    "#### Key API elements in this section\n",
    "* `ASHAScheduler` -> [Async Successive Halving](https://docs.ray.io/en/latest/tune/api_docs/schedulers.html#asha-tune-schedulers-ashascheduler) scheduler.\n",
    "* `tune.run(scheduler=...)` -> specify scheduler to use for tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shutdown Ray runtime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ray.shutdown()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Disconnect the worker, and terminate processes started by `ray.init()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where to go next?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congrats!\n",
    "\n",
    "You just finished the micro tutorial on how to run and scale hyperparameter optimization with LightGBM and Tune.\n",
    "\n",
    "Now, please go to the [micro tutorial README](https://github.com/kamil-kaczmarek/ray-tune-micro-tutorial/blob/kk/dev/README.md), to learn more about next steps, and options to reach out and connect with the community."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}