tutorials/Tutorial_4_Handle_null_targets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## This tutorial shows how to handle NaN targets in multioutput tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "# Optional: set the device to run\n",
    "os.environ[\"CUDA_DEVICE_ORDER\"] = \"PCI_BUS_ID\"\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"1\"\n",
    "\n",
    "os.makedirs('../data', exist_ok=True)\n",
    "import numpy as np\n",
    "import joblib\n",
    "from sklearn.datasets import make_regression\n",
    "\n",
    "# simple case - just one class is used\n",
    "from py_boost import GradientBoosting\n",
    "from py_boost.multioutput.sketching import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate dummy multilabel task with NaN values in target\n",
    "\n",
    "Some times it happends that some target values in multioutput task are missing. For example, you are solving multilabel task and some labels are unknown for some of the rows, so acually your target could be one of 0/1/NaN. Normaly you can not using ML algorithms directly in that case, so you can do one of the following:\n",
    "\n",
    "- Drop NaN rows, but that case you are going to miss some part of the data\n",
    "- Train binary models separately, but your model will be more complex and probably overfitted\n",
    "- Fill NaNs with 0 or 1, so your labeling will become wrong\n",
    "- Use Neural Networks with masked loss function\n",
    "\n",
    "In Py-Boost you can write the loss wrapper to handle such scenario and train your model directly on known labels ignoring NaNs, and here is shown how.\n",
    "\n",
    "We will create it as the regression task and then thresholding the target. And then add some random NaNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.33 s, sys: 1.66 s, total: 3.99 s\n",
      "Wall time: 876 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "X, y = make_regression(150000, 100, n_targets=10, random_state=42)\n",
    "# binarize\n",
    "y = (y > y.mean(axis=0)).astype(np.float32)\n",
    "# add some NaNs\n",
    "y[np.random.rand(150000, 10) > 0.5] = np.nan\n",
    "\n",
    "\n",
    "X_test, y_test = X[:50000], y[:50000]\n",
    "X, y = X[-50000:], y[-50000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NaN loss and metric wrappers\n",
    "\n",
    "Here it is shown how to write loss wrapper ignoring NaNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cupy as cp\n",
    "from py_boost.gpu.losses import BCELoss\n",
    "\n",
    "class BCEWithNaNLoss(BCELoss):\n",
    "    \n",
    "    def base_score(self, y_true):\n",
    "        # Replace .mean with nanmean function to calc base score\n",
    "        means = cp.clip(cp.nanmean(y_true, axis=0), self.clip_value, 1 - self.clip_value)\n",
    "        return cp.log(means / (1 - means))\n",
    "    \n",
    "    def get_grad_hess(self, y_true, y_pred):\n",
    "        # first, get nan mask for y_true\n",
    "        mask = cp.isnan(y_true)\n",
    "        # then, compute loss with any values at nan places just to prevent the exception\n",
    "        grad, hess = super().get_grad_hess(cp.where(mask, 0, y_true), y_pred)\n",
    "        # invert mask\n",
    "        mask = (~mask).astype(cp.float32)\n",
    "        # multiply grad and hess on inverted mask\n",
    "        # now grad and hess eq. 0 on NaN points\n",
    "        # that actually means that prediction on that place should not be updated\n",
    "        grad = grad * mask\n",
    "        hess = hess * mask\n",
    "        \n",
    "        return grad, hess\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here is column-wise roc-auc metric ignoring NaNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from py_boost.gpu.losses.metrics import Metric, auc\n",
    "\n",
    "class NaNAucMetric(Metric):\n",
    "    \n",
    "    def __call__(self, y_true, y_pred, sample_weight=None):\n",
    "        \n",
    "        aucs = []\n",
    "        mask = ~cp.isnan(y_true)\n",
    "        \n",
    "        for i in range(y_true.shape[1]):\n",
    "            m = mask[:, i]\n",
    "            w = None if sample_weight is None else sample_weight[:, 0][m]\n",
    "            aucs.append(\n",
    "                auc(y_true[:, i][m], y_pred[:, i][m], w)\n",
    "            )\n",
    "            \n",
    "        return np.mean(aucs)\n",
    "    \n",
    "    def compare(self, v0 ,v1):\n",
    "\n",
    "        return v0 > v1    \n",
    "            \n",
    "            \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[08:31:38] Stdout logging level is INFO.\n",
      "[08:31:38] GDBT train starts. Max iter 1000, early stopping rounds 200\n",
      "[08:31:39] Iter 0; Sample 0, score = 0.7906884535541213; \n",
      "[08:31:41] Iter 100; Sample 0, score = 0.9687261163054176; \n",
      "[08:31:44] Iter 200; Sample 0, score = 0.9785187659166686; \n",
      "[08:31:46] Iter 300; Sample 0, score = 0.9844858052685057; \n",
      "[08:31:49] Iter 400; Sample 0, score = 0.9883780152591723; \n",
      "[08:31:51] Iter 500; Sample 0, score = 0.9908004122540589; \n",
      "[08:31:54] Iter 600; Sample 0, score = 0.9923353340683694; \n",
      "[08:31:57] Iter 700; Sample 0, score = 0.9935137491384962; \n",
      "[08:31:59] Iter 800; Sample 0, score = 0.9943018456130359; \n",
      "[08:32:02] Iter 900; Sample 0, score = 0.9949417958344802; \n",
      "[08:32:04] Iter 999; Sample 0, score = 0.9954331107999328; \n",
      "CPU times: user 32.1 s, sys: 1.59 s, total: 33.7 s\n",
      "Wall time: 31.9 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<py_boost.gpu.boosting.GradientBoosting at 0x7f5e559de730>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "model = GradientBoosting(BCEWithNaNLoss(), NaNAucMetric(), lr=0.01,\n",
    "                         verbose=100, ntrees=1000, es=200, multioutput_sketch=RandomProjectionSketch(1))\n",
    "\n",
    "model.fit(X, y, eval_sets=[{'X': X_test, 'y': y_test},])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "rapids-22.06",
   "language": "python",
   "name": "rapids-22.06"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}