examples/quick-starts/optimization/XGBoostBayesianOptimizer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# `optimization.XGBoostBayesianOptimizer`"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "import slickml\n",
    "\n",
    "print(f\"Loaded SlickML Version = {slickml.__version__}\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Loaded SlickML Version = 0.2.0\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "from slickml.optimization import XGBoostBayesianOptimizer\n",
    "\n",
    "help(XGBoostBayesianOptimizer)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Help on class XGBoostBayesianOptimizer in module slickml.optimization._bayesianopt:\n",
      "\n",
      "class XGBoostBayesianOptimizer(slickml.base._estimator.BaseXGBoostEstimator)\n",
      " |  XGBoostBayesianOptimizer(num_boost_round: Optional[int] = 200, sparse_matrix: Optional[bool] = False, scale_mean: Optional[bool] = False, scale_std: Optional[bool] = False, importance_type: Optional[str] = 'total_gain', params: Optional[Dict[str, Union[str, float, int]]] = None, n_iter: Optional[int] = 10, n_init_iter: Optional[int] = 5, n_splits: Optional[int] = 4, metrics: Optional[str] = 'auc', objective: Optional[str] = 'binary:logistic', acquisition_criterion: Optional[str] = 'ei', params_bounds: Optional[Dict[str, Tuple[Union[int, float], Union[int, float]]]] = None, early_stopping_rounds: Optional[int] = 20, stratified: Optional[bool] = True, shuffle: Optional[bool] = True, random_state: Optional[int] = 1367, verbose: Optional[bool] = True) -> None\n",
      " |  \n",
      " |  XGBoost Hyper-Parameters Tuner using Bayesian Optimization.\n",
      " |  \n",
      " |  This is wrapper using Bayesian Optimization algorithm [bayesian-optimization]_ to tune the\n",
      " |  hyper-parameter of XGBoost [xgboost-api]_ using ``xgboost.cv()`` functionality with n-folds\n",
      " |  cross-validation iteratively. This feature can be used to find the set of optimized set of\n",
      " |  hyper-parameters for both classification and regression tasks.\n",
      " |  \n",
      " |  Notes\n",
      " |  -----\n",
      " |  The optimizier objective is always to maximize the target values. Therefore, in case of using a\n",
      " |  metric such as ``logloss``, ``error``, ``mae``, ``rmse``, or ``rmsle``, the negative value of the\n",
      " |  metric will be maximized. One of the big pitfall of the current implementation is the way we are\n",
      " |  sampling hyper-parameters from the ``params_bounds`` where we are looking for an integer which\n",
      " |  is not possible. Therefore, for some of cases i.e. ``max_depth`` we must cast the sampled value\n",
      " |  which is mathematically wrong (i.e. ``f(1.1) != f(1)``).\n",
      " |  \n",
      " |  Parameters\n",
      " |  ----------\n",
      " |  n_iter : int, optional\n",
      " |      Number of iteration rounds for hyper-parameters tuning after initialization, by default 10\n",
      " |  \n",
      " |  n_init_iter : int, optional\n",
      " |      Number of initial iterations to initialize the optimizer, by default 5\n",
      " |  \n",
      " |  n_splits : int, optional\n",
      " |      Number of folds for cross-validation, by default 4\n",
      " |  \n",
      " |  metrics : str, optional\n",
      " |      Metrics to be tracked at cross-validation fitting time depends on the task\n",
      " |      (classification vs regression) with possible values of \"auc\", \"aucpr\", \"error\", \"logloss\",\n",
      " |      \"rmse\", \"rmsle\", \"mae\". Note this is different than `eval_metric` that needs to be passed to\n",
      " |      `params` dict, by default \"auc\"\n",
      " |  \n",
      " |  objective : str, optional\n",
      " |      Objective function depending on the task whether it is regression or classification. Possible\n",
      " |      objectives for classification ``\"binary:logistic\"`` and for regression ``\"reg:logistic\"``,\n",
      " |      ``\"reg:squarederror\"``, and ``\"reg:squaredlogerror\"``, by default \"binary:logistic\"\n",
      " |  \n",
      " |  acquisition_criterion : str, optional\n",
      " |      Acquisition criterion method with possible options of ``\"ei\"`` (Expected Improvement),\n",
      " |      ``\"ucb\"`` (Upper Confidence Bounds), and ``\"poi\"`` (Probability Of Improvement), by default \"ei\"\n",
      " |  \n",
      " |  params_bounds : Dict[str, Tuple[Union[int, float], Union[int, float]]], optional\n",
      " |      Set of hyper-parameters boundaries for Bayesian Optimization where all fields are required,\n",
      " |      by default {\"max_depth\" : (2, 7), \"learning_rate\" : (0, 1), \"min_child_weight\" : (1, 20),\n",
      " |      \"colsample_bytree\": (0.1, 1.0), \"subsample\" : (0.1, 1), \"gamma\" : (0, 1),\n",
      " |      \"reg_alpha\" : (0, 1), \"reg_lambda\" : (0, 1)}\n",
      " |  \n",
      " |  num_boost_round : int, optional\n",
      " |      Number of boosting rounds to fit a model, by default 200\n",
      " |  \n",
      " |  early_stopping_rounds : int, optional\n",
      " |      The criterion to early abort the ``xgboost.cv()`` phase if the test metric is not improved,\n",
      " |      by default 20\n",
      " |  \n",
      " |  random_state : int, optional\n",
      " |      Random seed number, by default 1367\n",
      " |  \n",
      " |  stratified : bool, optional\n",
      " |      Whether to use stratificaiton of the targets (only available for classification tasks) to run\n",
      " |      ``xgboost.cv()`` to find the best number of boosting round at each fold of each iteration,\n",
      " |      by default True\n",
      " |  \n",
      " |  shuffle : bool, optional\n",
      " |      Whether to shuffle data to have the ability of building stratified folds in ``xgboost.cv()``,\n",
      " |      by default True\n",
      " |  \n",
      " |  sparse_matrix : bool, optional\n",
      " |      Whether to convert the input features to sparse matrix with csr format or not. This would\n",
      " |      increase the speed of feature selection for relatively large/sparse datasets. Consequently,\n",
      " |      this would actually act like an un-optimize solution for dense feature matrix. Additionally,\n",
      " |      this parameter cannot be used along with ``scale_mean=True`` standardizing the feature matrix\n",
      " |      to have a mean value of zeros would turn the feature matrix into a dense matrix. Therefore,\n",
      " |      by default our API banned this feature, by default False\n",
      " |  \n",
      " |  scale_mean : bool, optional\n",
      " |      Whether to standarize the feauture matrix to have a mean value of zero per feature (center\n",
      " |      the features before scaling). As laid out in ``sparse_matrix``, ``scale_mean=False`` when\n",
      " |      using ``sparse_matrix=True``, since centering the feature matrix would decrease the sparsity\n",
      " |      and in practice it does not make any sense to use sparse matrix method and it would make\n",
      " |      it worse. The ``StandardScaler`` object can be accessed via ``cls.scaler_`` if ``scale_mean`` or\n",
      " |      ``scale_strd`` is used unless it is ``None``, by default False\n",
      " |  \n",
      " |  scale_std : bool, optional\n",
      " |      Whether to scale the feauture matrix to have unit variance (or equivalently, unit standard\n",
      " |      deviation) per feature. The ``StandardScaler`` object can be accessed via ``cls.scaler_``\n",
      " |      if ``scale_mean`` or ``scale_strd`` is used unless it is ``None``, by default False\n",
      " |  \n",
      " |  importance_type : str, optional\n",
      " |      Importance type of ``xgboost.train()`` with possible values ``\"weight\"``, ``\"gain\"``,\n",
      " |      ``\"total_gain\"``, ``\"cover\"``, ``\"total_cover\"``, by default \"total_gain\"\n",
      " |  \n",
      " |  verbose : bool, optional\n",
      " |      Whether to show the Bayesian Optimization progress at each iteration, by default True\n",
      " |  \n",
      " |  Methods\n",
      " |  -------\n",
      " |  fit(X, y)\n",
      " |      Fits the Bayesian optimization algorithm to tune the hyper-parameters\n",
      " |  \n",
      " |  get_optimizer()\n",
      " |      Returns the fitted Bayesian Optimiziation object\n",
      " |  \n",
      " |  get_results()\n",
      " |      Returns all the optimization results including target and params\n",
      " |  \n",
      " |  get_best_results()\n",
      " |      Return the results based on the best (tuned) hyper-parameters\n",
      " |  \n",
      " |  get_best_params()\n",
      " |      Returns the tuned hyper-parameters as a dictionary\n",
      " |  \n",
      " |  get_params_bounds()\n",
      " |      Returns the parameters boundaries\n",
      " |  \n",
      " |  Attributes\n",
      " |  ----------\n",
      " |  optimizer_ :\n",
      " |      Returns the fitted Bayesian Optimiziation object\n",
      " |  \n",
      " |  results_\n",
      " |      Returns all the optimization results including target and params\n",
      " |  \n",
      " |  best_params_\n",
      " |      Returns the tuned hyper-parameters as a dictionary\n",
      " |  \n",
      " |  best_results_\n",
      " |      Return the results based on the best (tuned) hyper-parameters\n",
      " |  \n",
      " |  References\n",
      " |  ----------\n",
      " |  .. [xgboost-api] https://xgboost.readthedocs.io/en/latest/python/python_api.html\n",
      " |  .. [bayesian-optimization] https://github.com/fmfn/BayesianOptimization\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      XGBoostBayesianOptimizer\n",
      " |      slickml.base._estimator.BaseXGBoostEstimator\n",
      " |      abc.ABC\n",
      " |      sklearn.base.BaseEstimator\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __eq__(self, other)\n",
      " |  \n",
      " |  __init__(self, num_boost_round: Optional[int] = 200, sparse_matrix: Optional[bool] = False, scale_mean: Optional[bool] = False, scale_std: Optional[bool] = False, importance_type: Optional[str] = 'total_gain', params: Optional[Dict[str, Union[str, float, int]]] = None, n_iter: Optional[int] = 10, n_init_iter: Optional[int] = 5, n_splits: Optional[int] = 4, metrics: Optional[str] = 'auc', objective: Optional[str] = 'binary:logistic', acquisition_criterion: Optional[str] = 'ei', params_bounds: Optional[Dict[str, Tuple[Union[int, float], Union[int, float]]]] = None, early_stopping_rounds: Optional[int] = 20, stratified: Optional[bool] = True, shuffle: Optional[bool] = True, random_state: Optional[int] = 1367, verbose: Optional[bool] = True) -> None\n",
      " |  \n",
      " |  __post_init__(self) -> None\n",
      " |      Post instantiation validations and assignments.\n",
      " |  \n",
      " |  __repr__(self)\n",
      " |  \n",
      " |  fit(self, X: Union[pandas.core.frame.DataFrame, numpy.ndarray], y: Union[List[float], numpy.ndarray, pandas.core.series.Series]) -> None\n",
      " |      Fits the main hyper-parameter tuning algorithm.\n",
      " |      \n",
      " |      Notes\n",
      " |      -----\n",
      " |      At each iteration, one set of parameters gets passed from the `params_bounds` and the\n",
      " |      evaluation occurs based on the cross-validation results. Bayesian optimizier always\n",
      " |      maximizes the objectives. Therefore, based on the `metrics` we should be careful\n",
      " |      when using `self.metrics` that are supposed to get minimized i.e. `error`. For those,\n",
      " |      we can maximize `(-1) * metric`. One of the big pitfall of the current implementation\n",
      " |      is the way we are sampling hyper-parameters from the `params_bounds` where we are looking\n",
      " |      for an integer which is not possible. Therefore, for some of cases i.e. `max_depth` we\n",
      " |      must cast the sampled value which is mathematically wrong (i.e. f(1.1) != f(1)).\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      X : Union[pd.DataFrame, np.ndarray]\n",
      " |          Input data for training (features)\n",
      " |      \n",
      " |      y : Union[List[float], np.ndarray, pd.Series]\n",
      " |          Input ground truth for training (targets)\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      None\n",
      " |  \n",
      " |  get_best_params(self) -> Dict[str, Union[str, float, int]]\n",
      " |      Returns the tuned results of the optimization as the best set of hyper-parameters.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      Dict[str, Union[str, float, int]]\n",
      " |  \n",
      " |  get_best_results(self) -> pandas.core.frame.DataFrame\n",
      " |      Returns the performance of the best (tuned) set of hyper-parameters.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      pd.DataFrame\n",
      " |  \n",
      " |  get_optimizer(self) -> bayes_opt.bayesian_optimization.BayesianOptimization\n",
      " |      Return the Bayesian Optimization object.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      BayesianOptimization\n",
      " |  \n",
      " |  get_params_bounds(self) -> Optional[Dict[str, Tuple[Union[int, float], Union[int, float]]]]\n",
      " |      Returns the hyper-parameters boundaries for the tuning process.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      Dict[str, Tuple[Union[int, float], Union[int, float]]]\n",
      " |  \n",
      " |  get_results(self) -> pandas.core.frame.DataFrame\n",
      " |      Returns the hyper-parameter optimization results.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      pd.DataFrame\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes defined here:\n",
      " |  \n",
      " |  __abstractmethods__ = frozenset()\n",
      " |  \n",
      " |  __annotations__ = {'acquisition_criterion': typing.Optional[str], 'ear...\n",
      " |  \n",
      " |  __dataclass_fields__ = {'acquisition_criterion': Field(name='acquisiti...\n",
      " |  \n",
      " |  __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...\n",
      " |  \n",
      " |  __hash__ = None\n",
      " |  \n",
      " |  acquisition_criterion = 'ei'\n",
      " |  \n",
      " |  early_stopping_rounds = 20\n",
      " |  \n",
      " |  importance_type = 'total_gain'\n",
      " |  \n",
      " |  metrics = 'auc'\n",
      " |  \n",
      " |  n_init_iter = 5\n",
      " |  \n",
      " |  n_iter = 10\n",
      " |  \n",
      " |  n_splits = 4\n",
      " |  \n",
      " |  num_boost_round = 200\n",
      " |  \n",
      " |  objective = 'binary:logistic'\n",
      " |  \n",
      " |  params_bounds = None\n",
      " |  \n",
      " |  random_state = 1367\n",
      " |  \n",
      " |  scale_mean = False\n",
      " |  \n",
      " |  scale_std = False\n",
      " |  \n",
      " |  shuffle = True\n",
      " |  \n",
      " |  sparse_matrix = False\n",
      " |  \n",
      " |  stratified = True\n",
      " |  \n",
      " |  verbose = True\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from slickml.base._estimator.BaseXGBoostEstimator:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes inherited from slickml.base._estimator.BaseXGBoostEstimator:\n",
      " |  \n",
      " |  params = None\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from sklearn.base.BaseEstimator:\n",
      " |  \n",
      " |  __getstate__(self)\n",
      " |  \n",
      " |  __setstate__(self, state)\n",
      " |  \n",
      " |  get_params(self, deep=True)\n",
      " |      Get parameters for this estimator.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      deep : bool, default=True\n",
      " |          If True, will return the parameters for this estimator and\n",
      " |          contained subobjects that are estimators.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      params : dict\n",
      " |          Parameter names mapped to their values.\n",
      " |  \n",
      " |  set_params(self, **params)\n",
      " |      Set the parameters of this estimator.\n",
      " |      \n",
      " |      The method works on simple estimators as well as on nested objects\n",
      " |      (such as :class:`~sklearn.pipeline.Pipeline`). The latter have\n",
      " |      parameters of the form ``<component>__<parameter>`` so that it's\n",
      " |      possible to update each component of a nested object.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      **params : dict\n",
      " |          Estimator parameters.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      self : estimator instance\n",
      " |          Estimator instance.\n",
      "\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Example 1: `XGBoostBayesianOptimizer` for `classification` using `breast-cancer` data set"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "\n",
    "clf_data = load_breast_cancer()\n",
    "X_clf, y_clf = clf_data.data, clf_data.target"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "xbo_clf = XGBoostBayesianOptimizer(\n",
    "    n_iter=10,\n",
    "    metrics=\"auc\",\n",
    "    objective=\"binary:logistic\",\n",
    "    acquisition_criterion=\"ei\",\n",
    ")\n",
    "xbo_clf.fit(X_clf, y_clf)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "|   iter    |  target   | colsam... |   gamma   | learni... | max_depth | min_ch... | reg_alpha | reg_la... | subsample |\n",
      "-------------------------------------------------------------------------------------------------------------------------\n",
      "| \u001b[0m1        \u001b[0m | \u001b[0m0.9931   \u001b[0m | \u001b[0m0.8975   \u001b[0m | \u001b[0m0.04571  \u001b[0m | \u001b[0m0.6628   \u001b[0m | \u001b[0m4.238    \u001b[0m | \u001b[0m1.436    \u001b[0m | \u001b[0m0.3064   \u001b[0m | \u001b[0m0.7136   \u001b[0m | \u001b[0m0.1931   \u001b[0m |\n",
      "| \u001b[0m2        \u001b[0m | \u001b[0m0.9926   \u001b[0m | \u001b[0m0.7904   \u001b[0m | \u001b[0m0.6447   \u001b[0m | \u001b[0m0.9152   \u001b[0m | \u001b[0m3.334    \u001b[0m | \u001b[0m3.238    \u001b[0m | \u001b[0m0.7772   \u001b[0m | \u001b[0m0.269    \u001b[0m | \u001b[0m0.9726   \u001b[0m |\n",
      "| \u001b[0m3        \u001b[0m | \u001b[0m0.9827   \u001b[0m | \u001b[0m0.8498   \u001b[0m | \u001b[0m0.6044   \u001b[0m | \u001b[0m0.6874   \u001b[0m | \u001b[0m6.651    \u001b[0m | \u001b[0m15.7     \u001b[0m | \u001b[0m0.061    \u001b[0m | \u001b[0m0.5114   \u001b[0m | \u001b[0m0.6848   \u001b[0m |\n",
      "| \u001b[0m4        \u001b[0m | \u001b[0m0.9917   \u001b[0m | \u001b[0m0.7297   \u001b[0m | \u001b[0m0.8513   \u001b[0m | \u001b[0m0.4627   \u001b[0m | \u001b[0m4.757    \u001b[0m | \u001b[0m4.965    \u001b[0m | \u001b[0m0.9328   \u001b[0m | \u001b[0m0.363    \u001b[0m | \u001b[0m0.9365   \u001b[0m |\n",
      "| \u001b[0m5        \u001b[0m | \u001b[0m0.9899   \u001b[0m | \u001b[0m0.5425   \u001b[0m | \u001b[0m0.5451   \u001b[0m | \u001b[0m0.8782   \u001b[0m | \u001b[0m6.633    \u001b[0m | \u001b[0m5.028    \u001b[0m | \u001b[0m0.1845   \u001b[0m | \u001b[0m0.333    \u001b[0m | \u001b[0m0.9125   \u001b[0m |\n",
      "| \u001b[0m6        \u001b[0m | \u001b[0m0.5      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m2.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.1      \u001b[0m |\n",
      "| \u001b[0m7        \u001b[0m | \u001b[0m0.9913   \u001b[0m | \u001b[0m0.6497   \u001b[0m | \u001b[0m0.7199   \u001b[0m | \u001b[0m0.6417   \u001b[0m | \u001b[0m5.556    \u001b[0m | \u001b[0m4.987    \u001b[0m | \u001b[0m0.6124   \u001b[0m | \u001b[0m0.35     \u001b[0m | \u001b[0m0.9263   \u001b[0m |\n",
      "| \u001b[0m8        \u001b[0m | \u001b[0m0.9919   \u001b[0m | \u001b[0m0.7904   \u001b[0m | \u001b[0m0.3073   \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m5.175    \u001b[0m | \u001b[0m2.467    \u001b[0m | \u001b[0m0.1111   \u001b[0m | \u001b[0m0.3876   \u001b[0m | \u001b[0m0.5543   \u001b[0m |\n",
      "| \u001b[0m9        \u001b[0m | \u001b[0m0.9916   \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m5.551    \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.1      \u001b[0m |\n",
      "| \u001b[0m10       \u001b[0m | \u001b[0m0.9854   \u001b[0m | \u001b[0m0.1      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m3.905    \u001b[0m | \u001b[0m4.329    \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m1.0      \u001b[0m |\n",
      "| \u001b[0m11       \u001b[0m | \u001b[0m0.967    \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m4.231    \u001b[0m | \u001b[0m3.914    \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.1      \u001b[0m |\n",
      "| \u001b[0m12       \u001b[0m | \u001b[0m0.9901   \u001b[0m | \u001b[0m0.1      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m5.15     \u001b[0m | \u001b[0m1.319    \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m1.0      \u001b[0m |\n",
      "| \u001b[0m13       \u001b[0m | \u001b[0m0.9873   \u001b[0m | \u001b[0m0.4164   \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m6.294    \u001b[0m | \u001b[0m7.489    \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m1.0      \u001b[0m |\n",
      "| \u001b[0m14       \u001b[0m | \u001b[0m0.989    \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m3.489    \u001b[0m | \u001b[0m7.525    \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m1.0      \u001b[0m |\n",
      "| \u001b[0m15       \u001b[0m | \u001b[0m0.5      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m5.435    \u001b[0m | \u001b[0m7.382    \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m1.0      \u001b[0m |\n",
      "=========================================================================================================================\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "# hyper-parameter optimization results (or xbo_clf.results_)\n",
    "xbo_clf.get_results()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>colsample_bytree</th>\n",
       "      <th>gamma</th>\n",
       "      <th>learning_rate</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>min_child_weight</th>\n",
       "      <th>reg_alpha</th>\n",
       "      <th>reg_lambda</th>\n",
       "      <th>subsample</th>\n",
       "      <th>auc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.897530</td>\n",
       "      <td>0.045712</td>\n",
       "      <td>0.662807</td>\n",
       "      <td>4</td>\n",
       "      <td>1.435660</td>\n",
       "      <td>0.306424</td>\n",
       "      <td>0.713585</td>\n",
       "      <td>0.193055</td>\n",
       "      <td>0.993052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.790404</td>\n",
       "      <td>0.644709</td>\n",
       "      <td>0.915190</td>\n",
       "      <td>3</td>\n",
       "      <td>3.238280</td>\n",
       "      <td>0.777161</td>\n",
       "      <td>0.269010</td>\n",
       "      <td>0.972576</td>\n",
       "      <td>0.992562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.849819</td>\n",
       "      <td>0.604370</td>\n",
       "      <td>0.687435</td>\n",
       "      <td>6</td>\n",
       "      <td>15.698338</td>\n",
       "      <td>0.061001</td>\n",
       "      <td>0.511379</td>\n",
       "      <td>0.684811</td>\n",
       "      <td>0.982725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.729727</td>\n",
       "      <td>0.851274</td>\n",
       "      <td>0.462704</td>\n",
       "      <td>4</td>\n",
       "      <td>4.964748</td>\n",
       "      <td>0.932765</td>\n",
       "      <td>0.362983</td>\n",
       "      <td>0.936539</td>\n",
       "      <td>0.991745</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.542456</td>\n",
       "      <td>0.545092</td>\n",
       "      <td>0.878165</td>\n",
       "      <td>6</td>\n",
       "      <td>5.028311</td>\n",
       "      <td>0.184497</td>\n",
       "      <td>0.333049</td>\n",
       "      <td>0.912511</td>\n",
       "      <td>0.989944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.649743</td>\n",
       "      <td>0.719911</td>\n",
       "      <td>0.641733</td>\n",
       "      <td>5</td>\n",
       "      <td>4.986847</td>\n",
       "      <td>0.612361</td>\n",
       "      <td>0.350024</td>\n",
       "      <td>0.926344</td>\n",
       "      <td>0.991269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.790400</td>\n",
       "      <td>0.307348</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>5</td>\n",
       "      <td>2.466598</td>\n",
       "      <td>0.111085</td>\n",
       "      <td>0.387586</td>\n",
       "      <td>0.554268</td>\n",
       "      <td>0.991890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>5</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.991584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "      <td>4.328676</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.985419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4</td>\n",
       "      <td>3.913643</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.967036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>5</td>\n",
       "      <td>1.318832</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.990118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.416379</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6</td>\n",
       "      <td>7.489246</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.987295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "      <td>7.524550</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.988952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "      <td>7.381881</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    colsample_bytree     gamma  learning_rate  max_depth  min_child_weight  \\\n",
       "0           0.897530  0.045712       0.662807          4          1.435660   \n",
       "1           0.790404  0.644709       0.915190          3          3.238280   \n",
       "2           0.849819  0.604370       0.687435          6         15.698338   \n",
       "3           0.729727  0.851274       0.462704          4          4.964748   \n",
       "4           0.542456  0.545092       0.878165          6          5.028311   \n",
       "5           1.000000  0.000000       0.000000          2          1.000000   \n",
       "6           0.649743  0.719911       0.641733          5          4.986847   \n",
       "7           0.790400  0.307348       1.000000          5          2.466598   \n",
       "8           1.000000  0.000000       1.000000          5          1.000000   \n",
       "9           0.100000  0.000000       1.000000          3          4.328676   \n",
       "10          1.000000  1.000000       1.000000          4          3.913643   \n",
       "11          0.100000  0.000000       1.000000          5          1.318832   \n",
       "12          0.416379  0.000000       1.000000          6          7.489246   \n",
       "13          1.000000  1.000000       1.000000          3          7.524550   \n",
       "14          1.000000  1.000000       0.000000          5          7.381881   \n",
       "\n",
       "    reg_alpha  reg_lambda  subsample       auc  \n",
       "0    0.306424    0.713585   0.193055  0.993052  \n",
       "1    0.777161    0.269010   0.972576  0.992562  \n",
       "2    0.061001    0.511379   0.684811  0.982725  \n",
       "3    0.932765    0.362983   0.936539  0.991745  \n",
       "4    0.184497    0.333049   0.912511  0.989944  \n",
       "5    1.000000    1.000000   0.100000  0.500000  \n",
       "6    0.612361    0.350024   0.926344  0.991269  \n",
       "7    0.111085    0.387586   0.554268  0.991890  \n",
       "8    0.000000    1.000000   0.100000  0.991584  \n",
       "9    0.000000    0.000000   1.000000  0.985419  \n",
       "10   0.000000    1.000000   0.100000  0.967036  \n",
       "11   1.000000    0.000000   1.000000  0.990118  \n",
       "12   1.000000    0.000000   1.000000  0.987295  \n",
       "13   1.000000    0.000000   1.000000  0.988952  \n",
       "14   0.000000    1.000000   1.000000  0.500000  "
      ]
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "# best results (or xbo_clf.best_results)\n",
    "xbo_clf.get_best_results()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>colsample_bytree</th>\n",
       "      <th>gamma</th>\n",
       "      <th>learning_rate</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>min_child_weight</th>\n",
       "      <th>reg_alpha</th>\n",
       "      <th>reg_lambda</th>\n",
       "      <th>subsample</th>\n",
       "      <th>auc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.89753</td>\n",
       "      <td>0.045712</td>\n",
       "      <td>0.662807</td>\n",
       "      <td>4</td>\n",
       "      <td>1.43566</td>\n",
       "      <td>0.306424</td>\n",
       "      <td>0.713585</td>\n",
       "      <td>0.193055</td>\n",
       "      <td>0.993052</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   colsample_bytree     gamma  learning_rate  max_depth  min_child_weight  \\\n",
       "0           0.89753  0.045712       0.662807          4           1.43566   \n",
       "\n",
       "   reg_alpha  reg_lambda  subsample       auc  \n",
       "0   0.306424    0.713585   0.193055  0.993052  "
      ]
     },
     "metadata": {},
     "execution_count": 7
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "# best (tuned) params (or xbo_clf.best_params_)\n",
    "xbo_clf.get_best_params()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "{'colsample_bytree': 0.897530255703279,\n",
       " 'gamma': 0.045711540976251186,\n",
       " 'learning_rate': 0.6628071330998222,\n",
       " 'max_depth': 4,\n",
       " 'min_child_weight': 1.4356602059080719,\n",
       " 'reg_alpha': 0.3064243570574502,\n",
       " 'reg_lambda': 0.7135848847812896,\n",
       " 'subsample': 0.19305530141672728}"
      ]
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "# optimizer object (or xbo_clf.optimizer_)\n",
    "xbo_clf.get_optimizer()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "<bayes_opt.bayesian_optimization.BayesianOptimization at 0x1319f69a0>"
      ]
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "# optimization params boundaries or (xbo_clf.params_bounds)\n",
    "xbo_clf.get_params_bounds()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "{'max_depth': (2, 7),\n",
       " 'learning_rate': (0.0, 1.0),\n",
       " 'min_child_weight': (1.0, 20.0),\n",
       " 'colsample_bytree': (0.1, 1.0),\n",
       " 'subsample': (0.1, 1.0),\n",
       " 'gamma': (0.0, 1.0),\n",
       " 'reg_alpha': (0.0, 1.0),\n",
       " 'reg_lambda': (0.0, 1.0)}"
      ]
     },
     "metadata": {},
     "execution_count": 10
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Example 2: `XGBoostBayesianOptimizer` for `regression` using `california-housing` data set"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "\n",
    "reg_data = fetch_california_housing()\n",
    "X_reg, y_reg = reg_data.data, reg_data.target"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "xbo_reg = XGBoostBayesianOptimizer(\n",
    "    n_iter=10,\n",
    "    metrics=\"rmse\",\n",
    "    objective=\"reg:squarederror\",\n",
    "    acquisition_criterion=\"ei\",\n",
    ")\n",
    "xbo_reg.fit(X_reg, y_reg)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "|   iter    |  target   | colsam... |   gamma   | learni... | max_depth | min_ch... | reg_alpha | reg_la... | subsample |\n",
      "-------------------------------------------------------------------------------------------------------------------------\n",
      "| \u001b[0m1        \u001b[0m | \u001b[0m-0.5724  \u001b[0m | \u001b[0m0.8975   \u001b[0m | \u001b[0m0.04571  \u001b[0m | \u001b[0m0.6628   \u001b[0m | \u001b[0m4.238    \u001b[0m | \u001b[0m1.436    \u001b[0m | \u001b[0m0.3064   \u001b[0m | \u001b[0m0.7136   \u001b[0m | \u001b[0m0.1931   \u001b[0m |\n",
      "| \u001b[95m2        \u001b[0m | \u001b[95m-0.5094  \u001b[0m | \u001b[95m0.7904   \u001b[0m | \u001b[95m0.6447   \u001b[0m | \u001b[95m0.9152   \u001b[0m | \u001b[95m3.334    \u001b[0m | \u001b[95m3.238    \u001b[0m | \u001b[95m0.7772   \u001b[0m | \u001b[95m0.269    \u001b[0m | \u001b[95m0.9726   \u001b[0m |\n",
      "| \u001b[0m3        \u001b[0m | \u001b[0m-0.5251  \u001b[0m | \u001b[0m0.8498   \u001b[0m | \u001b[0m0.6044   \u001b[0m | \u001b[0m0.6874   \u001b[0m | \u001b[0m6.651    \u001b[0m | \u001b[0m15.7     \u001b[0m | \u001b[0m0.061    \u001b[0m | \u001b[0m0.5114   \u001b[0m | \u001b[0m0.6848   \u001b[0m |\n",
      "| \u001b[95m4        \u001b[0m | \u001b[95m-0.4907  \u001b[0m | \u001b[95m0.7297   \u001b[0m | \u001b[95m0.8513   \u001b[0m | \u001b[95m0.4627   \u001b[0m | \u001b[95m4.757    \u001b[0m | \u001b[95m4.965    \u001b[0m | \u001b[95m0.9328   \u001b[0m | \u001b[95m0.363    \u001b[0m | \u001b[95m0.9365   \u001b[0m |\n",
      "| \u001b[0m5        \u001b[0m | \u001b[0m-0.562   \u001b[0m | \u001b[0m0.5425   \u001b[0m | \u001b[0m0.5451   \u001b[0m | \u001b[0m0.8782   \u001b[0m | \u001b[0m6.633    \u001b[0m | \u001b[0m5.028    \u001b[0m | \u001b[0m0.1845   \u001b[0m | \u001b[0m0.333    \u001b[0m | \u001b[0m0.9125   \u001b[0m |\n",
      "| \u001b[95m6        \u001b[0m | \u001b[95m-0.4803  \u001b[0m | \u001b[95m0.4336   \u001b[0m | \u001b[95m0.282    \u001b[0m | \u001b[95m0.3017   \u001b[0m | \u001b[95m4.288    \u001b[0m | \u001b[95m16.0     \u001b[0m | \u001b[95m0.06196  \u001b[0m | \u001b[95m0.8237   \u001b[0m | \u001b[95m0.8923   \u001b[0m |\n",
      "| \u001b[0m7        \u001b[0m | \u001b[0m-0.5153  \u001b[0m | \u001b[0m0.5277   \u001b[0m | \u001b[0m0.7101   \u001b[0m | \u001b[0m0.3167   \u001b[0m | \u001b[0m2.901    \u001b[0m | \u001b[0m9.436    \u001b[0m | \u001b[0m0.6913   \u001b[0m | \u001b[0m0.1258   \u001b[0m | \u001b[0m0.6468   \u001b[0m |\n",
      "| \u001b[0m8        \u001b[0m | \u001b[0m-0.6169  \u001b[0m | \u001b[0m0.1291   \u001b[0m | \u001b[0m0.225    \u001b[0m | \u001b[0m0.8967   \u001b[0m | \u001b[0m6.216    \u001b[0m | \u001b[0m3.886    \u001b[0m | \u001b[0m0.5093   \u001b[0m | \u001b[0m0.2564   \u001b[0m | \u001b[0m0.3911   \u001b[0m |\n",
      "| \u001b[95m9        \u001b[0m | \u001b[95m-0.4797  \u001b[0m | \u001b[95m0.5203   \u001b[0m | \u001b[95m0.3491   \u001b[0m | \u001b[95m0.3815   \u001b[0m | \u001b[95m4.778    \u001b[0m | \u001b[95m15.94    \u001b[0m | \u001b[95m0.06113  \u001b[0m | \u001b[95m0.76     \u001b[0m | \u001b[95m0.85     \u001b[0m |\n",
      "| \u001b[0m10       \u001b[0m | \u001b[0m-0.501   \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.375    \u001b[0m | \u001b[0m3.67     \u001b[0m | \u001b[0m5.321    \u001b[0m | \u001b[0m1.0      \u001b[0m | \u001b[0m0.3619   \u001b[0m | \u001b[0m1.0      \u001b[0m |\n",
      "| \u001b[95m11       \u001b[0m | \u001b[95m-0.4786  \u001b[0m | \u001b[95m0.424    \u001b[0m | \u001b[95m0.4342   \u001b[0m | \u001b[95m0.3012   \u001b[0m | \u001b[95m4.387    \u001b[0m | \u001b[95m14.96    \u001b[0m | \u001b[95m0.2057   \u001b[0m | \u001b[95m0.6931   \u001b[0m | \u001b[95m0.918    \u001b[0m |\n",
      "| \u001b[0m12       \u001b[0m | \u001b[0m-0.5118  \u001b[0m | \u001b[0m0.9766   \u001b[0m | \u001b[0m0.9591   \u001b[0m | \u001b[0m0.04778  \u001b[0m | \u001b[0m4.325    \u001b[0m | \u001b[0m15.57    \u001b[0m | \u001b[0m0.8468   \u001b[0m | \u001b[0m0.9745   \u001b[0m | \u001b[0m0.9728   \u001b[0m |\n",
      "| \u001b[0m13       \u001b[0m | \u001b[0m-0.5857  \u001b[0m | \u001b[0m0.1      \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m0.7919   \u001b[0m | \u001b[0m4.435    \u001b[0m | \u001b[0m15.38    \u001b[0m | \u001b[0m0.0      \u001b[0m | \u001b[0m0.3536   \u001b[0m | \u001b[0m0.7704   \u001b[0m |\n",
      "| \u001b[0m14       \u001b[0m | \u001b[0m-0.4887  \u001b[0m | \u001b[0m0.9627   \u001b[0m | \u001b[0m0.5188   \u001b[0m | \u001b[0m0.1911   \u001b[0m | \u001b[0m3.983    \u001b[0m | \u001b[0m4.026    \u001b[0m | \u001b[0m0.2606   \u001b[0m | \u001b[0m0.2525   \u001b[0m | \u001b[0m0.6247   \u001b[0m |\n",
      "| \u001b[0m15       \u001b[0m | \u001b[0m-0.5091  \u001b[0m | \u001b[0m0.4854   \u001b[0m | \u001b[0m0.8195   \u001b[0m | \u001b[0m0.2852   \u001b[0m | \u001b[0m3.0      \u001b[0m | \u001b[0m9.594    \u001b[0m | \u001b[0m0.5141   \u001b[0m | \u001b[0m0.1084   \u001b[0m | \u001b[0m0.3677   \u001b[0m |\n",
      "=========================================================================================================================\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "source": [
    "# hyper-parameter optimization results (or xbo_reg.results_)\n",
    "# Note: We always maximize the metrics; so, here we maximize (-1) * (rmse) in which we technically minimizing (rmse)\n",
    "xbo_reg.get_results()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>colsample_bytree</th>\n",
       "      <th>gamma</th>\n",
       "      <th>learning_rate</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>min_child_weight</th>\n",
       "      <th>reg_alpha</th>\n",
       "      <th>reg_lambda</th>\n",
       "      <th>subsample</th>\n",
       "      <th>rmse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.897530</td>\n",
       "      <td>0.045712</td>\n",
       "      <td>0.662807</td>\n",
       "      <td>4</td>\n",
       "      <td>1.435660</td>\n",
       "      <td>0.306424</td>\n",
       "      <td>0.713585</td>\n",
       "      <td>0.193055</td>\n",
       "      <td>-0.572385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.790404</td>\n",
       "      <td>0.644709</td>\n",
       "      <td>0.915190</td>\n",
       "      <td>3</td>\n",
       "      <td>3.238280</td>\n",
       "      <td>0.777161</td>\n",
       "      <td>0.269010</td>\n",
       "      <td>0.972576</td>\n",
       "      <td>-0.509370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.849819</td>\n",
       "      <td>0.604370</td>\n",
       "      <td>0.687435</td>\n",
       "      <td>6</td>\n",
       "      <td>15.698338</td>\n",
       "      <td>0.061001</td>\n",
       "      <td>0.511379</td>\n",
       "      <td>0.684811</td>\n",
       "      <td>-0.525073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.729727</td>\n",
       "      <td>0.851274</td>\n",
       "      <td>0.462704</td>\n",
       "      <td>4</td>\n",
       "      <td>4.964748</td>\n",
       "      <td>0.932765</td>\n",
       "      <td>0.362983</td>\n",
       "      <td>0.936539</td>\n",
       "      <td>-0.490718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.542456</td>\n",
       "      <td>0.545092</td>\n",
       "      <td>0.878165</td>\n",
       "      <td>6</td>\n",
       "      <td>5.028311</td>\n",
       "      <td>0.184497</td>\n",
       "      <td>0.333049</td>\n",
       "      <td>0.912511</td>\n",
       "      <td>-0.562027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.433647</td>\n",
       "      <td>0.281979</td>\n",
       "      <td>0.301710</td>\n",
       "      <td>4</td>\n",
       "      <td>15.997975</td>\n",
       "      <td>0.061965</td>\n",
       "      <td>0.823726</td>\n",
       "      <td>0.892268</td>\n",
       "      <td>-0.480277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.527662</td>\n",
       "      <td>0.710147</td>\n",
       "      <td>0.316749</td>\n",
       "      <td>2</td>\n",
       "      <td>9.435776</td>\n",
       "      <td>0.691293</td>\n",
       "      <td>0.125812</td>\n",
       "      <td>0.646834</td>\n",
       "      <td>-0.515257</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.129065</td>\n",
       "      <td>0.224998</td>\n",
       "      <td>0.896721</td>\n",
       "      <td>6</td>\n",
       "      <td>3.885759</td>\n",
       "      <td>0.509329</td>\n",
       "      <td>0.256423</td>\n",
       "      <td>0.391094</td>\n",
       "      <td>-0.616924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.520283</td>\n",
       "      <td>0.349118</td>\n",
       "      <td>0.381455</td>\n",
       "      <td>4</td>\n",
       "      <td>15.939258</td>\n",
       "      <td>0.061133</td>\n",
       "      <td>0.760001</td>\n",
       "      <td>0.850027</td>\n",
       "      <td>-0.479728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.375002</td>\n",
       "      <td>3</td>\n",
       "      <td>5.320686</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.361939</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.501036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.424002</td>\n",
       "      <td>0.434217</td>\n",
       "      <td>0.301192</td>\n",
       "      <td>4</td>\n",
       "      <td>14.961345</td>\n",
       "      <td>0.205673</td>\n",
       "      <td>0.693149</td>\n",
       "      <td>0.917955</td>\n",
       "      <td>-0.478577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.976605</td>\n",
       "      <td>0.959093</td>\n",
       "      <td>0.047784</td>\n",
       "      <td>4</td>\n",
       "      <td>15.571021</td>\n",
       "      <td>0.846757</td>\n",
       "      <td>0.974531</td>\n",
       "      <td>0.972774</td>\n",
       "      <td>-0.511808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.791874</td>\n",
       "      <td>4</td>\n",
       "      <td>15.379617</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.353631</td>\n",
       "      <td>0.770436</td>\n",
       "      <td>-0.585725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.962705</td>\n",
       "      <td>0.518793</td>\n",
       "      <td>0.191124</td>\n",
       "      <td>3</td>\n",
       "      <td>4.026440</td>\n",
       "      <td>0.260595</td>\n",
       "      <td>0.252538</td>\n",
       "      <td>0.624655</td>\n",
       "      <td>-0.488710</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.485429</td>\n",
       "      <td>0.819539</td>\n",
       "      <td>0.285176</td>\n",
       "      <td>3</td>\n",
       "      <td>9.593877</td>\n",
       "      <td>0.514119</td>\n",
       "      <td>0.108390</td>\n",
       "      <td>0.367663</td>\n",
       "      <td>-0.509121</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    colsample_bytree     gamma  learning_rate  max_depth  min_child_weight  \\\n",
       "0           0.897530  0.045712       0.662807          4          1.435660   \n",
       "1           0.790404  0.644709       0.915190          3          3.238280   \n",
       "2           0.849819  0.604370       0.687435          6         15.698338   \n",
       "3           0.729727  0.851274       0.462704          4          4.964748   \n",
       "4           0.542456  0.545092       0.878165          6          5.028311   \n",
       "5           0.433647  0.281979       0.301710          4         15.997975   \n",
       "6           0.527662  0.710147       0.316749          2          9.435776   \n",
       "7           0.129065  0.224998       0.896721          6          3.885759   \n",
       "8           0.520283  0.349118       0.381455          4         15.939258   \n",
       "9           1.000000  1.000000       0.375002          3          5.320686   \n",
       "10          0.424002  0.434217       0.301192          4         14.961345   \n",
       "11          0.976605  0.959093       0.047784          4         15.571021   \n",
       "12          0.100000  0.000000       0.791874          4         15.379617   \n",
       "13          0.962705  0.518793       0.191124          3          4.026440   \n",
       "14          0.485429  0.819539       0.285176          3          9.593877   \n",
       "\n",
       "    reg_alpha  reg_lambda  subsample      rmse  \n",
       "0    0.306424    0.713585   0.193055 -0.572385  \n",
       "1    0.777161    0.269010   0.972576 -0.509370  \n",
       "2    0.061001    0.511379   0.684811 -0.525073  \n",
       "3    0.932765    0.362983   0.936539 -0.490718  \n",
       "4    0.184497    0.333049   0.912511 -0.562027  \n",
       "5    0.061965    0.823726   0.892268 -0.480277  \n",
       "6    0.691293    0.125812   0.646834 -0.515257  \n",
       "7    0.509329    0.256423   0.391094 -0.616924  \n",
       "8    0.061133    0.760001   0.850027 -0.479728  \n",
       "9    1.000000    0.361939   1.000000 -0.501036  \n",
       "10   0.205673    0.693149   0.917955 -0.478577  \n",
       "11   0.846757    0.974531   0.972774 -0.511808  \n",
       "12   0.000000    0.353631   0.770436 -0.585725  \n",
       "13   0.260595    0.252538   0.624655 -0.488710  \n",
       "14   0.514119    0.108390   0.367663 -0.509121  "
      ]
     },
     "metadata": {},
     "execution_count": 13
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "source": [
    "# best results (or xbo_ref.best_results)\n",
    "xbo_reg.get_best_results()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>colsample_bytree</th>\n",
       "      <th>gamma</th>\n",
       "      <th>learning_rate</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>min_child_weight</th>\n",
       "      <th>reg_alpha</th>\n",
       "      <th>reg_lambda</th>\n",
       "      <th>subsample</th>\n",
       "      <th>rmse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.424002</td>\n",
       "      <td>0.434217</td>\n",
       "      <td>0.301192</td>\n",
       "      <td>4</td>\n",
       "      <td>14.961345</td>\n",
       "      <td>0.205673</td>\n",
       "      <td>0.693149</td>\n",
       "      <td>0.917955</td>\n",
       "      <td>-0.478577</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   colsample_bytree     gamma  learning_rate  max_depth  min_child_weight  \\\n",
       "0          0.424002  0.434217       0.301192          4         14.961345   \n",
       "\n",
       "   reg_alpha  reg_lambda  subsample      rmse  \n",
       "0   0.205673    0.693149   0.917955 -0.478577  "
      ]
     },
     "metadata": {},
     "execution_count": 14
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "source": [
    "# best (tuned) params (or xbo_ref.best_params_)\n",
    "xbo_reg.get_best_params()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "{'colsample_bytree': 0.42400176809128554,\n",
       " 'gamma': 0.4342167350864106,\n",
       " 'learning_rate': 0.30119172469180727,\n",
       " 'max_depth': 4,\n",
       " 'min_child_weight': 14.961344534029857,\n",
       " 'reg_alpha': 0.20567264201500887,\n",
       " 'reg_lambda': 0.6931492141579287,\n",
       " 'subsample': 0.9179546981788781}"
      ]
     },
     "metadata": {},
     "execution_count": 15
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Feel free to add your favorite `Example` via a `pull-request`.\n",
    "### More details can be found in our [Contributing Document](https://github.com/slickml/slick-ml/blob/master/CONTRIBUTING.md)."
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.9.12 64-bit ('.venv': poetry)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "interpreter": {
   "hash": "07bad374b921cdabaf2ef6a1d4ae5a7996d892e7452f8d9d13efced363d002df"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}