examples/Hyperparams And Distributions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparams And Distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This page introduces the hyperparams, and distributions in Neuraxle. You can find [Hyperparams Distribution API here](https://www.neuraxle.org/stable/api/neuraxle.hyperparams.distributions.html), and \n",
    "[Hyperparameter Samples API here](https://www.neuraxle.org/stable/api/neuraxle.hyperparams.space.html).\n",
    "\n",
    "Hyperparameter is a parameter drawn from a prior distribution. In Neuraxle, we have a few built-in distributions, and we are also compatible with scipy distributions. \n",
    "\n",
    "Create a [Uniform Distribution](https://www.neuraxle.org/stable/api/neuraxle.hyperparams.distributions.html#neuraxle.hyperparams.distributions.Uniform):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from neuraxle.hyperparams.distributions import Uniform\n",
    "\n",
    "hd = Uniform(\n",
    "    min_included=-10, \n",
    "    max_included=10, \n",
    "    null_default_value=0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sample the random variable using [rvs](https://www.neuraxle.org/stable/api/neuraxle.hyperparams.distributions.html#neuraxle.hyperparams.distributions.HyperparameterDistribution.rvs):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-5.873595345744127\n"
     ]
    }
   ],
   "source": [
    "sample = hd.rvs()\n",
    "print(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nullify the random variable using [nullify](https://www.neuraxle.org/stable/api/neuraxle.hyperparams.distributions.html#neuraxle.hyperparams.distributions.HyperparameterDistribution.nullify):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "nullified_sample = hd.nullify()\n",
    "assert nullified_sample == hd.null_default_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the probability distribution function value at `x` using [pdf](https://www.neuraxle.org/stable/api/neuraxle.hyperparams.distributions.html#neuraxle.hyperparams.distributions.HyperparameterDistribution.pdf):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pdf: 0.05\n"
     ]
    }
   ],
   "source": [
    "pdf = hd.pdf(1)\n",
    "print('pdf: {}'.format(pdf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the cumulative probability distribution function value at `x` using [cdf](https://www.neuraxle.org/stable/api/neuraxle.hyperparams.distributions.html#neuraxle.hyperparams.distributions.HyperparameterDistribution.cdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cdf: 0.55\n"
     ]
    }
   ],
   "source": [
    "cdf = hd.cdf(1)\n",
    "print('cdf: {}'.format(cdf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting And Updating Hyperparams\n",
    "\n",
    "\n",
    "In Neuraxle, each step has hyperparams of type [HyperparameterSamples](https://www.neuraxle.org/stable/api/neuraxle.hyperparams.space.html#neuraxle.hyperparams.space.HyperparameterSamples), and spaces of type [HyperparameterSpace](https://www.neuraxle.org/stable/api/neuraxle.hyperparams.distributions.html#neuraxle.hyperparams.distributions.HyperparameterDistribution).  \n",
    "\n",
    "Consider a simple pipeline that contains 2 MultiplyByN steps, and one PCA component inside a nested pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "from neuraxle.hyperparams.distributions import RandInt\n",
    "from neuraxle.hyperparams.space import HyperparameterSpace, HyperparameterSamples\n",
    "from neuraxle.pipeline import Pipeline\n",
    "from neuraxle.steps.numpy import MultiplyByN\n",
    "\n",
    "p = Pipeline([\n",
    "    ('step1', MultiplyByN(2)),\n",
    "    ('step2', MultiplyByN(2)),\n",
    "    Pipeline([\n",
    "        PCA(n_components=4)\n",
    "    ])\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can set or update the hyperparams, and spaces by doing the following:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline\n",
       "(\n",
       "\tPipeline(\n",
       "\tname=Pipeline,\n",
       "\thyperparameters=HyperparameterSamples([('step1', RecursiveDict([('multiply_by', 42)])),\n",
       "                       ('step2', RecursiveDict([('multiply_by', -10)])),\n",
       "                       ('Pipeline',\n",
       "                        RecursiveDict([('PCA',\n",
       "                                        RecursiveDict([('n_components',\n",
       "                                                        3)]))]))])\n",
       ")(\n",
       "\t\t[('step1',\n",
       "  MultiplyByN(\n",
       "\tname=step1,\n",
       "\thyperparameters=HyperparameterSamples([('multiply_by', 42)])\n",
       ")),\n",
       " ('step2',\n",
       "  MultiplyByN(\n",
       "\tname=step2,\n",
       "\thyperparameters=HyperparameterSamples([('multiply_by', -10)])\n",
       ")),\n",
       " ('Pipeline',\n",
       "  Pipeline\n",
       "(\n",
       "\tPipeline(\n",
       "\tname=Pipeline,\n",
       "\thyperparameters=HyperparameterSamples([('PCA', RecursiveDict([('n_components', 3)]))])\n",
       ")(\n",
       "\t\t[('PCA', <neuraxle.steps.sklearn.SKLearnWrapper(PCA(...)) object 0x108906810>)]\t\n",
       ")\n",
       "))]\t\n",
       ")\n",
       ")"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p.set_hyperparams(HyperparameterSamples({\n",
    "    'step1__multiply_by': 42,\n",
    "    'step2__multiply_by': -10,\n",
    "    'Pipeline__PCA__n_components': 2\n",
    "}))\n",
    "\n",
    "p.update_hyperparams(HyperparameterSamples({\n",
    "    'Pipeline__PCA__n_components': 3\n",
    "}))\n",
    "\n",
    "p.set_hyperparams_space(HyperparameterSpace({\n",
    "    'step1__multiply_by': RandInt(42, 50),\n",
    "    'step2__multiply_by': RandInt(-10, 0),\n",
    "    'Pipeline__PCA__n_components': RandInt(2, 3)\n",
    "}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can sample the space of random variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = p.get_hyperparams_space().rvs()\n",
    "\n",
    "assert 42 <= samples['step1__multiply_by'] <= 50\n",
    "assert -10 <= samples['step2__multiply_by'] <= 0\n",
    "assert samples['Pipeline__PCA__n_components'] in [2, 3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get all hyperparams:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = p.get_hyperparams()\n",
    "\n",
    "assert 42 <= samples['step1__multiply_by'] <= 50\n",
    "assert -10 <= samples['step2__multiply_by'] <= 0\n",
    "assert samples['Pipeline__PCA__n_components'] in [2, 3]\n",
    "assert p['Pipeline']['PCA'].get_wrapped_sklearn_predictor().n_components in [2, 3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neuraxle Custom Distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scipy Distributions \n",
    "\n",
    "To define a scipy distribution that is compatible with Neuraxle, you need to wrap the scipy distribution with ScipyDistributionWrapper: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from neuraxle.hyperparams.scipy_distributions import ScipyDistributionWrapper, BaseContinuousDistribution, BaseDiscreteDistribution\n",
    "from scipy.integrate import quad\n",
    "from scipy.special import factorial\n",
    "from scipy.stats import rv_continuous, norm, rv_discrete, rv_histogram, truncnorm, randint\n",
    "import numpy as np\n",
    "import math \n",
    "\n",
    "hd = ScipyDistributionWrapper(\n",
    "    scipy_distribution=randint(low=0, high=10),\n",
    "    is_continuous=False,\n",
    "    null_default_value=0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discrete Distributions\n",
    "For discrete distribution that inherit from [rv_discrete](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_discrete.html#scipy.stats.rv_discrete), you only need to implement _pmf. The rest is taken care of magically by scipy.  \n",
    "\n",
    "For example, here is a discrete poisson distribution: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Poisson(BaseDiscreteDistribution):\n",
    "    def __init__(self, min_included: float, max_included: float, null_default_value: float = None, mu=0.6):\n",
    "        super().__init__(\n",
    "            min_included=min_included,\n",
    "            max_included=max_included,\n",
    "            name='poisson',\n",
    "            null_default_value=null_default_value\n",
    "        )\n",
    "        self.mu = mu\n",
    "\n",
    "    def _pmf(self, x):\n",
    "        return math.exp(-self.mu) * self.mu ** x / factorial(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Continuous Distributions\n",
    "\n",
    "For continous distribution that inherit from [rv_continuous](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_continuous.html), you only need to implement _pdf function. The rest is taken care of magically by scipy.  \n",
    "\n",
    "For example, here is a continous gaussian distribution: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Gaussian(BaseContinuousDistribution):    \n",
    "    def __init__(self, min_included: int, max_included: int, null_default_value: float = None):\n",
    "        self.max_included = max_included\n",
    "        self.min_included = min_included\n",
    "\n",
    "        BaseContinuousDistribution.__init__(\n",
    "            self,\n",
    "            name='gaussian',\n",
    "            min_included=min_included,\n",
    "            max_included=max_included,\n",
    "            null_default_value=null_default_value\n",
    "        )\n",
    "\n",
    "    def _pdf(self, x):\n",
    "        return math.exp(-x ** 2 / 2.) / np.sqrt(2.0 * np.pi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Arguments \n",
    "\n",
    "If you want to add more properties to calculate your distributions, just add them in self. They will be available in all of the scipy private methods you can override like _pmf, and _pdf. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LogNormal(BaseContinuousDistribution):\n",
    "    def __init__(\n",
    "            self,\n",
    "            log2_space_mean: float,\n",
    "            log2_space_std: float,\n",
    "            hard_clip_min: float,\n",
    "            hard_clip_max: float,\n",
    "            null_default_value: float = None\n",
    "    ):\n",
    "        if null_default_value is None:\n",
    "            null_default_value = hard_clip_min\n",
    "\n",
    "        if hard_clip_min is None:\n",
    "            hard_clip_min = np.nan\n",
    "\n",
    "        if hard_clip_max is None:\n",
    "            hard_clip_max = np.nan\n",
    "\n",
    "        self.log2_space_mean = log2_space_mean\n",
    "        self.log2_space_std = log2_space_std\n",
    "\n",
    "        super().__init__(\n",
    "            name='log_normal',\n",
    "            min_included=hard_clip_min,\n",
    "            max_included=hard_clip_max,\n",
    "            null_default_value=null_default_value\n",
    "        )\n",
    "\n",
    "    def _pdf(self, x):\n",
    "        if x <= 0:\n",
    "            return 0.\n",
    "\n",
    "        cdf_min = 0.\n",
    "        cdf_max = 1.\n",
    "\n",
    "        pdf_x = 1 / (x * math.log(2) * self.log2_space_std * math.sqrt(2 * math.pi)) * math.exp(\n",
    "            -(math.log2(x) - self.log2_space_mean) ** 2 / (2 * self.log2_space_std ** 2))\n",
    "        return pdf_x / (cdf_max - cdf_min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scipy methods\n",
    "\n",
    "All of the scipy distribution methods are available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_many_samples_for(hd, num_trial):\n",
    "    return [hd.rvs() for _ in range(num_trial)]\n",
    "\n",
    "samples = get_many_samples_for(hd, 1000)\n",
    "\n",
    "for s in samples:\n",
    "    assert type(s) == int\n",
    "    \n",
    "hd = Gaussian(min_included=0, max_included=10, null_default_value=0)\n",
    "\n",
    "assert 0.0 <= hd.rvs() <= 10.0\n",
    "assert hd.pdf(10) < 0.001\n",
    "assert hd.pdf(0) < 0.42\n",
    "assert 0.55 > hd.cdf(5.0) > 0.45\n",
    "assert hd.cdf(0) == 0.0\n",
    "assert hd.logpdf(5) == -13.418938533204672\n",
    "assert hd.logcdf(5) == -0.6931477538632531\n",
    "assert hd.sf(5) == 0.5000002866515718\n",
    "assert hd.logsf(5) == -0.693146607256966\n",
    "assert np.all(hd.ppf([0.0, 0.01, 0.05, 0.1, 1 - 0.10, 1 - 0.05, 1 - 0.01, 1.0], 10))\n",
    "assert np.isclose(hd.moment(2), 50.50000000091249)\n",
    "assert hd.stats()[0]\n",
    "assert hd.stats()[1]\n",
    "assert np.array_equal(hd.entropy(), np.array(0.7094692666023363))\n",
    "assert hd.median()\n",
    "assert hd.mean() == 5.398942280397029\n",
    "assert np.isclose(hd.std(), 4.620759921685374)\n",
    "assert np.isclose(hd.var(), 21.35142225385382)\n",
    "assert np.isclose(hd.expect(), 0.39894228040143276)\n",
    "interval = hd.interval(alpha=[0.25, 0.50])\n",
    "assert np.all(interval[0])\n",
    "assert np.all(interval[1])\n",
    "assert hd.support() == (0, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SKLearn Hyperparams\n",
    "\n",
    "SKLearnWrapper wraps sklearn predictors so that they can be compatible with Neuraxle. When you set the hyperparams of an SKLearnWrapper, it automatically sets the params of the sklearn predictor for you: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from neuraxle.hyperparams.distributions import Choice\n",
    "from neuraxle.hyperparams.distributions import RandInt\n",
    "from neuraxle.hyperparams.space import HyperparameterSpace\n",
    "from neuraxle.steps.sklearn import SKLearnWrapper\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "\n",
    "decision_tree_classifier = SKLearnWrapper(\n",
    "    DecisionTreeClassifier(), \n",
    "    HyperparameterSpace({\n",
    "        'criterion': Choice(['gini', 'entropy']), \n",
    "        'splitter': Choice(['best', 'random']),\n",
    "        'min_samples_leaf': RandInt(2, 5), \n",
    "        'min_samples_split': RandInt(1, 3) \n",
    "    })\n",
    ").set_hyperparams(HyperparameterSamples({\n",
    "    'criterion': 'gini', \n",
    "    'splitter': 'best',\n",
    "    'min_samples_leaf': 3, \n",
    "    'min_samples_split': 3 \n",
    "}))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "neuraxle_notebooks",
   "language": "python",
   "name": "neuraxle_notebooks"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}