notebooks/mdn-talk-examples.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set(style=\"darkgrid\")\n",
    "sns.set_context(\"talk\")\n",
    "x = np.linspace(-15,15,1000) # 1000 linearly spaced numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normal Distribution\n",
    "\n",
    "Let's plot the PDF (probability density function) of a normal distribution. The function is as follows:\n",
    "\n",
    "$$\\mathcal{N}(x \\mid \\mu, \\sigma^2) = \\frac{1}{\\sqrt{2\\pi\\sigma^2} } e^{ -\\frac{(x-\\mu)^2}{2\\sigma^2} }$$\n",
    "\n",
    "Where $\\mu$ and $\\sigma$ are the mean and standard deviation.\n",
    "\n",
    "$\\mathcal{N}(x \\mid \\mu, \\sigma^2)$ is the 'relative' likelihood of drawing $x$ from the model parametrised by $\\mu$ and $\\sigma$.\n",
    "\n",
    "Here's the PDF in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normal_pdf(x, mu, sigma):\n",
    "    \"\"\"PDF for normal distribution\"\"\"\n",
    "    return 1/np.sqrt(2 * np.pi * sigma**2) * np.exp(- (x - mu)**2 / (2 * sigma**2) )\n",
    "\n",
    "# Let's plot a normal distribution\n",
    "sns.lineplot(x=x, y=normal_pdf(x, 0.0, 5.0))\n",
    "plt.savefig(\"normal_distribution_mu0_sd5.png\", dpi=150, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model multiple distributions\n",
    "\n",
    "What if the data is complicated? It be a combination of multiple normal-ish distributions.\n",
    "\n",
    "Let's try this out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's try modelling two different distributions\n",
    "s1 = np.random.normal(-5, 2.0, 1000)\n",
    "s2 = np.random.normal(5, 3.0, 1000)\n",
    "all_s = np.concatenate([s1, s2])\n",
    "\n",
    "sns.histplot(all_s, kde=False)\n",
    "plt.savefig(\"complex_distribution_hist.png\", dpi=150, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can try to model this data with a normal distribution by calculating the mean and standard deviation, but it leads to a very poor model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s_mean = np.mean(all_s) # calculate mean of the whole sample\n",
    "s_sd = np.std(all_s) # calculate standard deviation of the whole sample.\n",
    "\n",
    "print(\"Plotting with normal distribution with mu=\", s_mean, \"and sd=\", s_sd)\n",
    "sns.distplot(all_s, hist=True, kde=False, norm_hist=True)\n",
    "sns.lineplot(x, normal_pdf(x, s_mean, s_sd), linewidth=2, color='r')\n",
    "plt.savefig(\"complex_distribution_hist_and_normal.png\", dpi=150, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solution! Mixture Distribution\n",
    "\n",
    "A mixture distribution lets us mix multiple different distributions together. In our case, we're just considering mixtures of normal models, but you can mix other models as well. The idea is to weight each distribution, so some might be more or less important, and then add them together.\n",
    "\n",
    "- Suppose that you have $K$ mixture components (number of distributions).\n",
    "- Then you need $K$ distribution parameters ($\\mu$ and $\\sigma$), one for each component. Let's call these $\\mu_i$ and $\\sigma_i$ where $i \\in [1, \\ldots, K]$.\n",
    "- Finally we need some weights $\\pi_i$, $i \\in [1, \\ldots, K]$, such that $\\sum_{i=1}^K \\pi_i = 1$\n",
    "\n",
    "Now the PDF of this mixture model will be:\n",
    "\n",
    "$$p(x) = \\sum_{i=1}^K \\pi_i\\mathcal{N}(x \\mid \\mu, \\sigma^2)$$\n",
    "\n",
    "That is, the likelihood of $x$ occuring in the distribution is the sum of the weighted likelihoods for $x$ from each Guassian model.\n",
    "\n",
    "Let's implement this in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mixture_of_normals_pdf(x, K, pis, mus, sigmas):\n",
    "    \"\"\"PDF for a mixture model of K normal distributions\"\"\"\n",
    "    output = 0\n",
    "    for i in range(K):\n",
    "        output += pis[i] * normal_pdf(x, mus[i], sigmas[i])\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Back to the data...\n",
    "\n",
    "Returning to our modelling problem, let's plot the PDF of a evenly-weighted mixture of the two sample normal models.\n",
    "\n",
    "this means that:\n",
    "\n",
    "- $K = 2$\n",
    "- $\\boldsymbol\\pi = [0.5, 0.5]$\n",
    "- $\\boldsymbol\\mu = [-5, 5]$\n",
    "- $\\boldsymbol\\sigma = [2, 3]$\n",
    "\n",
    "(bold used to indicate the vector of parameters for each component)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sns.distplot(all_s, hist=True, kde=False, norm_hist=True)\n",
    "sns.lineplot(x, mixture_of_normals_pdf(x, 2, [0.5,0.5], [-5,5], [2,3]), linewidth=2, color='r')\n",
    "plt.savefig(\"complex_distribution_hist_and_mixture.png\", dpi=150, bbox_inches='tight')\n",
    "## Much better!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much better!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting a more complex mixture model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = 3\n",
    "pis = [0.25, 0.3, 0.45]\n",
    "mus = [-9, -0.5, 8]\n",
    "sds = [2, 3, 2.5]\n",
    "\n",
    "sns.lineplot(x=x, y=pis[0]*normal_pdf(x, mus[0], sds[0]), linewidth=2, color='b')\n",
    "sns.lineplot(x=x, y=pis[1]*normal_pdf(x, mus[1], sds[1]), linewidth=2, color='b')\n",
    "sns.lineplot(x=x, y=pis[2]*normal_pdf(x, mus[2], sds[2]), linewidth=2, color='b')\n",
    "sns.lineplot(x=x, y=mixture_of_normals_pdf(x, K, pis, mus, sds), linewidth=2, color='r')\n",
    "plt.savefig(\"complex_mixture.png\", dpi=150, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.lineplot(x=x, y=mixture_of_normals_pdf(x, K, pis, mus, sds), linewidth=2, color='r')\n",
    "plt.savefig(\"mixture_plot.png\", dpi=150, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = ['A','A','A','A','A','B','B','B','C','C','C','C','C','C','C']\n",
    "L = pd.Series(data).value_counts(normalize=True, sort=False)\n",
    "sns.barplot(x=L.index, y=L)\n",
    "plt.savefig(\"categorical_plot.png\", dpi=150, bbox_inches='tight')\n",
    "#sns.barplot(x=data) #,  estimator='percentage')\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}