theory2practice.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accurate Learning with Neural Networks - from Theory to Practice\n",
    "\n",
    "> Accompanying code for the paper 'Training ReLU Networks to high uniform accuracy is \n",
    "> intractable'. Implemented in [PyTorch](https://pytorch.org/), experiment execution and tracking using [Ray Tune](https://www.ray.io/ray-tune) \n",
    "> and [Weights & Biases](https://wandb.ai/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, HTML\n",
    "from pathlib import Path\n",
    "\n",
    "from theory2practice import utils, analysis\n",
    "\n",
    "%load_ext autoreload\n",
    "%load_ext tensorboard\n",
    "%autoreload 2\n",
    "\n",
    "display(HTML(\"<style>.container { width:100% !important; }</style>\"))\n",
    "spec_dir = Path(\"specs\")\n",
    "results_dir = Path(\"results\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project = \"test\"\n",
    "experiment = \"exp_0\"\n",
    "resume = False\n",
    "\n",
    "exp_file = spec_dir / project / f\"{experiment}.yaml\"\n",
    "runner_file = spec_dir / \"runner_resume.yaml\" if resume else spec_dir / \"runner.yaml\"\n",
    "!python main.py -e {exp_file} -r {runner_file}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensorboard --logdir {results_dir / project}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize Experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "690340a5df0a4acd9aadd4f172f67715",
      "436710f5db354175bc4a2ac3c9ffff5f",
      "6aa0edb19a8349b99bd6529f7a789c48"
     ]
    },
    "executionInfo": {
     "elapsed": 2629,
     "status": "ok",
     "timestamp": 1628963443343,
     "user": {
      "displayName": "Anonymously",
      "photoUrl": "",
      "userId": "08946025825257199521"
     },
     "user_tz": -120
    },
    "id": "4jr3tqgUHPyG",
    "outputId": "4ad10684-b984-4c38-9bb4-b03119886ae9"
   },
   "outputs": [],
   "source": [
    "# hold ctr to select\n",
    "project = \"1d_5x32\"\n",
    "\n",
    "selector = analysis.selector(path=results_dir / project)\n",
    "selector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1123,
     "status": "ok",
     "timestamp": 1628892228160,
     "user": {
      "displayName": "Anonymously",
      "photoUrl": "",
      "userId": "08946025825257199521"
     },
     "user_tz": -120
    },
    "id": "8tGPcxNqH_gR",
    "outputId": "eaec8c8f-b22e-4f75-848b-4db1dc2b14eb",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "visualizer = analysis.Visualizer(\n",
    "    exp_dirs=selector.value,\n",
    ")\n",
    "visualizer.initialize()\n",
    "\n",
    "metrics = list(visualizer.metrics.keys())\n",
    "print(f\"Available metrics: {metrics}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 12,
     "status": "ok",
     "timestamp": 1628892229107,
     "user": {
      "displayName": "Anonymously",
      "photoUrl": "",
      "userId": "08946025825257199521"
     },
     "user_tz": -120
    },
    "id": "7yDnpNe4hQxy",
    "outputId": "222f25c6-3a70-4adc-8424-5a2dab81420c"
   },
   "outputs": [],
   "source": [
    "# select a metric from the list in the output above (defaults to first metric)\n",
    "metric = metrics[0]\n",
    "\n",
    "for n_samples, df in visualizer.results[metric][\"summary\"].items():\n",
    "    print(f\"---- {n_samples} samples ----\")\n",
    "    display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now plot the min-max rates\n",
    "\n",
    "$$\n",
    "    \\min_{\\text{algo}} \\max_{\\text{target}} \\mathbb{E}\\left[ \n",
    "    \\text{metric}\\left(\\text{algo}(\\text{target})-\\text{target}\\right)\\right],\n",
    "$$\n",
    "\n",
    "the theoretical lower bound on the rates (with constant $1$ for simplicity), and the min-avg rates\n",
    "\n",
    "$$\n",
    "    \\min_{\\text{algo}}  \\frac{1}{\\text{n_targets}} \\sum_{\\text{target}} \\mathbb{E}\\left[ \n",
    "    \\text{metric}\\left(\\text{algo}(\\text{target})-\\text{target}\\right)\\right]\n",
    "$$\n",
    "\n",
    "over the sample size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note that only the projects `1d_5x32` and `3d_5x32` consider multiple sample sizes.\n",
    "# For the other projects (`test` and `1d_sine`) the exponential fit does not make sense and yields a RankWarning.\n",
    "figs = visualizer.plot()\n",
    "figs[0].show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot of runs contributing to the min_max rate (only if model checkpoints have been saved):\n",
    "for fig in figs[1:]:\n",
    "    fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show runs contributing to the min_max rate\n",
    "visualizer.results[metric][\"minmax\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show runs contributing to the min_avg rate\n",
    "visualizer.results[metric][\"minavg\"]"
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "accelerator": "GPU",
  "colab": {
   "authorship_tag": "ABX9TyOg06pV++UObvLamLPAfmbB",
   "collapsed_sections": [],
   "mount_file_id": "1NqgMQCVQBEfNvYISnU64o5YLXb5Z4-bA",
   "name": "theory2practice.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "t2p",
   "language": "python",
   "name": "t2p"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}