docs/chapter1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SciUnit Logo](https://raw.githubusercontent.com/scidash/assets/master/logos/SciUnit/sci-unit-tag.png)\n",
    "\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/scidash/sciunit/blob/master/docs/chapter1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "# Chapter 1. What is SciUnit?\n",
    "Everyone hopes that their model has some correspondence with reality.  Usually, checking whether this is true is done informally.\n",
    "### SciUnit makes this formal and transparent.\n",
    "SciUnit is a framework for validating scientific models by creating experimental-data-driven unit tests. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### If you are using this file in Google Colab, this block of code can help you install sciunit from PyPI in Colab environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-04T17:17:26.797848Z",
     "iopub.status.busy": "2021-05-04T17:17:26.796560Z",
     "iopub.status.idle": "2021-05-04T17:17:29.165793Z",
     "shell.execute_reply": "2021-05-04T17:17:29.167482Z"
    }
   },
   "outputs": [],
   "source": [
    "!pip install -q sciunit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After installation, let's begin with importing sciunit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-04T17:17:29.175051Z",
     "iopub.status.busy": "2021-05-04T17:17:29.174070Z",
     "iopub.status.idle": "2021-05-04T17:17:30.011470Z",
     "shell.execute_reply": "2021-05-04T17:17:30.009992Z"
    }
   },
   "outputs": [],
   "source": [
    "import sciunit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What does testing look like?  \n",
    "We'll start with a simple example from the history of cosmology, where we know that better models displaced their predecessors.<br>\n",
    "Suppose we have a test suite called \"Saturn Suite\" that aims to test cosmological models for their correspondence to empirical data about the planet Saturn.<br>\n",
    "*Everything in this example is hypothetical, but once you understand the basic ideas you can visit the [documentation for NeuronUnit](https://github.com/scidash/neuronunit/blob/master/docs/chapter1.ipynb) to see some working, interactive examples from a different domain (neuron and ion channel models).*  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from saturnsuite.tests import position_test,velocity_test,eccentricity_test # Examples of test classes.  \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's nothing specific to Saturn about position, velocity, or eccentricity.  They could apply to any cosmological body.\n",
    "### SciUnit test classes used with in one scientific domain (like cosmology) are located in a discipline-specific library. \n",
    "In this case, the test classes (hypothetically) come from a SciUnit library called CosmoUnit, and are instantiated with data specific to Saturn, in order to create tests specific to a model's predictions about Saturn:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "'''\n",
    "saturnsuite/tests.py # Tests for the Saturn suite.  \n",
    "'''\n",
    "from . import saturn_data # Hypothetical library containing Saturn data.  \n",
    "from cosmounit import PositionTest, VelocityTest, EccentricityyTest # Cosmounit is an external library.  \n",
    "position_test = PositionTest(observation=saturn_data.position)\n",
    "velocity_test = VelocityTest(observation=saturn_data.velocity)\n",
    "eccentricity_test = EccentricityTest(observation=saturn_data.eccentricity)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means the test *classes* are data-agnostic, but the test *instances* encode the data we want a model to recapitulate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's load some models that aim to predict the cosmological features being assessed by the tests above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from saturnsuite.models import ptolemy_model, copernicus_model, kepler_model, newton_model # Examples of models.  \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ptolemy's, Copernicus's, Kepler's, or Newton's models could similarly apply to any cosmological body.\n",
    "So these model classes are found in CosmoUnit, and the Saturn Suite contains model instances parameterized to emit predictions about Saturn specifically."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "'''\n",
    "saturnsuite/models.py # Models for the Saturn suite.  \n",
    "'''\n",
    "from cosmounit import PtolemyModel, CopernicusModel, KeplerModel, NewtonModel  \n",
    "ptolemy_model = PtolemyModel(planet='Saturn')\n",
    "copernicus_model = CopernicusModel(planet='Saturn')\n",
    "kepler_model = KeplerModel(planet='Saturn')\n",
    "newton_model = NewtonModel(planet='Saturn')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above each model takes a keyword argument 'planet' that determines about what planet the model will make predicitons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### All of our tests can be organized into a suite to compare results across related tests.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "'''\n",
    "saturnsuite/suites.py # Tests for the Saturn suite.  \n",
    "'''\n",
    "import sciunit\n",
    "from .tests import position_test, velocity_test, eccentricity_test\n",
    "saturn_motion_suite = sciunit.TestSuite([position_test, velocity_test, eccentricity_test])\n",
    "suites = (saturn_motion_suite,)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we can execute this entire test suite against our models.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from saturn_suite.suites import saturn_motion_suite\n",
    "saturn_motion_suite.judge([ptolemy_model, copernicus_model, kepler_model, newton_model])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The exact output will depend on your preferences (terminal, HTML, etc.) but the figure below illustrates both the results you get (center table) and the relationship between the components listed here. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Cosmo Example](https://raw.githubusercontent.com/scidash/assets/master/figures/cosmo_example.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The figure above also refers to SciDash, an in-development portal for accessing public test results, but for the remainder of this tutorial, we will focus on model/test development, execution, and visualization on your own machine.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In the next section we'll see how to create models and tests from scratch in SciUnit.  \n",
    "### Onto [Chapter 2](chapter2.ipynb)!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}