doc/examples/example_cli_famos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e28947de",
   "metadata": {},
   "source": [
    "# FAMoS in PEtab Select\n",
    "\n",
    "In this notebook, the FAMoS algorithm [1] is demonstrated.\n",
    "\n",
    "[1] Gabel M, Hohl T, Imle A, Fackler OT, Graw F (2019) FAMoS: A Flexible and dynamic Algorithm for Model Selection to analyse complex systems dynamics. PLOS Computational Biology 15(8): e1007230. https://doi.org/10.1371/journal.pcbi.1007230"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "484b7f3c",
   "metadata": {},
   "source": [
    "The model space contains 65536 models. In normal PEtab Select workflows, a calibration tool would take the place of the `example_cli_famos_calibration_tool.py` script. This script emulates a calibration tool: it takes PEtab Select models and assigns criterion values to them, based on previous calibration results for the same models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1f04dce0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cleanup the state and candidate models output by a previous run of this notebook\n",
    "import shutil\n",
    "from pathlib import Path\n",
    "\n",
    "from example_cli_famos_helpers import (\n",
    "    expected_criterion_values,\n",
    "    parse_summary_to_progress_list,\n",
    "    petab_select_problem_yaml,\n",
    ")\n",
    "\n",
    "output_path = Path().resolve() / \"output_famos\"\n",
    "output_path_str = str(output_path)\n",
    "shutil.rmtree(output_path_str)\n",
    "output_path.mkdir(exist_ok=False, parents=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a81560e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import petab\n",
    "\n",
    "import petab_select\n",
    "from petab_select import ESTIMATE, FamosCandidateSpace, Method, Model\n",
    "from petab_select.constants import Criterion\n",
    "from petab_select.model import default_compare\n",
    "\n",
    "state = str(output_path / 'state.dill')\n",
    "\n",
    "# Each iteration of model selection is described as a 2-tuple here.\n",
    "# First value is the model selection method.\n",
    "# Second value are relative change in parameter indices that correspond\n",
    "# to the best model from this iteration.\n",
    "# e.g. `(Method.FORWARD, {3})` states that the best model from a forward move\n",
    "# is the model that now estimates the parameter at index 3.\n",
    "expected_progress_list = [\n",
    "    (Method.LATERAL, set()),\n",
    "    (Method.LATERAL, {4, 15}),\n",
    "    (Method.LATERAL, {9, 13}),\n",
    "    (Method.FORWARD, set()),\n",
    "    (Method.FORWARD, {3}),\n",
    "    (Method.FORWARD, {11}),\n",
    "    (Method.BACKWARD, set()),\n",
    "    (Method.BACKWARD, {6}),\n",
    "    (Method.BACKWARD, {10}),\n",
    "    (Method.BACKWARD, {8}),\n",
    "    (Method.BACKWARD, {14}),\n",
    "    (Method.BACKWARD, {1}),\n",
    "    (Method.BACKWARD, {16}),\n",
    "    (Method.BACKWARD, {4}),\n",
    "    (Method.FORWARD, set()),\n",
    "    (Method.LATERAL, set()),\n",
    "    (Method.MOST_DISTANT, {2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 15}),\n",
    "    (Method.LATERAL, {16, 7}),\n",
    "    (Method.LATERAL, {5, 12}),\n",
    "    (Method.LATERAL, {13, 15}),\n",
    "    (Method.LATERAL, {1, 6}),\n",
    "    (Method.FORWARD, set()),\n",
    "    (Method.FORWARD, {3}),\n",
    "    (Method.FORWARD, {7}),\n",
    "    (Method.FORWARD, {2}),\n",
    "    (Method.FORWARD, {11}),\n",
    "    (Method.BACKWARD, set()),\n",
    "    (Method.BACKWARD, {7}),\n",
    "    (Method.BACKWARD, {16}),\n",
    "    (Method.BACKWARD, {4}),\n",
    "    (Method.FORWARD, set()),\n",
    "    (Method.LATERAL, set()),\n",
    "    (Method.LATERAL, {9, 15}),\n",
    "    (Method.FORWARD, set()),\n",
    "    (Method.BACKWARD, set()),\n",
    "    (Method.LATERAL, set()),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7202f6c6",
   "metadata": {},
   "source": [
    "The predecessor model is some model from the model space, and is defined in the PEtab Select problem YAML file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bb1a5144",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$petab_select_problem_yaml\" \"$output_path_str\"\n",
    "\n",
    "petab_select_problem_yaml=$1\n",
    "output_path_str=$2\n",
    "\n",
    "problem=$petab_select_problem_yaml\n",
    "state=$output_path_str/state.dill\n",
    "\n",
    "newly_calibrated_models=\"\"\n",
    "calibrated_models=\"\"\n",
    "empty_output=$(echo -e \"null\\n...\")\n",
    "\n",
    "for i in {1..40}\n",
    "do\n",
    "\n",
    "output=$output_path_str/models_$i.yaml\n",
    "calibrated_output=$output_path_str/calibrated_models_$i.yaml\n",
    "\n",
    "petab_select candidates \\\n",
    "--problem $problem \\\n",
    "--state $state \\\n",
    "--output $output \\\n",
    "$newly_calibrated_models \\\n",
    "$calibrated_models \\\n",
    "--relative-paths\n",
    "\n",
    "# Replace this line with a tool that calibrates the models.\n",
    "# The script also changes model IDs for easier analysis in this example.\n",
    "python example_cli_famos_calibration_tool.py $output $calibrated_output\n",
    "\n",
    "if [ \"$(cat $calibrated_output)\" == \"$empty_output\" ]\n",
    "then\n",
    "        # End the model selection if no models were provided by PEtab Select.\n",
    "        break\n",
    "else\n",
    "        newly_calibrated_models=\"--newly-calibrated-models \"$calibrated_output\n",
    "        calibrated_models+=\"--calibrated-models \"$calibrated_output\" \"\n",
    "fi\n",
    "\n",
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "93caf071",
   "metadata": {},
   "outputs": [],
   "source": [
    "progress_list = parse_summary_to_progress_list('output_famos/summary.tsv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cb61d0f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert progress_list == expected_progress_list"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}