docs/source/tutorials/8_decorating_datafiles.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decorating datafiles\n",
    "\n",
    "This notebook gives an example how to decorate your irregular or Big Table datafiles.\n",
    "It makes use of the provided example data in the `Example Data` directory. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1 - Decorating an \"irregular\" datafile\n",
    "---\n",
    "\n",
    "An \"irregular\" datafile stores it's assays in separate tables that are either above one another or next to one another. \n",
    "To decorate the assays we have to add the decorators `@qpcr:assay` and `@qpcr:normaliser` wihtin the _cell_ immediately above the *assay header*. \n",
    "\n",
    "> Note\n",
    ">\n",
    "> Your assay headers **must** be either all in the _same_ column or row!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how in the file below we have decorated the assay for 28S as a _normaliser_ and the one below for HNRNPL as an _assay_ :\n",
    "\n",
    "<br>\n",
    "\n",
    ".. image:: .resources/decorator1.png\n",
    "\n",
    "![](./.resources/decorator1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "This exact scheme we can now follow for any assay within our datafile. Note, this also works on multi-sheet excel files. However, when working with Excel, you will likely have to add a single tick `'` in front of the decorator to escapt it's being interpreted as a function.\n",
    "\n",
    "> Note\n",
    ">\n",
    "> If you decorate your data, and specify that it's decorated when reading with one of the `qpcr.Readers` **any and all** non-decorated assays will be ignored!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same approach naturally also works for horizontal irregular datafiles (this will require the option `transpose = True` when working with the `qpcr.Readers` or the `transpose()` method when working directly with the `qpcr.Parsers`).\n",
    "\n",
    "The decorated file below is from [Hernandez (2018)](https://zenodo.org/record/1437804#.Yg65EO7MKCc). It and all following files below were obtained from Zenodo for the purpose of this tutorial.\n",
    "\n",
    "<br>\n",
    "\n",
    ".. image:: .resources/decorator2.png\n",
    "\n",
    "![](./.resources/decorator2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2 - Decorating a Big Table datafile\n",
    "---\n",
    "Big Table files store all their assays wihtin a single datatable, which is either \"vertical\" or \"horizontal\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The `@qpcr` decorator column\n",
    "\n",
    "Both kinds of tables allow \"decoration\" by adding a `@qpcr` column that stores the values `assay` or `normaliser`. The purpose of the column is probably self-evident at this point.\n",
    "\n",
    "The file below is a vertical Big Table file from [Holmann et al. (2015)](https://zenodo.org/record/5005916#.Yg7AvO7MKCc). Notice the `@qpcr` column at the very right?\n",
    "\n",
    "<br> \n",
    "\n",
    ".. image:: .resources/decorator3.png\n",
    "\n",
    "\n",
    "![](./.resources/decorator3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The `@qpcr:group` decorator\n",
    "\n",
    "For \"horizontal\" Big Tables, we require some more decoration to read. Because \"horizontal\" Big Tables store the replicate Ct values in side-by-side columns `qpcr` requires the decorator `@qpcr:group` to be in the cell immediately above the _first_ replicate Ct value column of each group. \n",
    "\n",
    "The decorated file below is from [Garcia-Gonzalez et al. (2020)](https://zenodo.org/record/3775511#.Yg7GeO7MKCc). Notice, how a `@qpcr:group` decorator is placed above the first columns of the Ct and IMC513 groups. Also, there is a `@qpcr` column denoting assays and normalisers. A little disclaimer at this point, Garcia-Gonzalez et al. did not actually include the ActinB values, they were added for this overview. Why are they not included? In fact, the values in the table are already delta-delta-Ct values! However, for the purpose of this tutorial their data arrangement is the only thing of importance.\n",
    "\n",
    "\n",
    ".. image:: .resources/decorator4.png\n",
    "\n",
    "![](./.resources/decorator4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    " > Why does `qpcr` require the group-decorators? \n",
    " >\n",
    " > Well, the reason is best explained by looking at a file from [Howe et al. (2020)](https://zenodo.org/record/3999572#.Yg7Oe-7MKCc).\n",
    "Their data includes triplicates for assays _CRH_ and _18s rRNA_. However, they also include columns with the replicate means as well. Now, such additional data columns within the data,\n",
    "no matter where they are located within the Big Table, actually make it impossible to just \"split\" the table by rows and be done. Also, notice how the column\n",
    "names of the triplicate Ct values are not identical? \n",
    ">\n",
    "> Hence, `qpcr` can also not rely on identically named columns to find the data. To solve this problem without forcing the user to overly meddle with their data arrangement, `qpcr` opted to just rely on some additional decorators. The replicate values must be specified manually in this setting, of course. Therefore, a little more manual user effort is required to read such a datafile, but the decorator approach allows for an arbitrary amount of additional data to be stored alongside the raw Ct values, and is thus also appropriate for re-visiting and -analysing already exisiting data easily.\n",
    "\n",
    "<br>\n",
    "\n",
    ".. image:: .resources/decorator5.png\n",
    "\n",
    "![](./.resources/decorator5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "With this we have reached the end of this tutorial. Adding `decorators` to your data is really easy. How we can then work with the decorated files using the `qpcr.Readers` and `qpcr.Parsers` has already been introduced in the previous two tutorials. You now know a very powerful way to speed up your analyses. Congrats!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "2734b62963039aeeb60b771e82b61c77981815502f674c850ca7bca72bab0171"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}