notebooks/Convert ns-ALEX HT3 files to Photon-HDF5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convert ns-ALEX HT3 files to Photon-HDF5\n",
    "\n",
    "<p class=\"lead\">This <a href=\"https://jupyter.org/\">Jupyter notebook</a>\n",
    "will guide you through the conversion of a ns-ALEX data file from <b>HT3</b>\n",
    "to <a href=\"http://photon-hdf5.org\">Photon-HDF5</a> format. For more info on how to edit\n",
    "a jupyter notebook refer to <a href=\"http://nbviewer.jupyter.org/github/jupyter/notebook/blob/master/docs/source/examples/Notebook/Notebook%20Basics.ipynb#Overview-of-the-Notebook-UI\">this example</a>.</p>\n",
    "\n",
    "<i>Please send feedback and report any problem to the \n",
    "[Photon-HDF5 google group](https://groups.google.com/forum/#!forum/photon-hdf5).</i>\n",
    "\n",
    "# 1. How to run it?\n",
    "\n",
    "The notebook is composed by \"text cells\", such as this paragraph, and \"code cells\"\n",
    "containing the code to be executed (and identified by an `In [ ]` prompt). \n",
    "To execute a code cell, select it and press **SHIFT+ENTER**. \n",
    "To modify an cell, click on it to enter \"edit mode\" (indicated by a green frame), \n",
    "then type.\n",
    "\n",
    "You can run this notebook directly online (for demo purposes), or you can \n",
    "run it on your on desktop. For a local installation please refer to:\n",
    "\n",
    "- [Jupyter Notebook Quick-Start Guide](http://jupyter-notebook-beginner-guide.readthedocs.org) \n",
    "\n",
    "<br>\n",
    "<div class=\"alert alert-info\">\n",
    "Please run each each code cell using <b>SHIFT+ENTER</b>.\n",
    "</div>\n",
    "\n",
    "# 2. Prepare the data file\n",
    "\n",
    "## 2.1 Upload the data file\n",
    "\n",
    "<br>\n",
    "<div class=\"alert alert-info\">\n",
    "<b>Note:</b> if you are running the notebook locally skip to section <b>2.2</b>.\n",
    "</div>\n",
    "\n",
    "Before starting, you have to upload a data file to be converted to Photon-HDF5.\n",
    "You can use one of our example data files available\n",
    "[on figshare](http://dx.doi.org/10.6084/m9.figshare.1455963). \n",
    "\n",
    "To upload a file (up to 35 MB) switch to the \"Home\" tab in your browser, \n",
    "click the **Upload** button and select the data file\n",
    "and wait until the upload completes.\n",
    "To upload files larger than 35 MB (like some of our example files) please use the \n",
    "[Upload notebook](Upload data files.ipynb) instead.\n",
    "\n",
    "Once the file is uploaded, come back here and follow the instructions below.\n",
    "\n",
    "## 2.2 Select the file\n",
    "\n",
    "Specify the input data file in the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "outputs": [],
   "source": [
    "filename = 'data/Pre.ht3'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell will check if the `filename` location is correct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "try: \n",
    "    with open(filename): pass\n",
    "    print('Data file found, you can proceed.')\n",
    "except IOError:\n",
    "    print('ATTENTION: Data file not found, please check the filename.\\n'\n",
    "          '           (current value \"%s\")' % filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case of file not found, please double check the file name\n",
    "and that the file has been uploaded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Load the data\n",
    "\n",
    "We start by loading the software:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import phconvert as phc\n",
    "print('phconvert version: ' + phc.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we load the input file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "outputs": [],
   "source": [
    "d, meta = phc.loader.nsalex_pq(filename,\n",
    "                               donor = 1,\n",
    "                               acceptor = 0,\n",
    "                               alex_period_donor = (1540, 3050),\n",
    "                               alex_period_acceptor = (150, 1500),\n",
    "                               excitation_wavelengths = (470e-9, 635e-9),\n",
    "                               detection_wavelengths = (525e-9, 690e-9),\n",
    "                               )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta['hardware_name']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we plot the `nanotimes` histogram:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phc.plotter.alternation_hist(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous plot is the `nanotimes` histogram for the donor and acceptor channel separately.\n",
    "The shaded areas marks the donor (*green*) and acceptor (*red*) excitation periods.\n",
    "\n",
    "If the histogram looks wrong in some aspects (no photons, wrong detectors\n",
    "assignment, wrong period selection) please go back to the previous cell \n",
    "which loads the file and change the parameters until the histogram looks correct.\n",
    "\n",
    "You may also find useful to see how many different detectors are present\n",
    "and their number of photons. This information is shown in the next cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "detectors = d['photon_data']['detectors']\n",
    "\n",
    "print(\"Detector    Counts\")\n",
    "print(\"--------   --------\")\n",
    "for det, count in zip(*np.unique(detectors, return_counts=True)):\n",
    "    print(\"%8d   %8d\" % (det, count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Filtering photons\n",
    "\n",
    "Most photons are the detector 127, we can filter them out to save space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid = detectors != 127\n",
    "for field in ('detectors', 'timestamps', 'nanotimes'):\n",
    "    d['photon_data'][field] = d['photon_data'][field][valid]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Metadata\n",
    "\n",
    "In the next few cells, we specify some metadata that will be stored \n",
    "in the Photon-HDF5 file. Please modify these fields to reflect\n",
    "the content of the data file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "author = 'John Doe'\n",
    "author_affiliation = 'Research Institution'\n",
    "description = 'A demonstrative smFRET-nsALEX measurement.'\n",
    "sample_name = 'Doubly-labeled ssDNA partially hybridized to a complementary strand.'\n",
    "dye_names = 'ATTO488, ATTO647N'\n",
    "buffer_name = 'Tris20 mM Ph 7.8'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Conversion\n",
    "<br>\n",
    "<div class=\"alert alert-success\">\n",
    "<p>Once you finished editing the the previous sections you can proceed with\n",
    "the actual conversion. To do that, click on the menu <i>Cells -> Run All Below</i>.\n",
    "\n",
    "<p>After the execution go to <b>Section 6</b> to download the Photon-HDF5 file.\n",
    "</div>\n",
    "\n",
    "The cells below contain the code to convert the input file to Photon-HDF5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 Add metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['description'] = description\n",
    "\n",
    "d['sample'] = dict(\n",
    "    sample_name=sample_name,\n",
    "    dye_names=dye_names,\n",
    "    buffer_name=buffer_name,\n",
    "    num_dyes = len(dye_names.split(',')))\n",
    "\n",
    "d['identity'] = dict(\n",
    "    author=author,\n",
    "    author_affiliation=author_affiliation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For completeness, we also store all the raw metadata from the HT3 header in a user group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove some empty groups that may cause errors on saving\n",
    "_ = meta.pop('dispcurve', None)\n",
    "_ = meta.pop('imghdr', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#for k in meta:\n",
    "#    print('%s:\\n%s\\n' % (k, repr(meta[k])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['user'] = {'picoquant': meta}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 Save to Photon-HDF5\n",
    "\n",
    "This command saves the new file to disk. If the input data does not follows the Photon-HDF5 specification it returns an error (`Invalid_PhotonHDF5`) printing what violates the specs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phc.hdf5.save_photon_hdf5(d, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check it's content by using an HDF5 viewer such as [HDFView](https://www.hdfgroup.org/products/java/hdfview/).\n",
    "\n",
    "# 6. Load Photon-HDF5\n",
    "\n",
    "We can load the newly created Photon-HDF5 file to check its content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = d['_data_file'].filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h5data = phc.hdf5.load_photon_hdf5(filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phc.hdf5.dict_from_group(h5data.identity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phc.hdf5.dict_from_group(h5data.provenance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phc.hdf5.dict_from_group(h5data.setup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pprint(phc.hdf5.dict_from_group(h5data.photon_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h5data._v_file.close()"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "py36"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "nteract": {
   "version": "0.12.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}