notebooks/Convert ns-ALEX Becker-Hickl files to Photon-HDF5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convert ns-ALEX Becker-Hickl SPC/SET files to Photon-HDF5\n",
    "\n",
    "## Summary\n",
    "This executable document (it is called a [Jupyter notebook](http://ipython.org/notebook.html)) \n",
    "will guide you through the conversion of a ns-ALEX data file from **SPC/SET**\n",
    "to [Photon-HDF5](http://photon-hdf5.org) format.\n",
    "\n",
    "If you are reading this notebook online, please refer to \n",
    "[this quick-start guide](http://jupyter-notebook-beginner-guide.readthedocs.org) \n",
    "for instructions on how to install the required software and run the notebook \n",
    "on your machine. \n",
    "\n",
    "If it's the first time you are using a Jupyter notebook, please click on \n",
    "*Help* -> *User Interface Tour* for a quick tour of the interface.\n",
    "\n",
    "In this notebook there are \"text cells\", such as this paragraph, and \"code cells\",\n",
    "containing the code to be executed. To execute a code cell, select it and \n",
    "press SHIFT+ENTER. To edit a code cell it must be selected and with a green \n",
    "frame around it.\n",
    "\n",
    "## Prepare the data files\n",
    "Your can run this notebook using example data files available \n",
    "[on figshare](http://dx.doi.org/10.6084/m9.figshare.1455963). If you use these \n",
    "example files, please unzip them and put them in a folder named \"data\" (lower case) \n",
    "inside the folder containing this notebook.\n",
    "\n",
    "Alternatively, you can use your own SPC/SET files. In this case, you need to paste \n",
    "the full path of your SPC file in the following cell, instead of the path between \n",
    "single quotes `'`.\n",
    "\n",
    "> **NOTE**: if your path contains the `'` character please use `\"` as string delimiter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filename = r'data/dsdna_d7_d17_50_50_1.spc'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you also have a SET file it is expected to be in the same folder and \n",
    "have the same name as the SPC file, with extension \".set\".\n",
    "\n",
    "The next cell will check whether the `filename` location is correct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "try: \n",
    "    with open(filename): pass\n",
    "    print('Data file found, you can proceed.')\n",
    "except IOError:\n",
    "    print('ATTENTION: Data file not found, please check the filename.\\n'\n",
    "          '           (current value \"%s\")' % filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case of file not found, please double check that have you put the example \n",
    "data files in the \"data\" folder, or that the path you have pasted in `filename`\n",
    "is correct. Please re-execute the last two cells until the file is found."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data file description\n",
    "\n",
    "In the next few cells, we specify the additional metadata that will be stored \n",
    "in the Photon-HDF5 file. If you are using the example file you don't need to \n",
    "edit any of these. If are using your own file, please modify these description\n",
    "accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Author\n",
    "\n",
    "These fields will go in the [identity](http://photon-hdf5.readthedocs.org/en/latest/phdata.html#identity-group) group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "author = 'Eitan Lerner'\n",
    "author_affiliation = 'UCLA'\n",
    "creator = 'Antonino Ingargiola'\n",
    "creator_affiliation = 'UCLA'\n",
    "license = 'Public Domain'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample\n",
    "\n",
    "Except for `description`, all these fields will go in the [sample](http://photon-hdf5.readthedocs.org/en/latest/phdata.html#sample-group) group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "description = 'A demonstrative smFRET-nsALEX measurement.'\n",
    "sample_name = '50-50 mixture of two FRET samples'\n",
    "dye_names = 'ATTO550, ATTO647N'\n",
    "buffer_name = 'TE50'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please edit the previous cells and execute them (SHIFT+ENTER) to make sure \n",
    "there are no errors. Then proceed to the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data\n",
    "\n",
    "Before loading the data we need to load a few library functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import phconvert as phc\n",
    "print('phconvert version: ' + phc.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can load the input file and assign the measurement parameters \n",
    "([measurement_specs](http://photon-hdf5.readthedocs.org/en/latest/phdata.html#measurement-specs)), \n",
    "necessary to create a complete Photon-HDF5 file.\n",
    "\n",
    "If using your own file, please review all the parameters in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "d, meta = phc.loader.nsalex_bh(filename,\n",
    "                               donor = 4,\n",
    "                               acceptor = 6,\n",
    "                               laser_repetition_rate = 40e6,\n",
    "                               tcspc_range = 60e-9,\n",
    "                               timestamps_unit = 60e-9,\n",
    "                               alex_period_donor = (1800, 3300),\n",
    "                               alex_period_acceptor = (270, 1500),\n",
    "                               excitation_wavelengths = (532e-9, 635e-9),\n",
    "                               detection_wavelengths = (580e-9, 680e-9),\n",
    "                               allow_missing_set = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell plots a `nanotimes` histogram for the donor and acceptor channel separately.\n",
    "The shaded areas marks the donor (*green*) and acceptor (*red*) excitation periods.\n",
    "\n",
    "If the histogram looks wrong in some aspects (no photons, wrong detectors\n",
    "assignment, wrong period selection) please go back to the previous cell \n",
    "and tweak the relevant parameters until the histogram looks correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 4))\n",
    "phc.plotter.alternation_hist(d, ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may also find useful to see how many different detectors are present\n",
    "and their number of photons. This information is shown in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "detectors = d['photon_data']['detectors']\n",
    "\n",
    "print(\"Detector    Counts\")\n",
    "print(\"--------   --------\")\n",
    "for det, count in zip(*np.unique(detectors, return_counts=True)):\n",
    "    print(\"%8d   %8d\" % (det, count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# File conversion\n",
    "\n",
    "Once you finished editing the the previous sections you can proceed with\n",
    "the actual conversion. It is suggested to execute the notebook in\n",
    "one step by clicking on the menu *Cells* -> *Run All*.\n",
    "\n",
    "After that, you should find a new `.hdf5` file in the same folder of the input\n",
    "file. You can check it's content by using [HDFView](https://www.hdfgroup.org/products/java/hdfview/).\n",
    "\n",
    "The cells below contain the code to convert the input file to Photon-HDF5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "d['description'] = description\n",
    "\n",
    "d['sample'] = dict(\n",
    "    sample_name=sample_name,\n",
    "    dye_names=dye_names,\n",
    "    buffer_name=buffer_name,\n",
    "    num_dyes = len(dye_names.split(',')))\n",
    "\n",
    "d['identity'] = dict(\n",
    "    author=author,\n",
    "    author_affiliation=author_affiliation,\n",
    "    creator=creator,\n",
    "    creator_affiliation=creator_affiliation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add custom (user) data\n",
    "\n",
    "Here we save a custom (*user*) group to store all the metadata\n",
    "found in the original file (.SET). \n",
    "\n",
    "The important metadata has already been extracted and placed \n",
    "in standard Photon-HDF5 fields.\n",
    "However, here we dump the full content of the metadata in \n",
    "the input file to prevent any possible information loss\n",
    "during the conversion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Put `meta` in /user/becker_hickl\n",
    "d['user'] = dict(becker_hickl=meta)\n",
    "\n",
    "# Descriptions for custom fields\n",
    "user_descr = {'/user': 'A custom group.',\n",
    "              '/user/becker_hickl': 'Full metadata from original .SET file.'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save to Photon-HDF5\n",
    "\n",
    "This command saves the new file to disk. If the input data does not follows the Photon-HDF5 specification it returns an error (`Invalid_PhotonHDF5`) printing what violates the specs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "phc.hdf5.save_photon_hdf5(d, close=False, overwrite=True, user_descr=user_descr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "h5file = d['_data_file']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#d['_data_file'].close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Print HDF5 file content\n",
    "\n",
    "Finally, we print the file content to see what's inside the newly-created Photon-HDF5. Here we print the content of the root node:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "phc.hdf5.print_children(h5file.root)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here we retrieve some information from the user group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "phc.hdf5.dict_from_group(h5file.root.user.becker_hickl.identification)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#phc.hdf5.dict_from_group(h5file.root.user.becker_hickl.sys_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Photon-HDF5\n",
    "\n",
    "Finally we try to reload the file to check that there are no errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filename = d['_data_file'].filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "h5data = phc.hdf5.load_photon_hdf5(filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "phc.hdf5.dict_from_group(h5data.identity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "phc.hdf5.dict_from_group(h5data.setup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pprint(phc.hdf5.dict_from_group(h5data.photon_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "h5data._v_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the next cell output shows \"OK\" then the execution is terminated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print('OK')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}