docs/examples/provenance.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using the ctapipe Provenance service\n",
    "\n",
    "The provenance functionality is used automatically when you use most of ctapipe functionality (particularly `ctapipe.core.Tool` and functions in `ctapipe.io` and `ctapipe.utils`), so normally you don't have to work with it directly. It tracks both input and output files, as well as details of the machine and software environment on which a Tool executed. \n",
    "\n",
    "Here we show some very low-level functions of this system:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ctapipe.core import Provenance\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activities\n",
    "\n",
    "The basis of Provenance is an *activity*, which is generally an executable or step in a script. Activities can be nested (e.g. with sub-activities), as shown below, but normally this is not required:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = Provenance()  # note this is a singleton, so only ever one global provenence object\n",
    "p.clear()\n",
    "p.start_activity()\n",
    "p.add_input_file(\"test.txt\")\n",
    "\n",
    "p.start_activity(\"sub\")\n",
    "p.add_input_file(\"subinput.txt\")\n",
    "p.add_input_file(\"anothersubinput.txt\")\n",
    "p.add_output_file(\"suboutput.txt\")\n",
    "p.finish_activity(\"sub\")\n",
    "\n",
    "p.start_activity(\"sub2\")\n",
    "p.add_input_file(\"sub2input.txt\")\n",
    "p.finish_activity(\"sub2\")\n",
    "\n",
    "p.finish_activity()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.finished_activity_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Activities have associated input and output *entities*  (files or other objects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[ (x['activity_name'], x['input']) for x in p.provenance]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Activities track when they were started and finished:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[ (x['activity_name'],x['duration_min']) for x in p.provenance]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Full provenance\n",
    "\n",
    "The provence object is a list of activitites, and for each lots of details are collected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.provenance[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be better represented in JSON:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(p.as_json(indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing provenance info in output files\n",
    "\n",
    "* already this can be stored in something like an HDF5 file header, which allows hierarchies.\n",
    "* Try to flatted the data so it can be stored in a key=value header in a **FITS file** (using the FITS extended keyword convention to allow >8 character keywords), or as a table "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def flatten_dict(y):\n",
    "    out = {}\n",
    "\n",
    "    def flatten(x, name=''):\n",
    "        if type(x) is dict:\n",
    "            for a in x:\n",
    "                flatten(x[a], name + a + '.')\n",
    "        elif type(x) is list:\n",
    "            i = 0\n",
    "            for a in x:\n",
    "                flatten(a, name + str(i) + '.')\n",
    "                i += 1\n",
    "        else:\n",
    "            out[name[:-1]] = x\n",
    "\n",
    "    flatten(y)\n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = dict(activity=p.provenance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pprint(flatten_dict(d))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}